House of Commons Children, Schools and Families Committee
Policy and delivery: the National Curriculum tests delivery failure in 2008
Sixth Report of Session 2008–09
Report, together with formal minutes, oral and written evidence
Ordered by the House of Commons to be printed 15 July 2009
HC 205 [Incorporating HC 980-i and -ii, Session 2007–08] Published on 23 July 2009 by authority of the House of Commons London: The Stationery Office Limited £0.00
The Children, Schools and Families Committee
The Children, Schools and Families Committee is appointed by the House of Commons to examine the expenditure, administration and policy of the Department for Children, Schools and Families and its associated public bodies.
Membership at time Report agreed Mr Barry Sheerman MP (Labour, Huddersfield) (Chairman) Annette Brooke MP (Liberal Democrat, Mid Dorset & Poole North) Mr Douglas Carswell MP (Conservative, Harwich) Mr David Chaytor MP (Labour, Bury North) Mrs Sharon Hodgson MP (Labour, Gateshead East & Washington West) Paul Holmes MP (Liberal Democrat, Chesterfield) Fiona Mactaggart MP (Labour, Slough) Mr Andrew Pelling MP (Independent, Croydon Central) Mr Andy Slaughter MP (Labour, Ealing, Acton & Shepherd’s Bush) Helen Southworth MP (Labour, Warrington South) Mr Graham Stuart MP (Conservative, Beverley & Holderness) Mr Edward Timpson MP (Conservative, Crewe & Nantwich) Derek Twigg MP (Labour, Halton) Lynda Waltho MP (Labour, Stourbridge)
The following member was also a member of the committee during the inquiry.
Mr John Heppell MP (Labour, Nottingham East)
Powers The Committee is one of the departmental select committees, the powers of which are set out in House of Commons Standing Orders, principally in SO No 152. These are available on the Internet via www.parliament.uk
Publications The Reports and evidence of the Committee are published by The Stationery Office by Order of the House. All publications of the Committee (including press notices) are on the Internet at www.parliament.uk/csf/
Committee staff The current staff of the Committee are Kenneth Fox (Clerk), Sarah Thatcher (Second Clerk), Emma Wisby (Committee Specialist), Judith Boyce (Committee Specialist), Jenny Nelson (Senior Committee Assistant), Kathryn Smith (Committee Assistant), Jim Lawford (Committee Support Assistant), and Brendan Greene (Office Support Assistant)
Contacts All correspondence should be addressed to the Clerk of the Children, Schools and Families Committee, House of Commons, 7 Millbank, London SW1P 3JA. The telephone number for general enquiries is 020 7219 6181; the Committee’s e- mail address is [email protected].
Policy and delivery: the National Curriculum tests delivery failure in 2008 1
Contents
Report Page
Summary 3
1 The inquiries into the 2008 test delivery failure 5
2 Delegation of responsibility from DCSF to QCA 9
3 The role of DCSF observers 14 QCA and DCSF Observers 14 Ofqual and DCSF observers 16
4 National Curriculum test delivery post-ETS 18
5 Conclusion 22
Conclusions and recommendations 23
Formal Minutes 26
Witnesses 27
List of written evidence 27
List of unprinted evidence 28
List of Reports from the Committee during the current Parliament 29
Policy and delivery: the National Curriculum tests delivery failure in 2008 3
Summary
In 2008, the Qualifications and Curriculum Authority (QCA) and its contractor, ETS Europe, failed to deliver Key Stage 2 and Key Stage 3 National Curriculum tests on time. Although the majority of scripts had been marked by the deadline for delivery of 8 July, a significant minority were outstanding, resulting in considerable disruption for many schools and children.
On 7 July 2008, Lord Sutherland of Houndwood was appointed by both the regulator, Ofqual, and the Department for Children, Schools and Families (DCSF) to chair an inquiry into the causes of the delivery failure. On the same day, this Committee started taking evidence on what happened during the 2008 testing cycle. During our inquiry, we took evidence from all the major parties, including both written and oral evidence from the contractor, ETS Europe in September 2008. Shortly thereafter, ETS withdrew its personnel and resources from the UK as a result of its contract with QCA being terminated and it announced that it would not be taking part in Lord Sutherland’s inquiry. Lord Sutherland was, however, able to rely on the evidence we had already taken from ETS and documentary evidence available from QCA.
The Report of the Sutherland Inquiry, published on 16 December 2008, set out clearly the sequence of events leading up to the missed deadline for delivery of the National Curriculum tests in 2008 and made a series of recommendations. We have endorsed Lord Sutherland’s work. Our inquiry has had a rather different focus and we have considered more widely than did Lord Sutherland the role of the Department in the events leading up to the delivery failure.
We have concluded that the Government should revisit the conduct of its relationships with its delivery agencies. Whilst acknowledging the difficulty of establishing the appropriate dividing line between policy and delivery, we believe that DCSF has involved itself too much in the detail of delivery, placing undue constraints on the executive decision-making abilities of its agency, QCA. We recommend that the leadership of government agencies should be more prepared to stand up to the Government when it considers that directions from the Government to the agency are unreasonable or incapable of performance.
We have also considered the role of DCSF observers on various boards and committees of non-departmental public bodies generally and QCA in particular. We have no objection in principle to the presence of such observers within a public body. However, in the case of QCA, we are concerned about the undue influence which appears to have been wielded by observers in terms of ‘negotiating’ formal advice from QCA to ministers and in seeking to influence QCA’s decision-making. We consider that the role of departmental observers should be clarified and appropriate safeguards put in place to assure transparency and prevent inappropriate interference.
We recommend unequivocally that there should be no place for departmental observers within an independent regulator. The independence of the new regulator, Ofqual, should be put beyond question and that requires the relationship between DCSF and Ofqual to be
4 Policy and delivery: the National Curriculum tests delivery failure in 2008
conducted on a formal and transparent basis. The presence of CSF observers on Ofqual’s Board would, in this context, be totally inappropriate.
Finally, we welcome the abolition of mandatory national testing at Key Stage 3 as a means of reducing the burden of testing on English school pupils; but warn against the unduly hasty introduction of single-level tests. There are considerable logistical complexities involved in delivering single-level tests twice each year and we recommend that the Government satisfies itself that the logistical arrangements are robust before introducing these tests as a replacement for the current testing regime. We also recommend that any significant future reform of National Curriculum testing must include an evaluation of on- screen marking.
Policy and delivery: the National Curriculum tests delivery failure in 2008 5
1 The inquiries into the 2008 test delivery failure
1. As a result of a procurement process which started in 2006, the Qualifications and Curriculum Authority (QCA) appointed ETS Global BV (ETS) as the new supplier of test operation services from 2008. Over the five-year contract term, ETS would be responsible for the administration of all national testing in England at Key Stages 1, 2 and 3 and Year 7 progress tests. This would include responsibility for the external marking of the Key Stage 2 and 3 tests and the Year 7 progress tests.1
2. The parent company of the ETS division to which this contract was awarded is a US non-profit organisation with long experience in test administration and which was, in 2008, responsible for managing 50 million examinations in 180 different countries.2 Despite this vast experience, the 2008 National Curriculum testing cycle was not delivered in its entirety on time. A significant minority of test papers at Key Stage 2 and 3 were not marked and returned to schools in time for the deadline of 8 July and some schools and pupils were still awaiting their test results in the following Autumn Term.3 Members of this Committee are even aware of some cases where results were still outstanding in the spring term of 2009.4
3. The inevitability of ETS’s failure to deliver on time did not, it seems, become apparent to the QCA until the week beginning 23 June 2008. By the beginning of the following week, senior officials and Ministers at DCSF had been notified and, on 4 July, the QCA issued a press release advising of the delay in publication of Key Stage 2 and 3 results.5 On that same day, both Ofqual, the regulator with responsibility for overseeing National Curriculum tests, and DCSF declared their intention to set up an independent inquiry into what went wrong. On 7 July, it was announced that Lord Sutherland of Houndwood would chair the inquiry and report to both Ofqual and DCSF.6
4. Well before the announcement of the Sutherland Inquiry, members of this Committee had been aware that there were significant concerns about the 2008 testing cycle and were already alert to the serious problems with markers.7 We were, therefore, able to start taking oral evidence on the delivery failures on 7 July, when the then Minister for Schools and Learners, the Rt Hon Jim Knight MP appeared before the Committee; and we started to receive written evidence from concerned individuals and schools from mid-July.8
1 http://www.qca.org.uk/qca_9788.aspx 2 Ev 8 3 Report of the Sutherland Inquiry, para 1.4 4 Q 237 5 Q 26; Q 104; Q 144; Qq 346–349; Qq 379–380; http://testsandexams.qca.org.uk/18339.aspx 6 http://www.ofqual.gov.uk/1717.aspx; http://www.dcsf.gov.uk/pns/DisplayPN.cgi?pn_id=2008_0142 7 Q 192-193 8 Oral evidence of Rt Hon Jim Knight MP given during an oral evidence session on a separate inquiry into the National Curriculum, published as HC 651-iii. The full transcript is printed as part of the Committee’s Fourth Report of Session 2008-09, HC 344-II.
6 Policy and delivery: the National Curriculum tests delivery failure in 2008
5. The very next week, on 14 and 16 July respectively, we took oral evidence from both Dr Ken Boston, then Chief Executive of the QCA, and the Secretary of State, the Rt Hon Ed Balls MP.9 As the summer break for both Parliament and schools approached, it became apparent that a significant number of schools and children would remain without their test results. Unusually, we took the decision to sit during recess in order to take evidence from both ETS and Ofqual on 10 September. This would be the only formal, public occasion on which ETS gave evidence in relation to what went wrong with test delivery in 2008, and Lord Sutherland has acknowledged the timeliness and usefulness of our work in this respect when he thanked us:
…for focusing on the report and on the issues so promptly in July, as you did. I think you were the first above the line, making very important questions plain and putting them into the public arena. I found that a good starting point for me. I recognise that the Committee’s work has been integral throughout the sad period since June or July, when things started to go wrong. The sessions you held before the school holidays were very important. You picked out a number of major issues, and I was able to build on that in the evidence I asked for and in some of the cross-examinations that I carried out.10
6. We then suspended our inquiries pending Lord Sutherland’s investigation into the details of the 2008 testing cycle. He published his Report on 16 December 2008 and it amounted to a very thorough investigation of the sequence of events leading up to the missed deadline on 8 July 2008. In his Report, Lord Sutherland identified a large number of factors which contributed to the failure in 2008 and he cited this complexity when he gave evidence to us on 26 January 2009.11 It has not been the role of this Committee’s inquiries to examine the day-to-day management of test delivery at the level of detail considered by Lord Sutherland. We endorse the work carried out by Lord Sutherland in exposing the detail of the many factors contributing to the failures in the delivery of the 2008 National Curriculum tests.
7. Nevertheless, it is unfortunate that Lord Sutherland was eventually unable to take formal evidence from ETS. Because the contractor was a US organisation and likely to withdraw personnel and resources from the UK, time was of the essence in commencing formal inquiries. Lord Sutherland stated in the Introduction to his Report:
In conducting my Inquiry, I launched a call for written evidence and interviewed individuals from key organisations. I should say at the outset of this report that ETS declined to submit documentary evidence to the Inquiry. It has nonetheless been possible to do a thorough investigation drawing particularly on extensive materials submitted by QCA.12
9 Oral evidence of the Secretary of State given during an oral evidence session on a separate inquiry into Public Expenditure, published as HC 835-ii. The full transcript is printed as part of the Committee’s First Report of Session 2008-09, HC 46. 10 Q 237 11 Q 237 12 Report of the Sutherland Inquiry, p2
Policy and delivery: the National Curriculum tests delivery failure in 2008 7
From the point of view of demonstrable fairness, it was regrettable that ETS had withdrawn its resources from this country and chose not to engage with Lord Sutherland’s inquiry, although we understand that there was an exchange of correspondence between the inquiry and ETS in December shortly before publication of the Report.13 It was clear from Lord Sutherland’s report that he had relied heavily on the evidence this Committee took from ETS in September 2008, since ETS submitted neither written nor oral evidence to his inquiry.14 Nevertheless, Lord Sutherland considered that he was able to produce a fair report given the access he had to QCA documentation and exchanges of correspondence between the parties.15 He told us that:
I have no doubt that there was adequate evidence available to me to make the judgments that I believe are definitive about the role of ETS in this report.
[ETS] believed at one point, in view of something that it said to your Committee … that there was an embargo on it in some way. I spoke to the QCA chief executive, and he wrote to ETS to make it plain that there was no such embargo. That is point one. Point two: we extended the period available to it to submit evidence. Point three: it chose not to do so. Point four: the evidence was available to us, through the QCA documentation—we had all the exchanges of correspondence and contracts and so on, so I believe I had adequate evidence.16
8. This inquiry provides a good illustration of how a select committee can use its authority and powers in a timely manner to investigate problematic incidents as soon as they arise. Our swift action in this matter has enabled us to put on the public record both written and oral evidence from a party to the incident, ETS, which was not available by the time the official inquiry was fully operational. The Sutherland Inquiry was then able to rely on the evidence we had secured when it became clear that ETS would not engage with the process.
9. In carrying out his inquiries, Lord Sutherland was given separate, but related, terms of reference from both Ofqual and DCSF.17 Dr Boston expressed concern that, by issuing parallel terms of reference to Lord Sutherland, DCSF effectively closed down avenues of inquiry which should legitimately have been within Ofqual’s terms of reference to his Inquiry, namely what he saw as DCSF’s active role in the 2008 test delivery failure.18 Dr Ken Boston has argued that, in issuing its own terms of reference, the Department acted to prevent Lord Sutherland inquiring into the way it set up the framework of policies and other decisions within which QCA was required to work.19 This framework included the terms of the annual remit letter from DCSF, which is accepted by QCA and which sets out what the Government expects QCA to achieve each year.
13 Letter to the Inquiry from Dr Philip Tabbiner (ETS), 1 December 2008, referenced in the Report of the Sutherland Inquiry 14 Report of the Sutherland Inquiry, paras 3.43, 3.68, 3.74, 4.101, 5.20, 5.34, 5.89, 5.126, 5.142, 5.173 15 Q 244 16 Q 244 17 Report of the Sutherland Inquiry, Annex C and D 18 Q 311 19 Q 311
8 Policy and delivery: the National Curriculum tests delivery failure in 2008
10. It is indeed the case that the terms of reference issued to Lord Sutherland by DCSF mention DCSF’s active role in the delivery failure only in terms of the appropriateness of its arrangements to monitor the QCA’s delivery against its remit. The terms of reference went on to specify that Lord Sutherland could interview “appropriate people” from QCA, ETS Europe, Ofqual, DCSF and others.20 It is certainly not obvious from the terms of reference that Lord Sutherland was mandated to inquire into QCA’s remit letter and related instructions from DCSF. Nevertheless, the Secretary of State told us at the time the Inquiry was set up that:
Nothing is out of bounds for Lord Sutherland. He can look at the whole process, and I am very happy for him to look at whether the original remit was properly specified.21
In addition, Lord Sutherland has said that his terms of reference gave him:
…ample scope and every opportunity to examine the processes, roles and responsibilities of all of the key players: not only QCA and ETS, but also Ministers and officials at the DCSF.22
11. It is not clear to us that the DCSF having issued parallel terms of reference prevented Ofqual asking Lord Sutherland to inquire into the DCSF’s role, had Ofqual considered that appropriate. Lord Sutherland chose not to consider the Department’s role beyond its oversight of the particular circumstances of the delivery of the National Curriculum tests in 2008. However, this Committee is able to go further than he did and comment in more general terms on the line between policy and execution.
12. Specifically, we will consider whether there was anything in the way the Department set up the framework within which QCA was required to operate which led to an increased risk of delivery failure. We will also consider the role of departmental observers and the implications for QCA decision-making and the independence of the new regulator, Ofqual. We develop these themes in more detail below.
20 Report of the Sutherland Inquiry, Annex D 21 Q 139 22 Q 360, Secretary of State quoting a statement of Lord Sutherland given to the Press Association
Policy and delivery: the National Curriculum tests delivery failure in 2008 9
2 Delegation of responsibility from DCSF to QCA
13. The Government insisted that QCA manages the testing system on behalf of and at arm’s length from DCSF. Both the Secretary of State and the Minister for Schools and Learners emphasised at different times that they thought it would be inappropriate for ministers to be involved in the administration of the testing process. They thought that to do otherwise may lead to suspicion that they might be influencing results on the basis of which the Government’s record in education is judged.23 The Secretary of State explained to us the lines of responsibility and accountability within the testing system:
It is Ministers who are accountable to Parliament, directly and through the Select Committee, for the operation of our schools system, including the testing regime, so in the end the accountability comes to Ministers. …
… We are accountable for the funding of the regime and the way that it operates. We ask a non-departmental public body, the QCA, to deliver the tests on our behalf at arm’s length from us. The QCA then contracts independently of Ministers with the people who do the practical delivery of the tests. So the accountability is as follows: ETS is accountable to the QCA for the delivery of its contract; the QCA is accountable to us and more widely for ensuring that that contract is effectively delivered; and I am accountable for ensuring that the QCA fulfils its responsibilities and for the overall operation of the regime. …24
14. Lord Sutherland also gave us his view of the responsibilities of DCSF and QCA:
The DCSF set the policy, and it provides the resources—that is its responsibility. It has to ensure that delivery is possible, which means it must have an appropriate mechanism. It believed it had, and on the basis of the two or three previous years it did have, because delivery was made. Below that, we come to QCA, which has a set of responsibilities, and so on, down the system it goes.
In my understanding, the DCSF is responsible for policy, resource and setting up the overall mechanism. The kind of mechanism that is set up relates to how well it might work.25
15. An important part of the framework set by DCSF within which QCA had to work is to be found in the remit letter, issued by DCSF and accepted by QCA in March each year.26 This letter covers the areas for which QCA will assume responsibility and sets out the funding to be made available. In fact, the letter itself does not go into great detail. The letter
23 Q 193; Q 364 24 Q 118 25 Q 251 26 The remit letter for 2007-08 is at http://www.qca.org.uk/libraryAssets/media/Alan_Johnson_27_03_2007_- _Remit_ltr.pdf; the remit letter for 2008–09 is at http://www.qca.org.uk/libraryAssets/media/Ed_Balls_to_Sir_Anthony_Greener_12032008.pdf.
10 Policy and delivery: the National Curriculum tests delivery failure in 2008
sent by the Secretary of State in March 2008 specifies in relation to National Curriculum tests only that they:
…are delivered successfully in May 2008, with an improvement in key metrics over 2007, such as quality of marking, reduction in number of lost scripts, and an improved service to schools. Secure timely preparation for 2009 delivery.27
16. The significance of the remit letter can perhaps be viewed in terms of its symbolic nature: it is the point at which QCA formally agrees to work within the framework of formal and informal instructions set out by DCSF for the next year. The Department maintains that, as a result of the issue and acceptance of the remit letter on an annual basis, responsibility for the matters contained in it passes from DCSF to QCA (although accountability is arguably a different issue).28 Lord Sutherland noted the prevalence of the model of central government delegating delivery responsibilities to non-departmental public bodies and agencies. He also noted that the model of delegation is “predicated on the sponsor department’s confidence in the NDPB’s capacity and capability to deliver what is asked in the remit letter”.29
17. The remit letter, then, is intended to demarcate the responsibilities of DCSF and QCA. However, there did appear to be some confusion in relation to the degree to which QCA is separate from DCSF in the context of policy execution. At one point, the Secretary of State told us that:
… Ministers are accountable to Parliament for the overall delivery of our schools policy, including the national testing regime. It is my responsibility to ensure that that happens, and I do so in an arm’s-length way through an independent body, the QCA, which contracts with ETS.30
However, he later told us that:
… the QCA is a non-departmental public body. It is not independent; it reports to Ministers, who are then accountable to Parliament for the national curriculum and the delivery of tests.
Ken Boston was not an independent decision maker or commentator on Government policy.31
18. Similar inconsistencies were expressed by Dr Ken Boston. In the context of a discussion about the level of Government control over the details of the testing system, he said that:
The QCA’s problem has been that it is not at arm’s length—to use the Government’s term—yet it can end up carrying the blame for a whole set of decisions over which it had no control.32
27 http://www.qca.org.uk/libraryAssets/media/Ed_Balls_to_Sir_Anthony_Greener_12032008.pdf 28 Q 364 29 Report of the Sutherland Inquiry, para 3.104 30 Q 126 31 Q 364
Policy and delivery: the National Curriculum tests delivery failure in 2008 11
On the other hand, in the context of test delivery he later stated that “Ministers were not getting involved in the detail of it at all”.33
19. Although we do not wish to make too much of these apparent inconsistencies—they are, perhaps, explicable in context—it does highlight the broader issue of the extent to which DCSF did, in fact, have a hand in the details of the management of the testing system.
20. Ken Boston has emphasised the many areas over which the Government had and retained control:
… the Government, through the DCSF, determine the nature of the tests to be procured, the pupils who are to be tested, which subjects they are to be tested in, how and when they are to be tested and how much money has to be spent on testing. The interval between the date of the test and the date for results—the eight weeks’ interval—is set by the Government. They also determine whether the tests are marked manually or onscreen; which results data are collected and how they are collected; the form in which the data are required; and how the results are returned to schools. In those core elements of test development and delivery, neither the QCA nor the supplier has any discretion—decisions are made by the Government alone. The development and the delivery of these tests are not at arm’s length from the Government; the Ministers and the officials are at arm’s length only from the detail of the test questions, and from the marking and level setting.34
Many of the details to which Dr Boston referred in this passage were not set out explicitly in the remit letter. This highlights the fact that the framework within which QCA is required to operate is actually a complex series of formal and informal, written and oral instructions and policies issued by DCSF over a period of time and not in one letter in March each year.35
21. Questions arise where the remit offered to QCA appears to be incapable of performance, the budget appears unreasonable or both. Lord Sutherland expressed the view in his evidence to us that it was always open to the leadership of QCA to resign if they thought that the offer from DCSF was unreasonable and its performance likely impossible.36 The fact was that they did not do so. Lord Sutherland later conceded that resignation would be a “nuclear option” and that alternatives would be public statements or a letter from QCA to DCSF stating that the QCA was being asked to do too much. However, he found no evidence of such a letter.37
22. Dr Boston himself told us that he had come to regret not resigning in 2006 on the basis that he thought the remit could not be achieved.38 He said that decisions made by
32 Q 326 33 Q 332 34 Q 311 35 There is also a Memorandum of Understanding between the former Department for Education and Skills and QCA, but this does not appear to have been updated since the creation of DCSF; http://www.qca.org.uk/qca_4979.aspx 36 Q 300 37 Q 301 38 Q 325
12 Policy and delivery: the National Curriculum tests delivery failure in 2008
ministers and officials in 2006 had a “marked impact” on the failure in 2008.39 We asked him why, when delivery of National Curriculum tests had failed in 2004 and come close to failure in subsequent years, he did not raise the matter publicly when he failed to make any progress with ministers. He told us:
That is a good question, and it is one that I have asked myself. I have frequently spoken publicly about different approaches to testing and the problems with this particular approach and how it was administered. I have also pressed privately for reform with Ministers and officials—there is no question about that. Over the past few months I have asked myself why I did not resign in 2006. At that point it was clear to me that I was not going to get Government buy-in to necessary reform. I continued through 2007 and 2008 to run an organisation that was faced with close to an impossible task, which I had seen go belly up in 2004 and had the prospect of doing so again. Probably my greatest regret in all of this is that I was not more honest with myself at that time and did not simply say, “It can’t be done.”40
23. We are reminded of a report of our predecessor committee on Individual Learning Accounts (ILA) which detailed the “considerable shortcomings” of another contractor in the delivery of the ILA scheme, as well as the failings of the Department for Education and Skills itself.41 In that case, the Department retained “even the smallest details of policy design” and missed an opportunity to transfer to the private contractor fuller responsibility for the management of the scheme. As a result, it was unclear who was responsible, the Department or the contractor, for specific outcomes of the ILA project. The Committee concluded:
We do not under-estimate the difficulty of getting right the balance between policy and delivery, but we question whether the DfES could have been bolder and given Capita a wider brief to deliver the desired outcomes of the ILA project.42
The relationship between DCSF, QCA and ETS is, of course, quite different from the relationship between the DfES and the contractor in the ILA case. However, there is a broader principle here which is that, if Government is to sub-contract certain projects, it must not attempt to micro-manage the details of that project, for example, by over- specifying the means by which it wishes to see delivery made or through the mechanism of observers. The role of departmental observers within QCA will be considered in greater detail below (paragraphs 28 to 35), but they have essentially acted as mediators of information flowing between DCSF and QCA.
24. We maintain the view of our predecessor Committee, that the difficulty of establishing the correct dividing line between policy and execution should not be underestimated. However, we are concerned that DCSF appears to be specifying in considerable detail the ways in which it wishes to see its policies executed. The DCSF
39 Q 311 40 Q 325 41 Education and Skills Committee, Individual Learning Accounts, Third Report of Session 2001–02, HC 561-I, p3 42 Ibid., para 155
Policy and delivery: the National Curriculum tests delivery failure in 2008 13
achieves this both formally, through the QCA’s remit letter, and informally, through regular contact with its agency and through the mechanism of observers.
25. We agree with the Secretary of State that he is accountable for ensuring that the QCA fulfils its responsibilities for the overall operation of the testing regime. However, he is also accountable for the policies and other decisions of his Department, including the terms of the QCA’s remit letter; and he is responsible for ensuring that the directions issued by DCSF are capable of being carried out by the organisations tasked with their execution.
26. If significant areas of Government policy are to continue to be delivered by non- departmental public bodies and other agencies, the leadership of those bodies should be prepared to demonstrate their professionalism by challenging the Government if they consider that the impossible is being asked of them. Appropriate mechanisms should be put in place to allow them to make such a challenge. We recognise that this should never be used as a means to hold the Government to ransom or to impede the execution of legitimate public policy. Nevertheless, there needs to be a formal and transparent dialogue between the Department and its agencies.
27. In anticipation of new legislation, the QCA’s regulatory function has already been split from its core activities and hived off to the regulator, Ofqual, which is still technically part of QCA but is operating largely as if it were already independent. If the Apprenticeships, Skills, Children and Learning Bill is passed, what remains of the QCA will be transformed into the Qualifications and Curriculum Development Agency (QCDA). The Director General of the Schools Directorate at DCSF has told us that the Government intends for Ofqual to be entirely independent, whereas QCDA will be “significantly less” independent and is being established specifically as “a delivery agency of Government”.43
28. In the light of the repeated problems we have identified, we consider that DCSF should, in an updated Memorandum of Understanding or equivalent with each public body, set out more clearly how it will conduct its relationships with those bodies. We urge the Government to adopt a far less prescriptive approach when issuing instructions to the new Qualifications and Curriculum Development Agency. If it is considered appropriate that policy should be executed by an agency rather than by the Department itself, the potential benefits of such an arrangement are significantly diluted if the Government fails to trust the expertise and experience of the professionals chosen to run such agencies. It is clear that micro-management from the Department would introduce a degree of confusion into the lines of responsibility and accountability, as previously happened in the case of Individual Learning Accounts, investigated by our predecessor Committee, and the mistake has been repeated with arrangements for the delivery of National Curriculum tests.
43 Uncorrected transcript of evidence taken before the Children, Schools and Families Committee on 8 July 2009, HC 353-vi, (2008–09) Q 489
14 Policy and delivery: the National Curriculum tests delivery failure in 2008
3 The role of DCSF observers
QCA and DCSF Observers 29. One aspect of the constant process of instruction and negotiation between DCSF and QCA is the position of DCSF observers on various boards and committees of the QCA. In evidence to Lord Sutherland, QCA described the role of a DCSF observer in the context of the procurement process as “someone who could contribute to discussions and comment on the process, but was not someone who had a voting right in terms of evaluating the potential suppliers”.44 Lord Sutherland concluded in his report that, although DCSF viewed QCA as a “trusted delivery partner”, it nevertheless had observers monitoring the QCA at various levels of the organisation, including at operational meetings between QCA and ETS.45
30. As we have stated, it is not our intention in this Report to rehearse the detail of what happened in the particular case of the test delivery in 2008, which Lord Sutherland has already addressed admirably. However, there is a more general point about observers which we wish to make here. On crucial issues which would go to the credibility of the testing system, Dr Ken Boston was unambiguous, that the Government did not interfere with QCA’s operations:
I make it absolutely clear to you that in my time as Chief Executive of the QCA I have never had any interference from any Minister or any Government official to do with the nature of the paper, the quality of the marking, the standard that should be set, or the figure that should be reached in terms of performance, and if I had I would be the first person to declare that publicly.46
31. However, Dr Boston was highly critical of the role of DCSF observers within the QCA generally, stating that they undermined its authority:
By reducing the formality of the relationship between the two organisations—the Government and the QCA—it leads to negotiated compromises from time to time, which erode public accountability. The role of observers on boards and committees has been to take part in discussions, to provide advice to Ministers separately from the QCA and to relay ministerial feedback. It is in that third area where the problem arises. Observers typically advise in committees and boards that Ministers would be “minded to” or “not minded to” agree with this or that proposal or “content to” or “not content to” agree to a particular recommendation, or even that, “If I put that idea to the Minister it would be laughed out of court.” Now certainly, prior consultation with the DCSF is important to inform subsequent decisions by the QCA on what advice it should offer Ministers, but too often boards such as that of the QCA are put into a position where it is expected that they will seek to negotiate
44 Report of the Sutherland Inquiry, para 2.45 45 Report of the Sutherland Inquiry, para 3.105 46 Q 15
Policy and delivery: the National Curriculum tests delivery failure in 2008 15
that advice in advance. I think that is a pernicious process that compromises integrity and independence…47
32. This relationship between DCSF and QCA, mediated by DCSF observers, paints a picture of a Government which is not keeping delivery of national testing at arm’s length, but is in constant dialogue with its delivery agency in relation to the method of delivery. Indeed, Lord Sutherland told us that:
I believe there is so much [delegation of responsibility for delivery to government agencies] now that it is time to pause and say, “Have we got the best possible arrangement that allows such an allegedly arms-length delivery?”. It sounds grand, but the arm is a real thing and messages move up and down. It is happening across Government.48
33. The Secretary of State assured us that the role of observers was to ensure an appropriate flow of information between the Department and QCA. Observers were not deployed in order to influence decisions of the QCA, but to furnish it with policy information relevant to the decisions it was taking. In the other direction, observers could relay to the Department any concerns they might have about the service delivery (although Lord Sutherland considered that, in the case of the 2008 testing cycle, they had not done this effectively).49 The Secretary of State pointed to the presence of a Treasury observer on the Bank of England Monetary Policy Committee as another example of where observers were used without any allegations of undue influence.50
34. Lord Sutherland did not question the legitimacy of delegating responsibility for national testing and other issues from DCSF to QCA, quite the contrary; but he did consider that the role of DCSF observers on QCA’s operational, programme and corporate boards was not entirely clear. He therefore recommended that their role be clarified on a case-by-case basis.51
35. We have some serious concerns in relation to the role of observers and question whether, in many cases, they are necessary. In appropriate cases, departmental observers may perform a useful function in helping ministers remain accountable for policy delivery in a meaningful manner. However, formal advice to ministers from a public body should not be negotiated through departmental observers; nor should observers exert undue influence over the decision-making of a public body.
36. We concur with Lord Sutherland’s recommendation, accepted by Government, that the role of departmental observers should be clarified. We believe that the Memorandum of Understanding or equivalent between DCSF and its non- departmental public bodies should, in each case, set out clearly the appropriate role for departmental observers so that the public can be reassured that observers are not acting inappropriately to influence the work and decisions of public bodies.
47 Q 311 48 Q 254 49 Q 421; Report of the Sutherland Inquiry paras 4.143–4.146 50 Q 418 51 Report of the Sutherland Inquiry, para 3.144
16 Policy and delivery: the National Curriculum tests delivery failure in 2008
Ofqual and DCSF observers 37. The Office of the Qualifications and Examinations Regulator, Ofqual, was established on 8 April 2008. Prior to that, the regulatory function had been carried out by QCA in addition to its other duties.52 Ken Boston has pointed out the objection to this arrangement:
I support absolutely the notion of a regulatory authority reporting to Parliament rather than to Ministers. That has always been a difficulty with the QCA: it is responsible for maintaining the assessment standard, the height of the hurdle, yet reports at the same time to the Ministers who want to drive up the performance standard—the number of people who leap the hurdle. There is a real inconsistency there.53
38. The Apprenticeships, Skills, Children and Learning Bill will, if passed, establish Ofqual as an independent regulator reporting to Parliament. In the meantime, Ofqual remains technically part of QCA, and QCA has set up a new governance structure in order to allow Ofqual some independence under the current statutory framework. Nevertheless, Lord Sutherland recommended that, ahead of legislation, Ofqual’s reporting arrangements should be clarified.54
39. In relation to the National Curriculum tests in 2008, the role of Ofqual was to oversee delivery of the tests rather than to monitor the contract.55 According to Isabel Nisbet, Ofqual’s Chief Executive, Ofqual’s responsibility was to:
… make sure that our regulatory requirements were met. They are that the levels are properly set; the standards are properly made; the quality of the marking is good; and the tests are delivered in a way that meets the standards of previous years.56
40. Lord Sutherland expressed the view that the relationship between this Committee and Ofqual would become an important aspect of the regulator’s duty to report to Parliament.57 He considered that many of the problems of the current arrangements could be overcome with the establishment of a properly independent regulator with appropriate powers and resources.58 He thought that Ofqual should, in future, be consulted on the nature and details of the contract for delivery of National Curriculum test results awarded by QCA. This would enable Ofqual to ensure, for example, that the contractual requirement to provide the kind of information Ofqual needs for quality assurance is built into the contract from the start.59
52 Q 172 53 Q 317 54 Report of the Sutherland Inquiry, paras 6.107–6.110 55 Q 179 56 Q 178 57 Q 237 58 Q 254 59 Q 265
Policy and delivery: the National Curriculum tests delivery failure in 2008 17
41. However, doubts have been expressed about the prospects for Ofqual’s independence, both now and once it has been established as a statutory body. Dr Boston warned that the presence of DCSF observers on the Board of Ofqual could seriously compromise the de facto independence of the regulator. He referred to three functions of departmental observers: the provision of specialist expertise; the provision of separate advice to ministers; and speaking on behalf of ministers. He considered that the first function could be achieved by direct consultation between Ofqual and DCSF and that the second and third were quite inappropriate in the context of an independent regulator reporting to Parliament.60
42. Dr Boston recognised that a good exchange of information between DCSF and Ofqual was necessary but thought that the presence of observers on Ofqual’s Board may lead to the relationship becoming “too cosy” and would adversely affect both the regulator’s true independence and its strict accountability to Parliament alone.61 He illustrated his point with the example of how DCSF observers worked with QCA. Given that formal advice from QCA to ministers is made public in due course and other correspondence is potentially available under Freedom of Information rules, DCSF officials liked to see draft correspondence and often sought to negotiate its terms so that it could be made acceptable to both ministers and the public. Dr Boston considered that such practices would be wholly unacceptable if applied to an independent regulator.62 He stated that:
Ofqual will not be seen as conspicuously and unquestionably separate from Government so long as it has it accepts DCSF observers on its Board and committees. The public and the Parliament need to be absolutely certain that Ministers are being told what they need to know, not what they or their officials might want to hear.63
43. We expressed our concerns in relation to the potential for departmental observers to exert undue influence on the advice from and decision-making of public bodies. Whilst we are content that such observers should continue to have a role within bodies such as QCA with appropriate safeguards, we are opposed to the presence of departmental observers in the context of an independent regulator. The independence of Ofqual should be put beyond question and that requires contact between DCSF and Ofqual to be put on a much more formal footing than is suggested by the presence of observers on Ofqual’s Board.
60 Ev 76 61 Q 353 62 Ev 76 63 Ev 77
18 Policy and delivery: the National Curriculum tests delivery failure in 2008
4 National Curriculum test delivery post- ETS
44. The Committee stated in its Report on Testing and Assessment of May 2008 that children in England were tested too much and too often.64 We heard evidence that English school pupils were amongst the most tested in the world and that the national test burden was displacing real learning and deep understanding of a subject.65 We found significant evidence that, due to the high-stakes nature of national tests, some schools were teaching to the test and narrowing the taught curriculum in an effort to maximise performance and improve standings in school performance tables which were based on test results.66
45. The Government maintained that the amount of time spent by pupils actually taking exams was very limited, but this missed the point that the burden of testing is felt by pupils and teachers more in terms of the amount of time and effort spent in preparation than in sitting the tests themselves.67 The Government’s view, set out in its Response to our Report, was that there was no necessity for national testing to result in teaching to the test and narrowing of the curriculum and that the Government did not support excessive time spent on test preparation.68 This ignored the strong evidence that we received, from Ofsted amongst others, to the effect that these practices were both widespread and damaging.69
46. Nevertheless, the demise of the QCA’s contract with ETS, which was terminated in August 2008 as a result of the delivery failure, provided the Government with an opportunity to address our concerns that there was too much national testing.70 On 14 October 2008, the Secretary of State made an oral Statement in the House to announce the abolition of Key Stage 3 tests as a compulsory component of the national testing regime.71 The Secretary of State said that:
In my statement [to the House on 22 July 2008], I … made it clear that the current testing and assessment regime is not set in stone. I know that some hon. Members were disappointed that I was unable to go further at that time, but it was important that we evaluated the case for change before making decisions. Over the summer, we have been able to study the Select Committee’s report on testing and assessment, which was debated in the House last week. …
…I take seriously the concerns raised by the Select Committee, teachers and parents. Testing, assessment and accountability must encourage and reward the best teaching
64 Children, Schools and Families Committee, Testing and Assessment, Third Report of Session 2007–08, HC 169-I 65 Ibid. paras 141–149 66 Ibid. para 130 67 Ibid, paras 148–149 68 Children, Schools and Families Committee, Testing and Assessment: Government and Ofsted Responses to the Committee’s Third Report of Session 2007–08, Fifth Special Report of Session 2007–08, HC 1003, para 15 69 Children, Schools and Families Committee, Testing and Assessment, Third Report of Session 2007–08, HC 169-I, para 130 70 http://www.qca.org.uk/qca_19029.aspx 71 HC Deb, 14 Oct 2008, Col 677
Policy and delivery: the National Curriculum tests delivery failure in 2008 19
so that it properly supports pupils in their learning and development. Schools should be judged fairly on how they support the progress and well-being of every child. …
Having looked hard at the current testing regime, we do not believe that the three principles that I have set out [the provision of information to parents, teachers and the public for various purposes] justify the key stage 3 testing arrangements in their current form.72
We welcomed the abolition of mandatory Key Stage 3 testing, believing that a reduction in the burden of testing would give teachers greater freedom to offer children a balanced education.73
47. In line with our recommendations in our Report on Testing and Assessment of May 2008, we welcomed the abolition of mandatory national testing at Key Stage 3 when it was announced in October 2008 and we remain convinced that the decision to reduce the burden of national testing was a good one.
48. However, the logistics of National Curriculum test delivery remain complex, even after the abolition of Key Stage 3 testing. As Dr Ken Boston told us, the history of delivery of national tests is not a happy one:
We failed in 2004 when Key Stage 3 was late. We were within hours of failing in 2005, even though we had extended the Key Stage 3 results date. In 2006 and 2007 we were fine. In 2008 the whole thing collapsed again. The enterprise, the whole archaic nature of this thing, is incredible, and we are persisting with it for 2009. Lord Sutherland has made recommendations about end-to-end testing which simply cannot be met for 2009. Key Stage 3 is not being run this year but that does not halve the risk. It simply means that there will potentially be more markers who will be able to do Key Stage 2.74
Nor are we encouraged by the progress of the Government’s pilot study into ‘single-level tests’, which are intended as a possible replacement for the current ‘end of Key Stage’ testing regime. Single-level tests are based on the concept of testing when ready, with the pupil being tested at a set level to establish whether or not he or she has achieved that standard. Children would work their way through each level in turn and would be entered for single-level tests by their teacher once the teacher considers that they are ready for the relevant level.
49. If single-level tests were to be rolled out to replace the current testing regime, pupils would sit a number of tests over the course of a Key Stage rather than taking a single set of tests at the end of the Key Stage. Tests would be made available to schools twice each year and would be externally marked. In making the transition from end of Key Stage tests to single-level tests, there is the potential for a significant increase in the testing burden, with attendant logistical complexities.
72 HC Deb, 14 Oct 2008, Col 677 73 Children, Schools & Families Committee Press Notice, 14 October 2008, http://www.parliament.uk/parliamentary_committees/csf/csfpn141008a.cfm 74 Q 321
20 Policy and delivery: the National Curriculum tests delivery failure in 2008
50. We repeat our warning, set out in our Report on Testing and Assessment, that new tests should not be introduced nationally with undue haste.75 We are concerned that, if single-level tests are rolled out as a replacement for end of Key Stage testing, there is potential for significant complexity in the logistical arrangements for sending out, marking and returning a large number of single-level test scripts twice each year. Evidence from the pilot studies so far is not encouraging and demonstrates that there remain serious problems to be overcome. We remind the Government that we are opposed to an increase in the burden of national testing generally; and we warn that the Government must be quite sure that the logistical arrangements for single-level tests are robust before they are rolled out nationally.
51. Ken Boston told us that test administration has made significant progress in recent years, a view which is, perhaps, surprising given the regularity with which there have been near misses and outright failures in National Curriculum test delivery. However, he argues strongly in favour of a move towards on-screen marking as the obvious means of increasing the reliability of test delivery whilst, at the same time, improving the validity and reliability of marking.76 He pointed to the successful use of on-screen marking of GCSE and A Level scripts by Awarding Bodies as evidence that such a system could improve the marking of National Curriculum tests.77
52. What clearly does not work, as Lord Sutherland has pointed out, is a hybrid system such as that used by ETS in 2008 whereby scripts are marked manually but many different data points for each script are entered into a database manually to capture question-level data.78 Even leaving aside the particular problems with the systems developed by ETS (identified in Chapter 5 of Lord Sutherland’s report), manual marking coupled with manual data entry into a database is an extremely labour-intensive process, ill-suited to the pressured environment in which markers must deliver results within very tight time limits.
53. We have heard a significant amount of argument between Dr Boston on the one hand and DCSF on the other about who has and has not recommended on-screen marking over the years.79 Dr Boston told us that, from 2004, DCSF “declined to accept QCA advice to introduce on-screen marking” on the basis that it should first be demonstrated in a pilot study that the results would be the same or better than those resulting from manual marking. Dr Boston argued that such a trial would have had no value as on-screen marking could not be judged on the degree to which it replicated the “inferior process” of manual marking. He considered that the results would inevitably have been different, but not necessarily worse.80
54. The Secretary of State, on the other hand, said that:
75 Children, Schools and Families Committee, Testing and Assessment, Third Report of Session 2007–08, HC 169-I, para 198 76 Q 23 77 Q 8; Q 330 78 Report of the Sutherland Inquiry, paras 5.104–5.110.Question-level data gives information about how particular questions were answered, either by individual pupils or by groups of pupils. Schools can use this data to check, for example, that teachers are covering the curriculum effectively. 79 Qq 322, 330, 350, 360, 367, 419, 420; Ev 74 80 Ev 74
Policy and delivery: the National Curriculum tests delivery failure in 2008 21
While it is true that Dr. Boston was clearly in favour of onscreen marking, it was the QCA itself that decided on the basis of trials that were carried out in 2005 not to proceed. Indeed, in his statement about the onscreen marking trials conducted during the 2005 test cycle, it was Mr. David Gee [then Managing Director of the National Assessment Agency, a division of QCA] who said publicly, “Following analysis, I have concluded that it is inappropriate to introduce onscreen marking for either maths at Key Stage 2 or English at Key Stage 3 in 2006.”81
Dr Boston replied that:
It is wrong for the Secretary of State to claim that the QCA itself decided not to proceed with onscreen marking. We were told by ministers and officials that it was not an option. The final stumbling block was the Government requirement that whole scripts be returned to schools at the same time as the results, as in the past. …
The purpose of the letter signed by David Gee on 3 November 2005 was to resolve uncertainty for 2006, and to paper over what would otherwise have been a very public rift between the QCA and Government. With the benefit of hindsight, we negotiated for far too long: I should have stood our ground and put QCA advice publicly and in writing.82
55. What matters now is how the testing system is reformed in the future and what safeguards are necessary to ensure that testing failures do not recur. Lord Sutherland has made a series of recommendations in his report, including a recommendation that “online marking” should be piloted. He points to the potential of both enhanced quality of marking and greater security for scripts.83 In addition, we believe that on-screen marking is a viable way to capture electronically question-level data for diagnostic purposes. We noted in our Report on Testing and Assessment the benefits to teaching and learning of question-level data on test results, and would consider the routine provision of such data to be a valuable step forwards.84
56. We remain concerned about the consequences attached to high-stakes testing. We consider that, if children are to sit national tests, they should derive the maximum benefit from that experience. We noted in our Report on Testing and Assessment the benefit of receiving question-level data for diagnostic purposes and consider this to be a valuable step forwards. However, it seems to us that separating the marking process from data capture risks a repeat of the delivery problems experienced in 2008 and must surely lead to an increase in opportunity for human error. We believe that any significant future reform of National Curriculum testing must include an evaluation of on-screen marking from which question-level data can automatically be derived.
81 Q 360 82 Ev 74 83 Report of the Sutherland Inquiry, para 5.232 84 Children, Schools and Families Committee, Testing and Assessment, Third Report of Session 2007–08, HC 169-I, para 125
22 Policy and delivery: the National Curriculum tests delivery failure in 2008
5 Conclusion
57. In this Report, we have not sought to revisit the intricate details of how the delivery of National Curriculum tests in 2008 failed. Lord Sutherland has carried out a meticulous inquiry into those details and we endorse his report as an excellent account of events. Our inquiries, however, have taken a rather different approach and we have considered the delivery failure in the broader context of how Government has set up the framework for delivery of its testing policies.
58. A common theme has run through our inquiries: that of the independence of the public bodies involved in test delivery. We have concluded that, by interfering too much in the detail of policy delivery, the DCSF increased the likelihood of its failure. DCSF observers may have a legitimate role within QCA, but we consider that that role becomes illegitimate if observers seek to negotiate the terms of the “advice” issued by QCA to ministers, or to exert undue influence over QCA decision-making.
59. In terms of the independence of Ofqual, we believe that the presence of observers on its Board is totally unacceptable. Ofqual must be, and be seen to be, independent, otherwise the legitimacy of the testing system itself might be cast into doubt. This Committee will be monitoring the work of Ofqual on an ongoing basis and we will be taking a particular interest in how it guards and exercises its independence.
Policy and delivery: the National Curriculum tests delivery failure in 2008 23
Conclusions and recommendations
The inquiries into the 2008 test delivery failure 1. We endorse the work carried out by Lord Sutherland in exposing the detail of the many factors contributing to the failures in the delivery of the 2008 National Curriculum tests (Paragraph 6)
2. This inquiry provides a good illustration of how a select committee can use its authority and powers in a timely manner to investigate problematic incidents as soon as they arise. Our swift action in this matter has enabled us to put on the public record both written and oral evidence from a party to the incident, ETS, which was not available by the time the official inquiry was fully operational. The Sutherland Inquiry was then able to rely on the evidence we had secured when it became clear that ETS would not engage with the process. (Paragraph 8)
3. It is not clear to us that the DCSF having issued parallel terms of reference prevented Ofqual asking Lord Sutherland to inquire into the DCSF’s role, had Ofqual considered that appropriate. Lord Sutherland chose not to consider the Department’s role beyond its oversight of the particular circumstances of the delivery of the National Curriculum tests in 2008. However, this Committee is able to go further than he did and comment in more general terms on the line between policy and execution. (Paragraph 11)
Delegation of responsibility from DCSF to QCA 4. We maintain the view of our predecessor Committee, that the difficulty of establishing the correct dividing line between policy and execution should not be underestimated. However, we are concerned that DCSF appears to be specifying in considerable detail the ways in which it wishes to see its policies executed. The DCSF achieves this both formally, through the QCA’s remit letter, and informally, through regular contact with its agency and through the mechanism of observers. (Paragraph 24)
5. We agree with the Secretary of State that he is accountable for ensuring that the QCA fulfils its responsibilities for the overall operation of the testing regime. However, he is also accountable for the policies and other decisions of his Department, including the terms of the QCA’s remit letter; and he is responsible for ensuring that the directions issued by DCSF are capable of being carried out by the organisations tasked with their execution. (Paragraph 25)
6. If significant areas of Government policy are to continue to be delivered by non- departmental public bodies and other agencies, the leadership of those bodies should be prepared to demonstrate their professionalism by challenging the Government if they consider that the impossible is being asked of them. Appropriate mechanisms should be put in place to allow them to make such a challenge. We recognise that this should never be used as a means to hold the Government to ransom or to impede the
24 Policy and delivery: the National Curriculum tests delivery failure in 2008
execution of legitimate public policy. Nevertheless, there needs to be a formal and transparent dialogue between the Department and its agencies. (Paragraph 26)
7. In the light of the repeated problems we have identified, we consider that DCSF should, in an updated Memorandum of Understanding or equivalent with each public body, set out more clearly how it will conduct its relationships with those bodies. We urge the Government to adopt a far less prescriptive approach when issuing instructions to the new Qualifications and Curriculum Development Agency. If it is considered appropriate that policy should be executed by an agency rather than by the Department itself, the potential benefits of such an arrangement are significantly diluted if the Government fails to trust the expertise and experience of the professionals chosen to run such agencies. It is clear that micro-management from the Department would introduce a degree of confusion into the lines of responsibility and accountability, as previously happened in the case of Individual Learning Accounts, investigated by our predecessor Committee, and the mistake has been repeated with arrangements for the delivery of National Curriculum tests. (Paragraph 28)
QCA and DCSF observers 8. We have some serious concerns in relation to the role of observers and question whether, in many cases, they are necessary. In appropriate cases, departmental observers may perform a useful function in helping ministers remain accountable for policy delivery in a meaningful manner. However, formal advice to ministers from a public body should not be negotiated through departmental observers; nor should observers exert undue influence over the decision-making of a public body. (Paragraph 35)
9. We concur with Lord Sutherland’s recommendation, accepted by Government, that the role of departmental observers should be clarified. We believe that the Memorandum of Understanding or equivalent between DCSF and its non- departmental public bodies should, in each case, set out clearly the appropriate role for departmental observers so that the public can be reassured that observers are not acting inappropriately to influence the work and decisions of public bodies. (Paragraph 36)
Ofqual and DCSF observers 10. We expressed our concerns in relation to the potential for departmental observers to exert undue influence on the advice from and decision-making of public bodies. Whilst we are content that such observers should continue to have a role within bodies such as QCA with appropriate safeguards, we are opposed to the presence of departmental observers in the context of an independent regulator. The independence of Ofqual should be put beyond question and that requires contact between DCSF and Ofqual to be put on a much more formal footing than is suggested by the presence of observers on Ofqual’s Board. (Paragraph 43)
Policy and delivery: the National Curriculum tests delivery failure in 2008 25
National Curriculum test delivery post-ETS 11. In line with our recommendations in our Report on Testing and Assessment of May 2008, we welcomed the abolition of mandatory national testing at Key Stage 3 when it was announced in October 2008 and we remain convinced that the decision to reduce the burden of national testing was a good one. (Paragraph 47)
12. We repeat our warning, set out in our Report on Testing and Assessment, that new tests should not be introduced nationally with undue haste. We are concerned that, if single-level tests are rolled out as a replacement for end of Key Stage testing, there is potential for significant complexity in the logistical arrangements for sending out, marking and returning a large number of single-level test scripts twice each year. Evidence from the pilot studies so far is not encouraging and demonstrates that there remain serious problems to be overcome. We remind the Government that we are opposed to an increase in the burden of national testing generally; and we warn that the Government must be quite sure that the logistical arrangements for single-level tests are robust before they are rolled out nationally. (Paragraph 50)
13. We remain concerned about the consequences attached to high-stakes testing. We consider that, if children are to sit national tests, they should derive the maximum benefit from that experience. We noted in our Report on Testing and Assessment the benefit of receiving question-level data for diagnostic purposes and consider this to be a valuable step forwards. However, it seems to us that separating the marking process from data capture risks a repeat of the delivery problems experienced in 2008 and must surely lead to an increase in opportunity for human error. We believe that any significant future reform of National Curriculum testing must include an evaluation of on-screen marking from which question-level data can automatically be derived. (Paragraph 56)
26 Policy and delivery: the National Curriculum tests delivery failure in 2008
Formal Minutes
Wednesday 15 July 2009
Members present:
Mr Barry Sheerman, in the Chair
Annette Brooke Fiona Mactaggart Mr David Chaytor Mr Graham Stuart Paul Holmes Derek Twigg
Draft Report (Policy and delivery: the National Curriculum tests delivery failure in 2008), proposed by the Chairman, brought up and read.
Ordered, That the Chairman’s draft Report be read a second time, paragraph by paragraph.
Paragraphs 1 to 59 read and agreed to.
Summary agreed to.
Resolved, That the Report be the Sixth Report of the Committee to the House.
Ordered, That the Chairman make the Report to the House.
Ordered, That embargoed copies of the Report be made available, in accordance with the provisions of Standing Order No. 134.
Written evidence was ordered to be reported to the House for printing with the Report.
Written evidence was ordered to be reported to the House for placing in the Library and Parliamentary Archives.
[Adjourned till Monday 20 July at 3.30pm
Policy and delivery: the National Curriculum tests delivery failure in 2008 27
Witnesses
Monday 14 July 2008 Page
Dr Ken Boston AO, Chief Executive, Qualifications and Curriculum Authority Ev 1 (QCA).
Wednesday 10 September 2008
Dr Philip Tabbiner, Director and Chairman of the Supervisory Board, ETS Global; and Andrew Latham, Vice President, ETS Europe. Ev 9
Isabel Nisbet, Chief Executive, and Kathleen Tattersall OBE, Chair, Ofqual. Ev 21
Monday 26 January 2009
Lord Sutherland of Houndwood KT Ev 34
Wednesday 22 April 2009
Dr Ken Boston AO, former Chief Executive, Qualifications and Curriculum Ev 49 Authority.
Wednesday 20 May 2009
Rt Hon Ed Balls MP, Secretary of State for Children, Schools and Families; Rt Hon Jim Knight MP, Minister for Schools and Learners; and David Bell, Ev 61
Permanent Secretary, Department for Children, Schools and Families.
List of written evidence
1 Letter to the Chairman submitted by Dr Ken Boston, Chief Executive, Qualifications and Curriculum Authority Ev 7 2 ETS Ev 8 3 Ofqual Ev 31 4 Lord Sutherland Ev 47 5 Letter to the Chairman from Dr Ken Boston AO Ev 48: Ev 73 6 Letter to Lord Sutherland from the Rt Hon Jim Knight MP, Minister for Schools and Learners, Department for Children, Schools and Families (DCSF) Ev 77 7 Letter to the Chairman from the Rt Hon Jim Knight MP, Minister for Schools and Learners, DCSF Ev 78: Ev 79 8 Letter to the Rt Hon Jim Knight MO, Minister for Schools and Learners, DCSF, from Lord Sutherland Ev 78
28 Policy and delivery: the National Curriculum tests delivery failure in 2008
List of unprinted evidence
The following memoranda have been reported to the House, but to save printing costs they have not been printed and copies have been placed in the House of Commons Library, where they may be inspected by Members. Other copies are in the Parliamentary Archives, and are available to the public for inspection. Requests for inspection should be addressed to The Parliamentary Archives, Houses of Parliament, London SW1A 0PW (tel. 020 7219 3074). Opening hours are from 9.30 am to 5.00 pm on Mondays to Fridays.
Alan Bowles Dorothy Nelson Norma Powell Martin Fitzwilliam Joan Lynch John Atkinson Christina Evison Jenny Blackemore Susan Harr Stephen Mason Hugh Evans Association of School and College Leaders Tony Roberts, Lancashire National Association of Head Teachers Cherry Edwards Adrian Mitchell David Chaloner
Policy and delivery: the National Curriculum tests delivery failure in 2008 29
List of Reports from the Committee during the current Parliament
The reference number of the Government’s response to each Report is printed in brackets after the HC printing number.
Session 2008–09 First Report Public Expenditure HC 46 (HC 405) Second Report The Work of the Committee in 2007–08 HC 47 Third Report Looked-after Children HC 111-I and II (HC 787) Fourth Report National Curriculum HC 344-I and II (HC 645) Fifth Report Allegations Against School Staff HC 695 Sixth Report Policy and delivery: the National Curriculum tests HC 205 delivery failure in 2008
Session 2007–08 First Special Report Creative Partnerships and the Curriculum: HC 266 Government Response to the Eleventh Report from the Education and Skills Committee, Session 2006–07 Second Special Report Special Educational Needs: Assessment and Funding: HC 298 Government Response to the Tenth Report from the Education and Skills Committee, Session 2006–07 First Report Children and Young Persons Bill [Lords] HC 359 (HC 711) Second Report The Department for Children, Schools and Families HC 213 (HC 888) and the Children’s Plan Third Report Testing and Assessment HC 169-I and II (HC 1003) Fourth Report The Draft Apprenticeships Bill HC 1082 (HC 259 of Session 2008–09)
Processed: 17-07-2009 19:09:53 Page Layout: COENEW [SO] PPSysB Job: 406183 Unit: PAG1
Children, Schools and Families Committee: Evidence Ev 1 Oral evidence
Taken before the Children, Schools and Families Committee
on Monday 14 July 2008
Members present:
Mr Barry Sheerman, in the Chair
Annette Brooke Fiona Mactaggart Mr Douglas Carswell Mr Andy Slaughter Mr David Chaytor Mr Graham Stuart Paul Holmes Lynda Waltho
Witness: Dr Ken Boston AO, Chief Executive, Qualifications and Curriculum Authority (QCA), gave evidence.
Q1 Chairman: May I welcome Dr Ken Boston, chief schools and markers, establishing emergency executive of the Qualifications and Curriculum marking centres, supporting call centres, managing Authority, to our deliberations. We are glad he could and recruiting the data entry senders and supporting appear before us at short notice. He knows only too the management of scripts. In other words, we have well why we want to speak to him. I have said to Dr pushed to the absolute limit our capacity under the Boston that he will have an opportunity to say a few contract to advise, support and warn. The current words to get us started, but Dr Boston, the reason position is that in Key Stage 2, the marking is now why we have called you before the Committee is that 100% complete.1 The results will be published on the you will know better than most people all the stories website tomorrow to download and print. Data that have been running around in the education entry has been slower than marking. The volume of sector and outside the education sector about the results entered as of midnight last night was English delay in the key stage test results. It was quite hard 94.4%, maths 97.3% and science 97.3%. The Key to find out by this Monday morning what was truth Stage 3 maths and science are substantially and what was fiction, and you are the man to tell us, complete, but there is still marking to be done in so that is what we are here for, but do please make English. The majority of results will be posted on an opening statement. Friday. The schools that do not receive their English Dr Boston: Thanks, Chairman. Let me begin by results then will have them posted as they become saying I make it clear to this Committee as I have to available. I assure the Committee that we are pupils, schools, parents and Ministers that I exploring all possible commercial and legal avenues apologise for the failure of our contractor to have to ensure that suitable action is taken for the failure. the Key Stage test results available on 8 July and I We have a recovery plan in place, and we need to share their frustration at this quite unacceptable attend urgently to the issue of ensuring that the outcome. Lord Sutherland will establish the causes system will deliver in 2009. of the failure, which was foreshadowed by a number of problems in delivery. Among them were problems Q2 Chairman: What will be the latest date for all the with marker recruitment and retention, markers results to be out? being given wrong information about the location of Dr Boston: It is diYcult to be entirely precise on that. their training and the time of it, delay in getting We have predicted dates before and got it wrong. I think that it could be well into the holidays before papers to markers, unmarked scripts being returned some of those English results are available. to schools, data entry requirements impacting on the rate of marking, inadequate call centre capacity, and Q3 Chairman: That would be mainly the English slow and unpredictable data feeds. I want to assure results? you that we have responded immediately to every Dr Boston: Yes. At the moment we have maths and one of those signals and have done everything science substantially complete. We think that they possible operationally and within the terms of the will be out on Friday. contract to ensure that Educational Testing Service (ETS) would deliver to schools. Since the start of the Q4 Chairman: So this is a mess, but the results will test cycle in September last year, we have met weekly come out. Will they come out reliably and to a with ETS and, since 1 May, daily with ETS to review standard that you can swear by? risks. We have held ETS accountable for meeting Dr Boston: I do not believe that there is any reason service levels under the contract, and we have at the moment to have doubts about the quality of monitored it closely. We have escalated problems to the marking, despite the stories and fears that are the highest levels of ETS, including bringing the vice abroad. The people who have marked the papers are president to London. We have co-located 70 of our staV to ETS oYces. They have been providing 1 See supplementary evidence submitted by Dr Ken Boston; technical support, developing communications with Ev 7 Processed: 17-07-2009 19:09:53 Page Layout: COENEW [E] PPSysB Job: 406183 Unit: PAG1
Ev 2 Children, Schools and Families Committee: Evidence
14 July 2008 Dr Ken Boston the people who have always marked the papers. The regulations. It was reviewed by the OYce of papers have been developed in England by people Government Commerce on two occasions and got who know the English curriculum, English “green”, which is the highest rating for the process. standards and English schools. The mark schemes This company was selected with the highest rating have been developed by those people. The people on the various criteria that were identified. This is a who are marking this year are largely the same as very big company with a lot of experience. It passed those who marked last year—teachers and retired financial due diligence, which was conducted for us teachers. The company which failed in this contract by PricewaterhouseCoopers. References were has been responsible for logistics. The training of the checked out for it with organisations such as the markers has been done by team leaders and senior California State Board of Education, with which it experienced markers, who are also people who have had done, and continues to do, major assessment done the job before. The programme that was put in programmes. This company has a very good track place by ETS to benchmark quality this year is record. Its failure here is something that we certainly superior to the one used in the past. In the past, self- did not predict. selected scripts were examined by senior markers during the run-up to the start of marking. If the senior marker ticked them oV, the person began Q7 Lynda Waltho: I am wondering about the IT. We marking. There were some more self-selected scripts have significant information to suggest that changes given to senior markers once the marking was two- to IT, in terms of marking, were introduced. A thirds through. With this current contract, we system such as this was surely piloted. Was there no requested that great attention be given to improved indication that there could be problems? quality. There is a data bank of 80 scripts. They are live scripts that have been standardised by markers. Dr Boston: Certainly, the system was piloted At five stages during the marking process, markers extensively after the contract was awarded. During are required to access a batch of four and mark them. the awarding of the contract there was a competitive That is a common standard for all markers. If they dialogue process, whereby the specifications for the have drifted from the mark they are brought back IT and other things were refined in discussion with into line by their team leaders or senior examiners. If the people who were technically bidding for it. they cannot be brought back to the mark, they are However, despite the pilot, since the system was taken oV quickly. The quality of training has been scaled up there have been some significant problems. very sound and the quality assurance—the Marking has been going faster than it is possible to benchmark processing—is superior to what we have process the data and set levels against them. There had before. have been issues that we need to get to the bottom of, and that clearly is a matter that Lord Sutherland and we, from the legal point of view, will be pursuing Q5 Chairman: Who supervised all that quality? Dr Boston: The training was done by the senior closely. examiners, who are experienced English markers, and the marks for those benchmarks were determined by those examiners. The logistics, the Q8 Lynda Waltho: Do you think that the IT system delivery, the distribution of papers, the gathering of may have been overly ambitious? results, the data feeds, were part of the contract. The Dr Boston: I think the problem, frankly, was that it actual detail and quality of the work was not. The was not ambitious enough. The decisions about how final thing is that if there are problems with quality, marking would be done this year were made prior to they will be dealt with this year just as they have been the procurement. Decisions about what data would dealt with in the past. In English, particularly at Key be sought and how they would be presented were Stage 3, we know that a higher proportion of papers decisions of Government and were also made before are returned for review than in other subjects, the procurement was made, in 2006. We have come because English is a more diYcult paper to mark— a long way in recent years in the general the mark scheme is not as precise, and the capacity qualifications, the use of technology, and the use of for variation between markers is greater. Those on-screen marking, which is rapid, high quality, papers will be marked in exactly the same way. If reliable and secure. Over half the GCSEs at the there is a quality problem, it is most unlikely to be moment are marked on screen. Many of the A-levels greater than in previous years, but if there is one, it are marked on screen. None of the key stage tests are will be dealt with by the usual process. yet marked on screen. Although the paper is marked manually, we have asked the marker then to go to a Q6 Chairman: Whatever you say, people will blame computer and key in on computer the item-level the Qualifications and Curriculum Authority, marks—the question-level marks—which has because you hired those people who have failed. Is slowed the process, to a degree and, to a degree, that not what they will say? introduced the capacity for error. Next year one of Dr Boston: Certainly, the QCA conducted the the things we have to address is resolving that issue procurement process at the end of the previous of slowness, but it is very clear that in the immediate period. The procurement had to be done again term we need to move as quickly as possible to using according to the new rules. The process took place on-screen marking for key stage tests: fast, reliable, over a period of almost 12 months. It followed, at accurate, secure—all the benefits we have every step of the way, the requirements and demonstrated at the general qualifications level. Processed: 17-07-2009 19:09:53 Page Layout: COENEW [O] PPSysB Job: 406183 Unit: PAG1
Children, Schools and Families Committee: Evidence Ev 3
14 July 2008 Dr Ken Boston
Q9 Mr Carswell: There has been a cock-up. It has is inevitably going to be incompetent government; aVected tens of thousands of people. It has that we should not have a quango running undermined confidence in national testing and the national testing? reliability of the end product. Is it your fault? Should Dr Boston: I would have thought that you would the Minister take responsibility, or neither? Is the rather have a body that is at arm’s distance from quango chief yet again going to pass the buck? government running testing, than government Dr Boston: The issue is complex. There have been, themselves. I think one of the issues with the QCA potentially, problems in decisions made a couple of has been that there are certain functions it performs years ago; there have been, potentially, problems in that need to be further distant from government. the QCA and in its division, the National That is why I welcome the establishment of Ofqual, Assessment Agency. There have been, potentially, which is the regulatory side. I make it absolutely significant problems in ETS. Lord Sutherland has clear to you that in my time as Chief Executive of the been appointed to sort that out, and I believe he will. QCA I have never had any interference from any Minister or any government oYcial to do with the Q10 Mr Carswell: Can we see a copy of your nature of the paper, the quality of the marking, the contract, please? standard that should be set, or the figure that should Dr Boston: I do not have my contract with me. be reached in terms of performance, and if I had I would be the first person to declare that publicly. Q11 Mr Carswell: Could we see it? Dr Boston: Chairman, is that a reasonable Q16 Chairman: Before I move on to Annette, a requirement for this Committee? couple of times you mentioned the Sutherland Chairman: Members of the Committee have the inquiry. I take it that you are not holding any ability to ask you questions, Dr Boston. knowledge back from the Committee that you are Mr Carswell: And I believe the Clerk of the saving for a future occasion. Committee contacted your oYce beforehand to ask Dr Boston: Absolutely not. for a copy. Chairman: Just as long as that is clear. Dr Boston: Well, I am not aware of that contact, but Dr Boston: I am in a position where I must be my remuneration and so on is in the public record. cautious about what is said with regard to putting the commercial and legal side of the issue at risk. As I implied, we are in a strong position contractually, Q12 Mr Carswell: Sure, but the terms of the contract and we will seek remedies as a consequence of the that would allow this Committee to hold you to failure. I am holding nothing back from the account for how you do your job—can we see a Committee. copy, please? Dr Boston: I am prepared to make available a copy of the contract.2 Q17 Annette Brooke: Could you tell us more about the late recruitment of markers? In your first statement you told us that people are employed who Q13 Mr Carswell: Could we see that after this always mark the papers. Why have people been meeting? receiving phone calls until quite recently asking them Dr Boston: Not immediately after this meeting; I do to go and mark? not happen to carry it around with me. Dr Boston: I understand that there have been shortages in some subjects. They were partly caused Q14 Mr Carswell: Just as well, perhaps. Mr Brown by people who had previously accepted positions has rightly said that your successor needs to be dropping out and deciding not to continue, and appointed only after confirmation hearings by this therefore contact was made with other markers. Committee. Given what has happened and given Similarly, as part of the support that I mentioned, what some people would say is the need for proper when we found in May and June that diYculties accountability of the QCA and its head, would you were emerging that could put the system at risk and welcome that? Do you think it is right that your that needed addressing, we strongly advised ETS to successor should be appointed only after set up two marking centres and it agreed. We needed confirmation hearings, and perhaps that this to recruit additional people into those marking Committee should have the power to issue P45s centres. Subsequently, there are seven such centres when the QCA does not quite get things right? established around the country. We have recruited Dr Boston: I have not given that issue any thought at people to go into those, and they also distribute all. I welcomed the decision to have this Committee papers to people who mark at home. In an eVort to have confirmation hearings, but I cannot say it is an avoid the problem that has arisen, we put an issue that has exercised me. enormous eVort into getting more markers in, to concentrate and improve the rate of marking so that Q15 Mr Carswell: One final thing, if I may. I believe the deadlines had a chance of being met. That has you gave us evidence saying that you are a fan of the not been suYcient. idea of national testing. Does not this whole episode suggest that the idea of state-run national testing is Q18 Annette Brooke: You told us that the quality of wrong—that government by unaccountable quango training was sound and as per usual. How can those late recruits have had the full training that they 2 Supplied in confidence. Not printed. would normally have had? Processed: 17-07-2009 19:09:53 Page Layout: COENEW [E] PPSysB Job: 406183 Unit: PAG1
Ev 4 Children, Schools and Families Committee: Evidence
14 July 2008 Dr Ken Boston
Dr Boston: They do not start without the full alternative. In fact, that last time that we had a fully training and without achieving a standardisation successful round with key stage tests was in 2003. In process. They are given no live scripts until they have 2004, we were late with Key Stage 3, which was a been provided with some training. problem. In 2005, 2006 and 2007, we delivered because we kept Key Stage 3 English late, in mid- Q19 Annette Brooke: Some training. Is that full August. This year, we tried to bring back Key Stage training? 3, including English, to July and the result was that Dr Boston: Yes—full training. we failed not only Key Stage 3, but Key Stage 2. We really need to look at how the tests are delivered and Q20 Annette Brooke: Could you tell us where the administered, and we need rapid reform. shortages have been? Have they been at particular key stages or subjects, or right across the board? Q24 Mr Chaytor: To what extent is the underlying Dr Boston: It has been pretty well across the board. issue not only this year’s delays, but the problem The most acute problem has been in English. that has occurred in recent years that you have described of the sheer volume of assessment that Q21 Annette Brooke: Why? children in English schools experience? Is online Dr Boston: I think that people find marking English marking the solution, or will it merely gloss over a more challenging than marking subjects where the deeper problem caused by the volume of assessment mark scheme has perhaps less room for discretion that takes place? and interpretation. We have diYculty getting Dr Boston: I have given evidence on that to the markers. Committee before, and the alternative to which I referred at one point was single-level tests to confirm Q22 Annette Brooke: Are they paid enough? teacher assessment. In the longer term, that seems to Dr Boston: The rates vary from subject to subject, be a productive direction in which to go. The but most markers would make £750 to £1,000 or teachers come to a judgment about the level that a £2,000 from a marking season. People keep coming child is at; at the moment, £150 million is being spent back to do it. It may be that there is a need to through the strategies in training teachers in the increase payment. The greater benefit is to make the assessment of pupil performance. To then use the marking experience—I was going to say more single-level test to confirm whether the child has, in enjoyable, but perhaps I should say more fact, reached that level of performance seems a professionally rewarding. That will come when more sensible way forward. The alternative, which is to and more markers work on-screen, because we do continue with full cohort testing, is perfectly feasible, not seem to have the same problems with the general even given the short time frames, and even given that qualifications in attracting markets as we do with the we are only talking of a matter of nine weeks for Key key stage tests. Stage 3, but to do it, we have to use technology.
Q23 Mr Chaytor: Dr Boston, when you first came Q25 Mr Chaytor: You referred earlier to the weekly before the Committee some years ago, you famously meetings. The problem was identified publicly described the English assessment system as a cottage during the week beginning 30 June. For how long industry. To what extent have the characteristics of had the weekly meetings taken place before then? that cottage industry contributed to the problems Dr Boston: Since September there has been a over the past few weeks? fortnightly programme board meeting, chaired by Dr Boston: In the years since I made that comment, the managing director of the NAA and attended by we have seen tremendous change in a whole host of the very senior people from ETS. There has been a things in respect of general qualifications, including weekly meeting since September—it is run by the better administration in the exams oYces at schools. programme director in NAA, who reports to the No longer in any of the tests or examinations are managing director—attended by opposite numbers bundles of scripts left on doorsteps or at post oYce in ETS and in other agencies that are involved; we counters unaccounted for; they are barcoded, traced have other contractors apart from ETS, setting the and tracked. We know the whereabouts of parcels. papers and so on. As I said, from 1 May we have met However, we have not advanced with the marking daily with senior people in the organisation, process of the key stage tests beyond manual operationally, to look at risk across the board in marking. We now have a situation in which people communications, help desk, delivery, training, mark manually and then enter the scripts on market recruitment and retention, and so on. It has computer. The standard scripts that are done in five been very hands on. batches come on computer. The technology is there. There is only a sample of 40,000 of single-level tests Q26 Mr Chaytor: So if the weekly meetings have that we are trialling in December this year, but that been taking place since September, at some point will be safe and secure on-screen delivery and on- well before 1 May it must have become obvious that screen marking. We must move quickly into the use the programme was in diYculties. Did something of that technology with the tests. It will not be next happen on 29 June? year. We will not be able to scale up the key stage Dr Boston: It was always obvious that it was a very tests to on-screen marking for next year. We cannot challenging programme. Our concern was to identify have a repeat of what happened this year. We shall the risks and mitigate them right through that either need more time next year or to find some process, which we were doing. I make the point, as I Processed: 17-07-2009 19:09:53 Page Layout: COENEW [O] PPSysB Job: 406183 Unit: PAG1
Children, Schools and Families Committee: Evidence Ev 5
14 July 2008 Dr Ken Boston did in my introduction, that we were not there decision to make. However, when we look now at the actually managing the thing. Our capacity is to late results ahead of us, something has clearly gone advise, support and warn. ETS has a contractual wrong. It is clear that we not only need to deal with obligation to inform us when it has reached an the problem confronting us at the moment, but assessment that it is not going to meet a key ensure that it never happens again, and get this milestone, and 8 July was a key milestone. We running for next year. The clock is ticking, and the challenged ETS on many occasions throughout— development of these tests is a two and a quarter year particularly the past two months—on that issue. It cycle. The development of the tests for 2009 is well was only on 26 June that we got a formal, written under way, which is one reason why the transfer response from ETS, that it was not going to make it. online would be very diYcult at this stage. We subsequently directed ETS, on 27 June, that it was required to make it. Q30 Mr Stuart: It seems to have come as a surprise to you that ETS failed, yet the whole examinations Q27 Mr Chaytor: The question is, if the NAA community has been saying for months that the representatives had been meeting ETS since last contract was underpriced and that diYculties were September, why did the NAA not report to you that expected. How come you were the only people who there was a serious problem here? Why was there no did not seem to anticipate the outcome? regular performance monitoring from ETS? Dr Boston: I think we were more closely aware of the Dr Boston: There was regular performance problems that were emerging than the rest of the monitoring right through that, identifying problems community. We were dealing with them. We were in and mitigating them. We picked the problems oV one there working weekly and then daily with ETS— by one. Certainly there was an enormous number of issues, but some of them were small and all of them Q31 Mr Stuart: Do you think it is acceptable to were addressed. When we found, as we got into the notify Ministers as late as they were notified of the latter weeks, that the volume of calls that were going fact that it would not be delivered? to come to call centres was greatly underestimated, Dr Boston: It became clear, as I said, on the 26th, we immediately stepped in. We set up another call when contractually ETS had to tell us formally centre, which we funded through the NAA, to assist whether they were going to make it or not. It was ETS. We put additional resources into the call clear at that point that they were not. In recent weeks centres at Watford and in Northern Ireland, to assist up till then, we had had doubts about Key Stage 3 ETS to handle calls. We stepped in and found a English in particular, but we were doing everything backlog of 10,000 e-mails, which were responded to, that we could to ameliorate them and were finally, by NAA people rolling up their sleeves and reasonably certain that we would assist ETS to getting on with it. overcome those problems.
Q28 Chairman: So this was 10,000 e-mails to ETS Q32 Mr Stuart: So when did you, as the head of the that were not dealt with, and the NAA staV—the QCA, first think that the deadline was not going to staV of your wholly owned subsidiary—had to pick be met? that up? Dr Boston: When I was formally told by ETS on 26 Dr Boston: We stepped in and made it happen. June. But, as I said before, our folk had repeatedly challenged and pushed in the weeks before that, saying, “Are you really going to make it?” The Q29 Chairman: So was the wrong decision made constant assertion from ETS was, “Yes.” From our about the contractor, in retrospect? I know you have examination, the remedies that we were putting in all sorts of diYculties. The word on the street is that place were working. The 10,000 e-mails, for you did have a UK-based supplier which had much example, were solved. The call centre problem was more grasp of the technology, but you went for a solved. Enough markers were brought together to much cheaper option. do the job. Dr Boston: No, it was not a cheaper option—well, it was the lowest cost option, but it was not picked on those grounds. There were, finally, three bidders, Q33 Mr Stuart: Did the previous board bid for the from a field that started at six and went quickly back contract? to five. On the key criteria of capacity to do the job, Dr Boston: The previous board? this company was clearly up there with any of them. On understanding and delivery of the contract, and Q34 Mr Stuart: The board that had previously been on innovation in technology, it had a high doing these examinations. Did it bid? reputation. It ticked all the boxes and, in aggregate, Dr Boston: Yes, it did. came out well ahead of any competitor. On that basis, following that process, we made the right Q35 Mr Stuart: That was Edexcel, wasn’t it? decision. It was subject to all the public scrutiny Dr Boston: Yes. required, including very close scrutiny by the QCA board. Observers from the Department were present Q36 Mr Stuart: And did it bid? right through the procurement process and were able Dr Boston: Yes. to advise Ministers that our recommendation should be endorsed. Of course, the decision was our board’s Q37 Mr Stuart: Why did so many of the boards recommendation. It was clearly, on its merits, the not bid? Processed: 17-07-2009 19:09:53 Page Layout: COENEW [E] PPSysB Job: 406183 Unit: PAG1
Ev 6 Children, Schools and Families Committee: Evidence
14 July 2008 Dr Ken Boston
Dr Boston: I really cannot speculate on that. They letting a contract, we have had five major active may not have thought that they had the capacity, players. We certainly did not when the previous or— contract was let.
Q38 Mr Stuart: Dr Boston, how can you not Q44 Mr Stuart: Is it true that the parameters of speculate? They wrote to you and told you why they consistency, which I understand is a key requirement were not bidding, did they not? for testers, were relaxed to allow ETS to recruit the Dr Boston: Why they didn’t bid? examiners it required? Mr Stuart: They wrote to you and told you why they Dr Boston: No. were not bidding. Dr Boston: I’m not sure what you are referring to. Q45 Paul Holmes: In your opening comments, you observed that there had not been a satisfactory round of testing in SATs from 2003 onwards, so Q39 Mr Stuart: I am referring to letters sent to you against that background, when you were bringing in by other boards explaining why major boards in this a brand-new contractor, why did you also bring country did not bid for the contract. They wrote to forward the date of the SATs to July? Is that not and explained precisely why they were not bidding heaping the bonfire? for the contract. Dr Boston: We believed we had a better solution than Dr Boston: Well, I’m sorry, I— we had had previously and that that date would be met. In the competitive dialogue process, we talked Q40 Mr Stuart: Is that not true? through with all three final bidders the requirement Dr Boston: I do not have that information at my that in the first year it was still manual marking, but fingertips. If those letters were sent, presumably that with electronic marker capture. Everyone realised was back in 2006. I am afraid that I would need to go that that potentially increased the strain over online back to my files to confirm whether that happened. marking, but those who went through to the final bid convinced us that they could do it and had solutions to do it. Q41 Mr Stuart: But that is the source of it, and the truth is that many of the largest boards did not bid for the contract, because in the belief of many of the Q46 Paul Holmes: Obviously, as we have heard, most experienced and capable boards in this country, there is some concern about the role of the QCA, the it was badly drawn up and badly briefed. They did contract it put out and so on, but assuming that the not bid, and you therefore ended up with ETS, which inquiry finds that ETS has a lot to answer for—you you said had complete capacity to deliver, and it were very critical of ETS in your opening turns out that it had anything but. comments—what are the penalty clauses on ETS? I Dr Boston: We ended up with five very solid bidders, asked the Minister that a few days ago, and he was which went through the process. Three were selected suitably vague. You put 70 staV in to help sort it out. from the five, all of which were still in the ring. Three You set up an extra call centre. You answered 10,000 were selected from the five for the last part of the e-mails. Surely all that money immediately comes process. We had five very active— back, but what are the penalty clauses beyond that? Dr Boston: There are very significant penalty clauses and service credits, as they are called, where money Q42 Mr Stuart: Is it not disappointing that major is paid back to us for services that are not delivered. boards did not bid for this contract, and is that not I want to be very cautious here, because there are because they believed that it was poorly drawn up, various options, one of which is termination for it over-specified what was to be done and it did not failure to deliver, and that has very, very significant recognise their professionalism, and that failure at financial consequences. We are not at that point. We the original contract or briefing stage may well have are at the point where we are considering how to deal contributed to the current fiasco? with the current situation and how to move ahead on Dr Boston: I am not able to comment on that a more secure basis next year. As I said, we are in a without going back and looking at my files and strong position with the contract, and large sums of correspondence from several years ago, but we were money are potentially at risk here. very confident that we had contributed to building a better market for this than had existed previously. Q47 Paul Holmes: When I was first elected in 2001, one of the first inquiries I took part in with this Q43 Mr Stuart: So was that the thinking? Was it Committee or its predecessor related to the ILA desired to bring in new external contractors? Was it scandal. Over the years, it seems that large the desire to bring ETS from the United States companies such as Capita, EDS and, in this case, because you thought it could do a better job, bring ETS mess up on delivery of programmes such as better technology and raise testing standards? ILAs and programmes for the Department for Work Dr Boston: No, it was simply that the contract that and Pensions, but— we had with the previous organisation had finished Chairman: Individual Learning Accounts. and we were obliged to call for tenders again. We Paul Holmes: Yes. But nothing seems to happen. The were very anxious to ensure that there was a genuine argument that I have heard—and your partner over market, that there was real competition. I do not at ETS said it—is that ETS is one of the largest believe we have ever had a position before where, in players in the field. Capita is the largest player in the Processed: 17-07-2009 19:09:53 Page Layout: COENEW [O] PPSysB Job: 406183 Unit: PAG1
Children, Schools and Families Committee: Evidence Ev 7
14 July 2008 Dr Ken Boston
field. There are not many firms that you can Dr Boston: Yes, there will be, as in all years, a approach for such contracts, so do they have the proportion of them that require review. That is a Government over a barrel? Even though they mess normal element of the process. They will be dealt up on diVerent schemes, you cannot penalise them with. All the evidence that we have at the moment is that much or cut them out of the market, because that the marking is as secure now as it has ever been they are the only players out there. in previous years. Dr Boston: I think there is a risk of severe reputational damage in relation to a failure at this Q50 Chairman: You said in so many words to this level. Any large company is concerned with that. Committee that you blame not the Secretary of That probably is as significant as the financial State, yourselves, the NAA or Ofqual, but the private sector companies’ shortcomings. That is penalties, which could run into the tens of millions. what you have told this Committee. International companies obviously seek to operate Dr Boston: Well, if that is the impression, let me with footprints in various regions of the world. The make it clear that there are three parties to this experience at this stage in England is clearly arrangement. There is the Government who in unsatisfactory. 2006—not in 2008—determined what they wanted from the tests and how they wanted the tests conducted. We are commissioned to deliver that; we Q48 Paul Holmes: Last question. If there were do it through contractors. We, too, may well have significant financial penalties, would they be at a made mistakes—done things that have caused level to hurt the company? We have heard that some problems or not done things that might have companies write into the contract a penalty clause, avoided problems. When I say “we” I mean the QCA knowing that they cannot meet all the targets that and the NAA, which are a part. ETS may also have they promised. They build a penalty clause into the made some very significant mistakes. It is the body profit margin, so that they still make a profit, despite that failed to meet the deadline that it was the penalty. contractually committed to meet, and that it was Dr Boston: I do not believe that this contract is paid to meet. I believe that two things will happen costed on that basis. We went through it line by line now. Lord Sutherland’s review will be a very as part of the procurement process, arguing the thorough examination of those three components. detail of every piece of work that was contracted. I In the dealings between ourselves and ETS, there do not think there is fat in this system that would may well be litigation that runs alongside. There is allow for that sort of penalty and for profits still to more about the cause of this to be found than I am able to account to you in this session. be made. Q51 Chairman: Douglas Carswell has asked me about the contract. How soon can the Committee Q49 Chairman: Dr Boston, we are coming to the end see it? of the session, and another one is following this. Dr Boston: I could have a copy couriered across to With your hand on your heart, can you tell us as a you in an hour. Committee that, albeit with the delay, these results Chairman: Thank you. Dr Boston. Thank you for will come out and will be of a standard of which you appearing at such short notice before the can justifiably be proud to show to parents and Committee. students? Dr Boston: Thank you.
Letter to the Chairman submitted by Dr Ken Boston, Chief Executive, Qualifications and Curriculum Authority (QCA) In my evidence to the Select Committee yesterday I said that marking was 100% complete in all Key Stage 2 subjects. No further Key Stage 2 scripts were held by ETS. It has since come to light that a small number of wrongly assigned unmarked scripts had not been collected by ETS from markers to whom they had been sent. ETS has been directed to retrieve the scripts immediately and to mark them urgently. Today we have identified 384 Key Stage 2 unmarked scripts that have since been logged with marking panels for completion. This material represents less than 0.02% of the 1.7 million Key Stage 2 scripts for the National Curriculum tests 2008 cycle. While in percentage terms this is small, I do not under-estimate the impact even this small number has on schools, pupils and parents. We have called the schools that contacted us to explain the situation. I regret that the information I gave to the Committee did not take account of this small number of scripts remaining in some schools. July 2008 Processed: 17-07-2009 19:04:48 Page Layout: COENEW [SE] PPSysB Job: 406183 Unit: PAG2
Ev 8 Children, Schools and Families Committee: Evidence
Wednesday 10 September 2008
Members present:
Mr Barry Sheerman, in the Chair
Mr Douglas Carswell Mr Andrew Pelling Mr David Chaytor Mr Andy Slaughter Paul Holmes Lynda Waltho Fiona Mactaggart
Memorandum submitted by ETS
ExecutiveSummary There is no question that ETS experienced some operational and technical issues that aggravated the process of marking this year’s national curriculum tests in England early on—and for those we take full responsibility. These issues were exacerbated by program changes required by NAA, long delays by NAA on key project decisions and layering on of additional project deliverables. Despite all these challenges, marking quality was maintained.
Introduction andOverview ETS welcomes the opportunity to appear before the Children, Schools and Families Select Committee. Our most important stakeholders are the millions of English pupils, teachers, schools and parents who depended on ETS and QCA to deliver a well-run testing programme. They have been let down, and we take this opportunity to repeat our apology to them. I am a director and the chairman of the Supervisory Board of ETS Europe, the ETS entity responsible for delivery of the 2008 national assessment tests. In May 2008, I was given a mandate from ETS President Kurt Landgraf to investigate ETS and NAA issues aVecting the safe delivery of the testing program. ETS desires to provide full cooperation to this Committee and to Lord Sutherland’s independent inquiry. We remain bound by confidentiality obligations under the original contract with QCA and under the August settlement agreement. We have requested but not yet received permission from QCA to provide full information to the Sutherland inquiry and the documentary evidence requested by this Committee. We are hopeful that both this Committee and Lord Sutherland will be able to prevail upon QCA to give its consent. For the same reasons, we have made limited comment in the public domain. You should not take our silence as agreement with what has been said about us, but as our commitment to our contractual obligations. ETS is a non-profit organisation that administers more than 50 million exams to exceptionally high standards in all of the 180 countries we operate in. We bid this contract because helping pupils learn and teachers teach is our mission, not profit. We took it because we believed our expertise could improve educational measurement in England. We invented large-scale standardised assessment 60 years ago, we pioneered computer-based testing, we originated online marking and created the largest Internet-based testing network in the world, and in all that time, we never asked for early release from a standardised achievement contract. We worked closely with the NAA throughout the project and, whilst we have not achieved everything we should have, together we have made real progress in the quality of the marking and the detailed database of results provided to schools and students.
Quality ofMarking We are aware that the Committee is particularly interested in quality and we would like to dispel questions over this year’s marking. We can confirm that the quality of markers in 2008 was in fact even higher than in previous years. We introduced a more rigorous method of certifying markers to ensure adherence to the marking scheme and constantly monitored quality during marking. Early on we eliminated hundreds of markers who could not meet the required NAA standards. The training that we oVered was delivered by the same senior markers as in previous years. We also used the same pool of teachers and retired teachers as in previous years. We used the same criteria for screening new applicants as in previous years, and in addition all markers were recruited according to NAA guidelines. The ongoing marker reviews are being managed by the NAA and are being carried out by the same markers we recruited to mark the tests originally. Processed: 17-07-2009 19:04:48 Page Layout: COENEW [O] PPSysB Job: 406183 Unit: PAG2
Children, Schools and Families Committee: Evidence Ev 9
SharedResponsibility with the NAA There is no question that ETS experienced some operational and technical diYculties that hindered our ability to deliver test results on time and we have not shied away from taking responsibilities for these. For example, some school allocations were split, which meant that a school was given to two separate markers. In these cases one of the markers would be unable to view their allocation online. There were also instances in which scripts were wrongly allocated, so one marker would be able to view a school online, for which they did not have scripts, and another marker would receive scripts they could not view online so were unable to enter marks online. At the same time, NAA also shares significant responsibility for the delivery failure. Through a combination of the NAA making changes to the contract—against our advice, delaying critical decision- making and layering on additional responsibilities, we ended up with a much more complex and challenging task. Thus, I cannot point to just one or two things that contributed to the marking not being completed on schedule. It was, in fact, that the cumulative interaction of ETS and NAA created a compounding eVect. For example, the solution we presented in the bidding process and the contract we signed called for training about 5,000 experienced markers online instead of face-to-face. The online training was one of the innovations we were led to believe was pivotal in our being selected. In March 2008, just two months before marking was to begin, the NAA mandated that we should revert to face-to-face training for all 10,000 markers requiring us to find meeting venues, print and ship materials to those venues, co-ordinate marker invitations, travel schedules, costly overnight accommodations—all at the last moment. Not only did this specifically impact future delivery milestones, but it also prohibited markers from going online and accessing training materials early. This caused frustrations for markers because they had to understand how to use the entire system in a shortened time period. Additionally, we were not supplied with critical information on operational failures experienced by previous suppliers that could have informed our decisions. For example: — We were not made aware that recruitment of Key Stage 3 English markers has been a historical problem and that we would face diYculties identifying markers. — When the NAA set milestones, it indicated that the previous supplier had accomplished similar deadlines when in fact this was not the case. For example we were told 100% of results had been returned on time, when, as this Committee well knows, suppliers had historically had problems achieving this. Our view is that the NAA changes to the agreed program, the long delays by NAA in reaching decisions and the layering on by NAA of additional tasks, combined with ETS operational and technical issues, and compounded each other and we believe this lies at the heart of the delivery issues.
Conclusion As I stated before, we accept responsibility for all of ETS’s operational and technical issues that aVected the experience of markers and the return of results to schools. And, not withstanding the challenges presented by the NAA, I honestly believe we introduced more quality improvement measures into the assessment than ever before with the result that students, parents and schools got good quality scores. Once more I reiterate my apology and welcome this opportunity to answer your questions. Dr Philip S Tabbiner Director and Chairman of the Supervisory Board September 2008
Witnesses: Dr Philip Tabbiner, Director and Chairman of the Supervisory Board, ETS Global and Andrew Latham, Vice President, ETS Europe, gave evidence.
Q52 Chairman: May I welcome Dr Tabbiner and session would have been inappropriate. Also, we Andrew Latham to our proceedings? We are very have had Ministers and the Qualifications and grateful that you are able to appear before the Curriculum Authority in front of the Committee, Committee at this time of year. A certain Clerk who but we have not had the other side of the discussion, has an association with the Committee said that if so this is a chance to show that the Committee this is a precedent, it might be a dangerous one— believes in fair balance. Dr Tabbiner, on that note I under my chairmanship, this is the first time that we am going to ask you to say a few words about what have met in the recess. There are two reasons why we went wrong. are meeting today. First, this is a very important Dr Tabbiner: ETS welcomes the opportunity to issue. What happened in July disappointed many appear before the Children, Schools and Families parents and students, and many people feel very Select Committee. The reason we have not strongly that we should run an eYcient and eVective commented in detail in the public domain on this testing and assessment system. It seemed to us that year’s national curriculum tests is our to wait until October or November to have this confidentiality agreement with the QCA. It is Processed: 17-07-2009 19:04:48 Page Layout: COENEW [E] PPSysB Job: 406183 Unit: PAG2
Ev 10 Children, Schools and Families Committee: Evidence
10 September 2008 Dr Philip Tabbiner and Andrew Latham disappointing for us that we have been unable to co- Q58 Chairman: You do not know. Andrew, you do operate with Lord Sutherland’s inquiry to date for not know either? that reason. ETS is a not-for-profit organisation that Andrew Latham: No. I know that we have returned administers more than 50 million exams to to the schools all the materials that we had in the exceptionally high standards in 180 countries. We warehouse, but if, for example, a school discovers bid for this contract because we believe that helping that some of the test papers are unmarked when it pupils learn and teachers teach is our mission. Our opens them up, the test papers would come back, so mission is not to generate profits. We believe that we we do not know exactly when they will be finished. introduced more quality improvement measures through the quality control aspects in this year’s assessment than ever before, with the result that Q59 Chairman: Are we talking about large numbers parents, pupils and schools received good quality or about 1% or 2%? What are we talking about? results. We would like to apologise for not delivering Andrew Latham: We are talking about very small 100% of the marks to schools by the required numbers like 1% or 2%. deadline. There is no question that we experienced Chairman: For those 1% or 2%, it is not very some operational and technical diYculties, which encouraging. hindered our ability to deliver 100% of the test Andrew Latham: Absolutely. results on time. We have not shied away from taking responsibility for these issues. We cannot point to Q60 Chairman: Okay. Let us drill down into what just one or two factors that contributed to the marks you have said. Are you saying, Dr Tabbiner, that you not being completed on schedule. It was, in fact, the had glitches with the technology? From the very full combination of ETS’s and the National Assessment letter that you sent to the Committee, it seems that Agency or QCA’s operational diYculties that there were technical problems that you did not created a compounding eVect and the missing of the foresee. dates. ETS worked in close, transparent partnership Dr Tabbiner: I would call them operational more with the NAA, which guided the ETS throughout than technical, just to put the point on it. It was more the process. Through a combination of the NAA the nature of operational technology, as well as the making changes to the contract and delaying critical interactions with the NAA, that created knock-on decision making and the layering on of additional eVects, or operational circumstances, and that responsibilities, we ended up with a much more created issues for us. For example, we had significant complex and challenging undertaking—much more delays in terms of the response from the NAA as to so than we had originally envisioned or contracted whether we could conduct online training, which is for. Once more, I reiterate my apology on behalf of what we had originally contracted for, and we were ETS for the delays in returning results to schools. I required to do face-to-face training. That created welcome the opportunity to be with you today and operational issues around venues and having to answer your questions. training materials for all the trainers done in a very compressed period. Q53 Chairman: Thank you, Dr Tabbiner. Has every student at Key Stage 2 and Key Stage 3 now received Q61 Chairman: So you are saying that the NAA and their results? the QCA substantially changed the contract after Dr Tabbiner: No, not every student has. There are you had signed it. still marks under way, and we still are data entering Dr Tabbiner: That is correct. those that are being marked by the NAA.
Q54 Chairman: These should have been out Q62 Chairman: With your agreement? originally by 8 July? Dr Tabbiner: One of the issues as I looked at our side Dr Tabbiner: That is correct. of the operation was that we were quite amenable to changes. In a number of instances, we put forward Q55 Chairman: When can every student in the land change orders, which was the required process under expect to get all their results? the contract. But we also made a number of changes Dr Tabbiner: As of now, we are no longer responsible on the fly, as we might have been asked to do in the for the marking or for the reviews cycle in cycle 1. course of a day’s or a week’s work under the process, Our role is data entering the marks as they come for which we did not go through the rigorous process from unmarked scripts. At this point in time, that of seeking a change order. aspect is in the hands of the NAA. Q63 Chairman: Standing on the sidelines, as Q56 Chairman: It has been an extraordinary, drawn- members of the Committee felt we were, we got the out process this year for one reason or the other, has impression—we all know about your formidable it not? reputation in other parts of the world, including Dr Tabbiner: That is correct. North America and beyond—that when you arrived in this country, the people who took you on thought Q57 Chairman: Even standing on the sidelines, do that you would bring your whole operation from the you know when the last result will be out? United States and transplant a bit of it over here. Dr Tabbiner: At this point in time, I do not have the The word that seems to have come out is that you did answer for you as to when— not do that and that you started a brand-new Processed: 17-07-2009 19:04:48 Page Layout: COENEW [O] PPSysB Job: 406183 Unit: PAG2
Children, Schools and Families Committee: Evidence Ev 11
10 September 2008 Dr Philip Tabbiner and Andrew Latham company here. You therefore ran into a lot of Andrew Latham: No. We talked about changes to the diYculties because you were almost a start-up contract. The example that Dr Tabbiner used was company. Is that true? moving to a face-to-face training model, but many Dr Tabbiner: It is important to recognise what has other changes were asked for, such as the marking happened in the creation of this agreement. As you pilot test itself. In the contract that had been signed, often have when you get a five-year agreement, you we had said that we would do a small-scale pilot with then begin to pursue the course of that agreement. focus groups of about 25 people or so just to This agreement was, in very large measure, a supply demonstrate that the model worked. In fact, the agreement, and that is the part we fulfilled— NAA insisted on a very complex pilot with more fulfilling a standardised test that was already created than 800 diVerent markers. and using methodologies that were, in general, Chairman: We will drill down into a couple of those already accepted and dictated. We were then filling issues in a moment, but let us carry on looking at in certain supply elements—certain systems, delivery what happened. logistics and the accumulation and implementation of quality control—rather than fully developing the Q68 Mr Chaytor: Dr Tabbiner, is this the first time standardised methods that would be in the test. In that ETS has had to withdraw from a contract before that light, we used some people whom we already its completion? had, as well as building up staV locally. We also Dr Tabbiner: This is the first time that we have asked applied a significant amount of resources during the to withdraw from a standardised achievement heightened period of the May, June and July time contract, yes. frame, when we ran into significant delays. Q69 Mr Chaytor: Have you been asked to withdraw Q64 Chairman: Was there any element of that where from a contract? Is this the first time that you have you were working in a rather diVerent culture from asked to withdraw? Have you previously been asked the one you are used to working in and where some to withdraw? local knowledge was missing on issues such as how Dr Tabbiner: No, we have not. many qualified markers there might be in a pool to do your marking at a time when there was an awful Q70 Mr Chaytor: There has been no previous lot of marking to be done? experience such as this, in which the whole project Dr Tabbiner: We have the good fortune to be has got into such diYculties that the contract was working in 180 countries with many diVerent terminated? cultures. We do standardised testing in Serbia, Egypt Dr Tabbiner: That is correct. and Korea, so we have experience of many diVerent cultures. The other thing that I would say in the Q71 Mr Chaytor: In your written statement to the context of how we approached our agreement with Committee, you repeat the acceptance that there the QCA and the NAA relates to the issue of were technical and operational diYculties. Just now, complete transparency. We looked to them to give us you said that the diYculties were more operational guidance and direction on some of the issues that than technical. You give two examples of work from might be encountered. In many instances, when we schools being allocated to two diVerent markers, or would implement eVorts to resolve an issue, it was some markers not being able to access online the only later that the NAA would point out what had work of their students. Could you list all the traditionally happened. We found that some of the operational diYculties of which you are aware, for useful information or data that would have helped to which you or your company accepts responsibility? inform early decisions was not shared with us. Dr Tabbiner: One of the things that I would identify is that as you look back over a contract and you examine some of the issues that arose from the Q65 Chairman: Dr Tabbiner, looking at your date of delays, we can say that if we had moved on into cycle arrival, it sounds a little bit like you came in when 2, we would have made modifications to try to deal things started going wrong. You came over when with some of the technical issues. For example, we things were in trouble. had set up a process for the marker allocations. We Dr Tabbiner: I arrived in mid-May. did not bundle up the materials that were to go to the markers until such time as each mark was Q66 Chairman: May I switch to Andrew Latham standardised, then we submitted the materials. In who has been involved in the operation rather retrospect, that cost us some time. In a future year, longer? When did you first think that things were we would pre-bundle the materials and they would going wrong? be ready for standardisation. That is just an Andrew Latham: We knew that we had challenges as illustration. Another fact is that we had designed the far back as December, when we were pilot testing the systems in a way in which the schools had to fill in marking solution. From that time on, we worked the student register. If a school did not fill in the very closely with the NAA to try to address the issues register, the system was designed without a default that we saw arising. for it, and there was an absence or a block that was left empty. When teachers went to enter the marks and they ran into that block, they had to call the Q67 Chairman: Was that before the NAA asked for helpdesk to deal with a registry issue rather than an any changes in the contract? online issue. In retrospect, that is something that we Processed: 17-07-2009 19:04:48 Page Layout: COENEW [E] PPSysB Job: 406183 Unit: PAG2
Ev 12 Children, Schools and Families Committee: Evidence
10 September 2008 Dr Philip Tabbiner and Andrew Latham would have made adjustments for, knowing that Q80 Mr Chaytor: Which you did? schools, in some instances, would not necessarily fill Dr Tabbiner: Yes. in all the data points. The next point concerns the return of materials. Markers were asked to put the Q81 Mr Chaytor: Okay; and when was that done? marked scripts on the top of the box, which they did. Andrew Latham: In February or March, I believe. We took them to be 100% marked scripts and returned the boxes, as appropriate, to schools. Later, Q82 Mr Chaytor: Again, in retrospect, given that we found that, in some instances, there were tests take place in the early summer, was February or unmarked scripts in those boxes, which schools later March not too late to have completed the logistics returned to us. In a future circumstance, we would pilot, whatever the size of the logistics pilot? do a 100% visual inspection of the scripts. Dr Tabbiner: A number of facets of the contract were dictated for certain timing, but because of delays Q72 Mr Chaytor: Should those operational regarding the design of the online training and the diYculties not have been identified in the pilot design of contracts with teachers, we ran into severe scheme that was carried out last December? delays. We lost three to five months, depending on Dr Tabbiner: We intended to do an end-to-end pilot how you look at that, in the cycle as a result of the run, but we were refused the opportunity to do that. NAA procrastinating around decisions, online All we were allowed to do was a small pilot on training and the contracts for markers. logistics with the schools in only 25 schools.
Q73 Mr Chaytor: Andrew’s response just a moment Q83 Mr Chaytor: In respect of the online training, ago was that one of the NAA’s decisions was to then, you have said that there was a request to extend the scale of the pilot, not to reduce it. Can you change the contract. Did the original contract not clarify that? specify online training? Andrew Latham: Sure; I was talking about the Dr Tabbiner: It did. marking pilot in which we were supposed to demonstrate that the elements of our solution that Q84 Mr Chaytor: Is that not a fairly major were new this year were eVective, such as having the contractual change? Could you simply have refused markers actually enter their marks online or having to change the contract in that way? the markers be trained online, which was totally Dr Tabbiner: You have raised a valid point. One of separate from the logistics pilot. the things that I would say from our side—I would point to this as a responsibility of ours—is that we Q74 Mr Chaytor: Okay. So the marking pilot took were too amenable to change without saying, “No, place last December and, if I can put this to Andrew, we won’t do that.” This is a classic example of that. from your point of view was that completed By the time we were told not to do the training successfully? online—to do face-to-face training—we realised Andrew Latham: The marking pilot—yes. that the time frame meant that we would have to drive with complete focus and vigour to the goal of Q75 Mr Chaytor: But when was the logistics pilot 8 July. completed? Andrew Latham: But I would add to that that we Andrew Latham: We never conducted a full scale were very clear in cautioning that we felt that to logistics pilot in the end. We did a small scale make such a move would greatly increase the risk logistics pilot where we sent some packages to a and was therefore not advisable. couple of dozen schools and said, “Does this packaging look right; does it work; are the Q85 Mr Chaytor: You have referred to a delay of instructions clear?” It was that level of pilot. maybe three to five months because of the change from online training to face-to-face training. When Q76 Mr Chaytor: So in retrospect should there have were you told it had to be face-to-face training? been a full scale logistics pilot? What was the time scale? Andrew Latham: Absolutely. Dr Tabbiner: March.
Q77 Mr Chaytor: But that was not specified in the Q86 Mr Chaytor: And when should the training contract. have taken place? Andrew Latham: No. Dr Tabbiner: May. The training still took place in Dr Tabbiner: But we desired to do it and were refused May, but for us the rush was to find venues and staV, the opportunity. to obtain the materials, to conduct the training eYciently and to get the markers in place. Q78 Mr Chaytor: Right. So you asked the NAA to do it. Q87 Mr Chaytor: So if you were told that the Dr Tabbiner: If we could do it. training had to change from online to face-to-face in March, did you then not point out to the NAA that Q79 Mr Chaytor: If you could do it—for permission this would result in a delay of three to five months to do it. They refused for what reason—lack of time? and that the tests were supposed to be completed and Dr Tabbiner: They asked us to do the small 25- reported by July? I do not understand the time school logistics pilot. scale there. Processed: 17-07-2009 19:04:48 Page Layout: COENEW [O] PPSysB Job: 406183 Unit: PAG2
Children, Schools and Families Committee: Evidence Ev 13
10 September 2008 Dr Philip Tabbiner and Andrew Latham
Andrew Latham: We were worried that this was identified specific concerns on decisions that were going to come to pass, so from January we had been being made around implications, but the NAA was advising, “Please don’t do this, because we really fully briefed and fully aware throughout the process. believe that the online is the way to go, and that you will greatly increase the risk if you do.” We assumed Q92 Paul Holmes: Can I just clarify from what you that that was what they were eventually going to do said that you clearly asked the NAA to terminate in March and that that was the decision they were your contract, rather than the NAA telling you that going to come to, so we were trying frantically the contract was terminated? behind the scenes to line up venues and recruit Dr Tabbiner: I asked that we would mutually exit trainers and others in that eventuality. this contract, yes.
Q88 Mr Chaytor: Yes, but contractually you could Q93 Paul Holmes: And you asked that in June? have refused to have carried out face-to-face training Dr Tabbiner: I did. and stuck with the online training. Dr Tabbiner: Contractually, we could have used the Q94 Paul Holmes: So when Ministers and Ken change order mechanism, which is the way you deal Boston came before the Committee in July and I with alterations from the original contract. specifically asked them what the penalty clauses were for the company and so forth and they said, “That’s all ongoing,” and did not give an answer, they Q89 Mr Chaytor: This is my final question. Had the already knew that the contract was going to be contract continued into next year—the next cycle— terminated? which of the operational problems would have been Dr Tabbiner: I cannot speak to the fact as to whether the most diYcult to resolve, or do you think that Ken Boston knew the contract would be terminated they could all have been equally resolved in year at that point; what I can tell you is that in my two? discussions with the NAA managing director, I said, Dr Tabbiner: We certainly categorised and looked at “We either have to renegotiate the contract or we process changes for cycle 2. The challenge that we need to end this, because what we are operating would have faced is the lack of timely decision under is not the contract we signed and contracted making. If it were said, “What you did in cycle 1, we for.” want you to do in cycle 2,” it would be reasonable to capture those changes and successfully embed them Q95 Paul Holmes: We can assume that Ken Boston in the design and planning. But the endemic amount and Ministers knew that but did not tell us when we of decision making and the challenges of timely asked them the question in July. In all the material decision making made it very diYcult for me to that you have submitted, including the statement believe that there was a probability that we would dated today, you emphasise your massive see success in cycles 2, 3, 4 and 5, which is why I said, experience, that you started maths assessment 60 “We either completely re-craft this contract, or you years ago, that you are very experienced, that you let us out.” won the contract on quality and that the contract, as you have just said, was for delivery, logistics and Q90 Mr Chaytor: When did you say, “We either quality control. We will talk about the quality later, completely re-craft the contract”? but the delivery and logistics collapsed. How is it Dr Tabbiner: Approximately mid-June. I cannot that such an experienced and qualified company give you the specific date, but it was mid-June. oversaw that collapse? Are you saying that it is all down to the NAA changing criteria, contracts and requirements, or is some of it your fault? Q91 Mr Chaytor: Finally,at what point did you alert Dr Tabbiner: Clearly, I would want you to leave the NAA to your feeling that you would not be able today’s meeting with the view that we share to meet the July deadline, because of its request to responsibility. I am not here to place blame on change from online to face-to-face training? When anyone—it is for this Committee to determine the did you first alert it to the fact that you felt you predominance of blame and who holds it. We are would not meet the deadline? clearly responsible for activities and misses on the Dr Tabbiner: As I mentioned earlier, from the logistics and operational fronts. beginning we purposefully operated in a completely transparent fashion. The NAA had staV embedded Q96 Paul Holmes: On the logistics front, you seem to in our organisation in Watford on a day-to-day be saying that that is because the NAA would not let basis. They had early morning briefings and you run a full pilot, so was that its fault or your fault? meetings throughout the day, and they received all Dr Tabbiner: As I said in my opening statement, it is the management information that we saw, all the near-on impossible to pull out single factors and say, data that we saw and all the results day by day “That was the causative factor.” You have a throughout this process, so we operated as if there confluence of issues that create a knock-on eVect were no divide between the two organisations in the through a process, which had a dead end of 8 July. context of ongoing and daily operations. When we When you have that compounding eVect of knew data, they knew data; when we saw them, they decisions and actions against a fixed time frame, you saw them. So, it was not so much a point in time then spin out of control at that 8 July time frame where we would say, “There are concerns”; we with those compounding eVects. Clearly, we have Processed: 17-07-2009 19:04:48 Page Layout: COENEW [E] PPSysB Job: 406183 Unit: PAG2
Ev 14 Children, Schools and Families Committee: Evidence
10 September 2008 Dr Philip Tabbiner and Andrew Latham responsibilities, and clearly there were shortcomings helpdesk and then broadly at the whole project. The in how we dealt with them and responded to some of frank answer is that the helpdesk questions—the the issues on the table. backlog—were not completely dissipated until such time as markers had finished their work at their Q97 Paul Holmes: You said in the written statement home locations. The volume dropped oV by virtue of that you thought you were taking over an existing lack of calls. body of qualified markers and that, in fact, you had not been told that there were problems in recruiting Q101 Paul Holmes: As for Ken Boston’s apparent markers. Why had you not done the groundwork? claim in the minutes of the meeting on 14 July that Why had you not done a press search and found the 70 staV whom the NAA put in cleared it, are you that, in every year since 2003, there had been saying that you put in a greater proportion of the problems and that delays and targets had not been eVort that was made or an equal proportion? met? Why did you not know that? Dr Tabbiner: A greater proportion. From the overall Dr Tabbiner: We did due diligence, but I submit that contract, we expected to have about 60 staV involved we probably did not do a suYcient amount. At the in running the project. At its height, we had more same time, during the tendering process and dealing than 400 staV. with the NAA, when we outlined the fact that we wanted to use the online training method, we got Q102 Paul Holmes: How can such an experienced into dialogues about some of the issues that had company—which you keep emphasising that you existed in the past. We were assured that issues, such are—have made such a miscalculation in that you as marker availability,were no longer problematic or thought 60 staV would do, but you needed 400 in would have been dealt with in large measure by some the end? of the other programmes that we had put in place. Andrew Latham: The decision-making delays and losing five months in the cycle mean, in eVect, that Q98 Paul Holmes: Again, I want to be clear. You we should not have had to deliver the results until continued to use the online training as planned for November. To catch up that kind of distance, we had 5,000 experienced markers, but you said that the to be prepared to put a significant amount of NAA insisted that you do face-to-face training for additional resources into play. the 5,000 new or less experienced markers. Is that so? Andrew Latham: No. The original plan was for the Q103 Paul Holmes: So, you are back to saying that new inexperienced markers to attend face-to-face the most significant factor behind all the problems training, while the experienced markers in maths was a five-month decision-making delay, which was and science would be trained online. What happened the responsibility of QCA and NAA. was that the NAA asked us in March to make it Dr Tabbiner: That is certainly one illustration. Other optional and allow everyone to decide whether to do illustrations compounded the impact. face-to-face or online training. We said that that could not be done, because it would double the cost Q104 Paul Holmes: Finally,at exactly what point did and double the risk. We said that matters had to be you realise that you would not meet the 8 July decided one way or the other. The decision was to go deadline? with face-to-face training. Dr Tabbiner: We recognised that, and informed the QCA late on 25 June. Q99 Paul Holmes: For the whole 10,000? Andrew Latham: For the whole 10,000. Q105 Chairman: Can we just clear up one thing that came out of that? You came to the UK in what Q100 Paul Holmes: Dr Tabbiner, you have said that month? you put resources into the whole thing in May, June Dr Tabbiner: Mid-May. I took over responsibility and July, when it became apparent that problems for the NAA as a part of my role as director and were emerging. Ken Boston said in July that when he chairman of the Supervisory Board of ETS, the started to look into things, the QCA found, for global subsidiary. example, 10,000 e-mails that had not been answered and that that they were dealt with only because the Q106 Chairman: So you were in charge of the NAA put in 70 members of its staV. Did you clear relationship with the NAA? the backlog or did the NAA clear the backlog? Dr Tabbiner: That is correct. Dr Tabbiner: Frankly, the backlog in helpdesk e- mails and issues was not completely dealt with until Q107 Chairman: Andrew, you were here the whole such time as it was clear that the marking by markers time? at their home locations was over. We increased the Andrew Latham: That is right. helpdesk capability substantially. Originally, we had one helpdesk location, and we doubled the staV Q108 Chairman: At what level were you talking? there. We added a second location and then a third Who were you talking to on a daily and weekly basis location in Watford. We added a fourth location of in the QCA and the NAA? What sort of people? folks in the US who just did outbound calling—calls Andrew first. to markers. We then added a fifth location that did Andrew Latham: On a daily basis, I would deal with nothing but e-mail responses 24 hours a day, seven my counterpart at NAA, the project director, who is days a week. We threw huge resources at the a level below the managing director. Processed: 17-07-2009 19:04:48 Page Layout: COENEW [O] PPSysB Job: 406183 Unit: PAG2
Children, Schools and Families Committee: Evidence Ev 15
10 September 2008 Dr Philip Tabbiner and Andrew Latham
Q109 Chairman: Okay. Who was that? Q113 Fiona Mactaggart: You are putting a lot of Andrew Latham: His name is David Barrett. weight on that decision causing some of the problems. I am prepared to accept that it probably did cause quite a number of these problems. I am Q110 Chairman: When you arrived, Philip, everyone wondering how hard you pushed back, if you see must have said, “Wow! The boss is coming from what I mean. Did you make clear to the QCA the America.” Who were you talking to? consequences of doing that? If not, why not? Very Dr Tabbiner: Most of my interactions were with often, for people who do not understand the David Gee, who is the managing director of the technology, it is not their job to understand the NAA. technology and therefore the consequences in terms Chairman: Thank you for that. Fiona. of the operational aspects, the technology and the potential for frozen screens, and so on. That is what Q111 Fiona Mactaggart: I am interested that you you know; they do not need to know that, because have described the problems that you faced as being that is not their job. Unless you push back and say, largely operational, rather than technical. However, “These are the consequences that we foresee,” the when I speak to markers, I hear tales of really QCA is likely to think, “Oh well, this will be nice.” frustrating technical problems, such as not being Andrew Latham: I did not push back hard enough. I able to enter marks and not being able to get the certainly made it clear that I did not think that that right screen for the scripts. I do not want this issue was the right move and that it increased risks. If I to be avoided, because it seems to me that it cannot had known the consequences—I underestimated be anyone else’s fault—it is your technology, which how negative they would be—we would not have you developed. So will you tell us about the extent to proceeded. which those matters were part of the problem and why they were so bad? Q114 Fiona Mactaggart: How did you make your Dr Tabbiner: Certainly I can fully appreciate views known? markers’ frustrations and the issues that they faced. Andrew Latham: Through conversations and e- One of the distinct advantages of trying to do the mails. online training was giving markers the opportunity Dr Tabbiner: There were a number of other to work with the system and practice with it before circumstances regarding, for example, data feeds. having to go into it with the marking process. The We had set times when we were supposed to markers did not have that opportunity, which download the data over to the QCA for it to look at created issues. As I said earlier, there were also as we got closer and closer to the 8 July date. There technical issues. I know from the helpdesk calls and were prescribed dates. But there was a point in June the e-mails that we received that those issues really when, outside those prescribed dates, the QCA frustrated markers. As they had to standardise and demanded that we do a data run every single then conduct their quality assessments every 80 evening. It took our data staV from approximately 8 scripts, they would run into situations where the pm through to the morning to do it, clean it and pupil data did not match the registry data, which make it available for observation. I formally stated would create a freeze. Then they would have to call that that was an issue and that it could and would the helpdesk, for which, as we have already severely damage our ability to hit the prescribed data discussed, there was a very heavy backlog, which runs for 4, 5, 6, 7 and 8 July. The answer was, “Do they may not have been able to get through. That it.” might have driven them to trying to circumvent the problem or to making their own work-throughs, Q115 Fiona Mactaggart: Was that subject to the which was not something that a marker was really formal change process that you discussed with us? able to do. So it was a very frustrating process if a Dr Tabbiner: In that instance, I said to the QCA that marker hit a stop point in the programme. we would be willing to do it—that it would have to Programmes such as this one are designed with fairly pay. It agreed. But I also informed it of the potential rigid decisions or criteria. In cycle 1 you try to look risks that it would create. for the exceptions or break points to the rule and make adaptations for cycle 2. That problem was compounded by our not being able to provide the Q116 Fiona Mactaggart: Ken Boston told us that it online training that we would have liked to have was your fault—that is in the general summary of his provided, and also by some of the mismatches on the evidence. You say that specific changes that the attendance register that created the hit points for QCA made contributed significantly to your failure teachers. to deliver on time. We have heard of the change to the training, and we have now heard of the extra data runs that you had to provide. What other Q112 Fiona Mactaggart: Presumably you were told specific changes were made that compromised your why they wanted markers to have the option of face- ability to deliver on time? to-face training. What was the reason that you Dr Tabbiner: On the attendance register, the whole were given? plan and outline was that schools were required to Andrew Latham: The concern was that markers fill in “Absent” or “Present” for every student. As would not like online training and therefore would the time came to inform the schools, the QCA did not participate in the process. They were afraid that not allow us to say that they were “required” to put it too many people would not sign up. in that way, rather that it would be “preferred”. That Processed: 17-07-2009 19:04:48 Page Layout: COENEW [E] PPSysB Job: 406183 Unit: PAG2
Ev 16 Children, Schools and Families Committee: Evidence
10 September 2008 Dr Philip Tabbiner and Andrew Latham may seem like a simple change, a simple diVerence, Q121 Fiona Mactaggart: No, we will decide. I am but if schools took a default and did not put in the just interested in what your view is. student register, that had a very significant impact in Dr Tabbiner: What I can say in this situation—I am terms of the data running. It had an impact on the not qualified to answer more broadly than this—is helpdesk, because teachers ran into problems in that with regard to us, rather than the Government, marking and would then call the helpdesk, but the we needed to recognise that it was a supplier-buyer helpdesk was not designed to handle attendance relationship, where we were providing supply, and register—it was designed to handle other technical that notwithstanding the co-mingling and ongoing issues. So that is another illustration of some of the dialogues and relations, we needed to say, “Right, changes and issues that we faced that had significant that’s a change order, and we have to have it signed knock-on eVects for the project. oV and dealt with in that manner.” That is how I propose that it would be best handled in future. Q117 Fiona Mactaggart: Would it be possible for you to provide the Committee with a list of dates Q122 Fiona Mactaggart: So you are basically saying when things were changed? that the suppliers need to have more clear Dr Tabbiner: Just to clarify, are you referring to procedures themselves, as a routine, in order to deal dates when things were changed or dates that were with this relationship? diVerent? Dr Tabbiner: Correct.
Q118 Fiona Mactaggart: Dates when it was agreed— Q123 Fiona Mactaggart: I suppose the thing that I sometimes informally and sometimes formally— find most concerning about your evidence is the that there would be changes to the agreements that suggestion that the NAA withheld vital operational you had achieved in the initial contract about the information, such as the availability of markers. way you did things regarding, for example, the From our point of view, that is an important issue of additional data run and the change in the training. I public accountability. I would like you to be very would be interested in seeing a list of the changes specific about what that information was, why you that in your view had an impact on your ability to did not know it, and what you believe the NAA’s deliver. I realise that that will take you time, but you duty was to inform you of those matters. must have it somewhere. Dr Tabbiner: The issue that I would use as an Dr Tabbiner: There are certainly large-measure illustration is the number of markers who provide changes that are categorisable. As I mentioned marks for Key Stage 3 English. We found that after earlier, by virtue of the co-mingling of staV and of signing the contract on the understanding that that having people working together in a transparent was not going to be a problematic group, it was in fashion on a day-by-day basis, one of the distinct fact problematic. It was later stated that that is an disadvantages—this was my observation coming historical fact. into this in May—is that an ongoing dialogue develops over the course of many months of working Q124 Fiona Mactaggart: All you need to do is read together, so change can happen in a fairly The Times Educational Supplement to know that. straightforward, ubiquitous manner, which makes it You do not have to be a detective. more diYcult to characterise and demonstrate. Dr Tabbiner: We also found that same issue However, the answer is yes on large-measure regarding the attendance register in schools. As I changes. have said, we were not allowed to say, “You must fill in the attendance register,” but only to say that that Q119 Chairman: Fiona is only asking for the would be preferred. We later found that that was a significant ones. Can you supply that list? documented relationship issue, and that in previous Dr Tabbiner: Yes, we can.3 contractor experiences with a diVerent system it was handled by using a postcard system later in the cycle. Q120 Fiona Mactaggart: I am hearing that the level That information was not known to us at the front of engagement actually caused some of the end of this contract. All that damaged the time problems. The level of embedding of staV caused frame, and in some instances it damaged relations problems, you suggest, because the accountability with markers and markers’ view of our capabilities. went back to diVerent places and therefore got slightly muddled—I think that that is what you are Q125 Lynda Waltho: I am particularly concerned telling me. This is not an uncommon arrangement in about IT. At our meeting on 14 July, I asked Ken such contracts for public service issues. Are you Boston whether the IT was overly ambitious, suggesting that it would be wrong, in future, for a because it was my experience, having spoken to government agency or department to expect an markers, that they felt that that was a very great operational contract from a private company to diYculty. He said that he thought it was not work in such a way, with seconded staV and a more ambitious enough. Indeed, he said that although intimate relationship than you might have had there was a pilot, once it was upscaled, that is really elsewhere? when it hit the fan, so to speak. Scripts were marked Dr Tabbiner: It is not my role to opine as to what the in hard copy, with data entered electronically, rather Government should or should not do. than being marked on screen, which is what we mostly have with GCSEs and A-levels. An 3 Supplied in confidence. Not printed. organisation such as yours, with operations in more Processed: 17-07-2009 19:04:48 Page Layout: COENEW [O] PPSysB Job: 406183 Unit: PAG2
Children, Schools and Families Committee: Evidence Ev 17
10 September 2008 Dr Philip Tabbiner and Andrew Latham than 180 countries, presumably has this level of Lynda Waltho: Thanks very much. engagement across many markers. Whose decision was it to have a split between hand marking and data Q130 Mr Carswell: I may touch on one or two points entry? Was that specified in the contract, or was it that you have touched on before, but I want to be completely up to you how you delivered things? absolutely clear. In your view, was there a significant Dr Tabbiner: No, that was specified in the contract. diVerence between the contract that you signed and The request for proposal was online training with the contract that you ended up being asked to hard-copy marking and then entry by teachers. That deliver? is what we bid for, and that is what was set up. Dr Tabbiner: Absolutely.
Q126 Lynda Waltho: Do you do that elsewhere? Q131 Mr Carswell: Had the NAA and QCA known Dr Tabbiner: We certainly do paper-and-pencil what they wanted straight up, would that diVerence have been avoided? testing in other tests in other countries. Dr Tabbiner: Had they known what we ended up Andrew Latham: As part of the contract, we were working through, the contract would have been asked to prepare two innovation cases for future diVerent, and it would have been a completely innovations that the QCA would want to introduce: diVerent outline. one was for a full online marking solution, where the papers are scanned in and everything is marked online; the second was for delivering the tests via Q132 Chairman: Do you agree with that Andrew? computer. The intent was that these innovation cases You have been here throughout the whole cycle. would be implemented during the five-year course of Andrew Latham: One point of evidence I can point the contract. When we began exploring those cases to is that we went through a nine to 12-month procurement process, where we would go back and with the QCA over the summer and talking about a forth designing the contract and exactly what the timeline for when we could implement them in 2007, solution would be. Within the first six months or so it said, “Hold oV on the innovation cases. We’re not of winning the contract, five significant change going to proceed any further with those at this orders were put to us saying, “Could you examine point.” That was in the fall of 2007. this change?” I remember thinking at the time, “Why didn’t this come out during the previous nine to 12 Q127 Lynda Waltho: So had the relationship worked months?” a little better and things had worked out better, how would you have developed the system for the next Q133 Mr Carswell: So the client not being clear in cycle? Where would you make your innovations? their mind would be the way some people might Dr Tabbiner: One of the things that we do is capture characterise things, and you would not disagree the exceptions that were asked for throughout the with that. process and then make adaptations because of those Andrew Latham: I would not disagree that what was exceptions. We saw that we had to make exceptions wanted evolved over time. to handle the pupil register, so, in cycle 2 we would have adapted the technology to make a decision in Q134 Mr Carswell: Just one thing I may have the absence of a notation that the pupil had taken missed: why did you ask to be released from the the test. Later, we also had to do split allocations contract with the QCA? with markers, which was mentioned earlier. Some Dr Tabbiner: It was very clear to me that the contract markers were not able to access information online, was not what we were operating under. As I because they had a split allocation with another mentioned earlier, we are a not-for-profit marker. We would have adapted the system to allow organisation. The cost burden of this contract in two markers to enter results from one school in a cycle 1—and the fact that it was not going to be certain subject. diVerent in cycles 2, 3, 4 and 5—was such that it was prohibitive for us. Q128 Lynda Waltho: Have you ever had such a Q135 Mr Carswell: Apparently, you have repaid a catastrophic breakdown, as appears to have substantial amount of the fee for your services in happened here, in any other country or system? 2008. How much? Dr Tabbiner: As I said earlier, this was such that we Dr Tabbiner: The settlement agreement was £19.5 felt that this was a contract that we had to withdraw million. from, and we made that case clear. Q136 Mr Carswell: That is paid back? Q129 Lynda Waltho: And the changes to the Dr Tabbiner: That is what we paid as a settlement contract? Have you ever experienced that under any agreement. other system? Dr Tabbiner: To my knowledge, this is a significant Q137 Mr Carswell: Do you have a view of the change in the way the contract was originally let. All conduct of the QCA and its head, Ken Boston? Are contracts have some adaptation, but the nature, tone they up to the job? and tenor of this was beyond that which I had seen Dr Tabbiner: That, I would submit, is for someone previously. else to consider. Processed: 17-07-2009 19:04:48 Page Layout: COENEW [E] PPSysB Job: 406183 Unit: PAG2
Ev 18 Children, Schools and Families Committee: Evidence
10 September 2008 Dr Philip Tabbiner and Andrew Latham
Q138 Mr Carswell: Given your experience around the contract in June. You must have been pretty the world—in Egypt, Korea and wherever—do you worried to come over in May and realise what was have a view about the process and how we need to happening. change the process to avoid this sort of public policy Dr Tabbiner: I would submit that part of the reason procurement failure? Do other countries have a why I was asked to take on this project was that we better process of public procurement contracts? If saw some of the risks and we were aware of them. so, what is it that they do diVerently? They asked for me to have oversight and to come in Dr Tabbiner: I was not here during the procurement and provide my perspective. process. I know that it was lengthy, involved and complex. I do not believe that it is the procurement Q143 Chairman: So how early on after you arrived process, but the implementation of a contract. If a in May did you meet Ken Boston? contract is implemented as written, in large measure, Dr Tabbiner: I did not meet Ken Boston until mid- things will go much more eVectively. July.
Q139 Mr Carswell: Were you always clear—I do not Q144 Chairman: Mid-July? Surely, if things were mean as individuals but as a company—as to who going as wrong as you say they were, you would have was working with whom, where the reporting lines been banging the desk and saying, “I need to speak were, and who had oversight on each side of the to them.” fence? Was that always clear? Dr Tabbiner: I spoke with David Gee. That was the Dr Tabbiner: Certainly, from a contractual and person whom I interacted with on a high-frequency operational view it was very clear who the basis. relationship management person was, and who the next and the third escalation points were. From a Q145 Chairman: You never thought that a £156 functional group beneath Andy, there were straight million contract to your organisation—whether it is relationships from our team to folks at QCA or not-for-profit or not—was so important that you NAA. should have gone to see Ken Boston. Dr Tabbiner: It was very clear that the escalation of Q140 Mr Carswell: Do you think that they were the contract called for the interactions with David always quite so clear as to who was in charge on their Gee, and that is the pursuit that I had. side of the fence? Dr Tabbiner: When you say “on their side of the Q146 Chairman: Andrew, you never thought that fence”, do you mean our staV? that was necessary—that really going for the top on Mr Carswell: No, the QCA and the NAA people. this would have been advisable. Dr Tabbiner: I know that there were times when it Andrew Latham: No. I felt that my responsibility was very clear and we got good feedback. There were was to inform my counterparts at NAA. I had some times when we had more than one person responding interactions with David Gee as well, and then some on a specific item with a diVering view, giving us to escalate the issue within my organisation, so that diVering directions. Phil and my chief executive oYcer and others were very closely apprised of what was going on. Q141 Mr Carswell: Being quite a new Member of the House of Commons, I am quite surprised at how Q147 Chairman: We will turn to the future, then. remote this whole process is from any elected, Some things you have said worry me, in the sense democratically accountable Minister. That that schools are schools. In your business you must personally surprises me. Does it surprise you? know that schools are ineYcient—sometimes. They Dr Tabbiner: When you say this process, do you can be average, brilliant, sometimes awful, but they mean the procurement process? will fill in the forms and do things. That is your Mr Carswell: Yes, and the management and business and you should have anticipated that. You oversight of it. should have known about the markers issue, but if Dr Tabbiner: It is for the Government to determine the decision to move from online training to face-to- how it should best be operated in the public trust. face training was so crucial, you should have gone to the top and said, “Come on: this changes the whole Q142 Chairman: Just on that point, as you have been nature of the contract and will end with serious giving evidence, it strikes me that, as grown-ups, we consequences.” At what stage did you say that— all know that with vast contracts such as this, really robustly? Surely you should have done. stretching over five years and costing £156 million of Andrew, you were here. taxpayers’ money, there is bound to be both give and Andrew Latham: I reiterate what I said earlier. When take. You were two teams working together. This the decision was being made, I advised, “I really could have worked out absolutely fantastically, think this is the wrong move and increases risk,” but, could it not? Day to day, you have to have an in fairness, I did not anticipate how badly it would embedded relationship in something this complex. impact us. I underestimated the negative impact. You seemed to know quite late on, Dr Tabbiner, that things were going wrong. You say that you arrived Q148 Chairman: Someone on the NAA must have in May. You decided—you did not make it clear at said to you, “Look guys, this has got to change.” what level—that you were going to withdraw from Who was it? Who said it? Processed: 17-07-2009 19:04:48 Page Layout: COENEW [O] PPSysB Job: 406183 Unit: PAG2
Children, Schools and Families Committee: Evidence Ev 19
10 September 2008 Dr Philip Tabbiner and Andrew Latham
Andrew Latham: I believe that the decision was whoever’s fault it was, was a shambles and went probably made by David Gee, working with David belly up, but you are saying that quality control is Barrett. Based on the pilot test, they made the better this year than it has ever been. You say in the decision: “We feel people won’t like online training, written statement, “We can confirm that the quality therefore you have to make it optional.” of markers in 2008 was . . . higher than in previous Dr Tabbiner: The other thing is that from our years. We introduced a more rigorous method of interactions with, and feedback from, David Gee certifying markers”. Do you stand by that? and David Barrett, it was very clear that they were in Dr Tabbiner: Yes, I do. The marker pool and the communication with people up-line from them and work that the markers did and the eVort they put with Ministers, because they were certainly asking us forward was very good and we appreciated their for data and information going to meetings. So we perseverance through some very challenging and were of the view that the information we were diYcult times. I would also point to the quality passing on in a very transparent way was being used control measures that we put in place. At the end of and harnessed throughout their organisation, much their training, markers had to go online and as we were escalating issues in ours. standardise, and then every 80 scripts that they entered, they were called on to do a qualification Q149 Chairman: But I wanted to get back to you on check and it was determined whether they could the heart of the problem, which seems to be markers continue or not, so every 80 scripts a quality control and marking. I have listened to every word of your step was introduced into the process. evidence, and something seriously went wrong with markers and marking. You were already pretty Q152 Fiona Mactaggart: How many failed that step? stretched in looking for qualified markers, especially Andrew Latham: Four hundred and five—about 4%. in English, so did someone in the NAA suspect that a new online marking system with which markers were Q153 Fiona Mactaggart: Out of how many markers? unfamiliar would deter even more markers from Andrew Latham: About 10,000. trying to do the job? Dr Tabbiner: Clearly, the heart of the whole contract Q154 Paul Holmes: The National Association of is markers and marking; that is the essence of the Head Teachers sent round yesterday a summary of kernel of everything that we had to do and the hundreds of e-mails and phone calls it has had everything that the QCA was concerned about. But from head teachers during the summer period. I have there is no doubt that there were times when we here 30 edited highlights, of which I shall read just thought an assertive approach would have been three or four. A Barnsley school says the quality of good, but that was not necessarily a shared view. I the marking is “abysmal”. A Rochdale school says can appreciate that aspect, too, because we did not there were 30 marking mistakes out of 56 pupils’ want to be oV-putting, so we also reflected on the papers. The markers were “unable to apply the NAA’s historical views about how best to approach marking criteria accurately”. Weak candidates were schools, markers and circumstances. given the same marks as strong candidates; that is a Yorkshire school. Warrington’s primary strategy Q150 Chairman: But, Dr Tabbiner, you are managers recommended that all their English papers professional managers, and it seems to me that at be returned. The quality of marking is “very low, some stage something went wrong with your sending 24 papers back, marking shows a complete professional management. Otherwise, explain to me lack of understanding of marking scheme”; that is why you have walked away from this contract—on Northants. There are also comments from quite generous terms, it seems to me, from looking at Nottinghamshire—it just goes on and on and on. the figures—with the Government and the QCA. How do we square what the head teachers are saying You have not robustly said, “Look, we were unfairly now they have the papers back and what you are treated, we shouldn’t give an inch.” You have made saying? You have said the quality is better than ever big financial concessions to walk away from this and the head teachers say it is worse then ever. contract. Dr Tabbiner: All I am able to tell you is that we put in Dr Tabbiner: Looking at contracts and ending quality control steps throughout this process. There contracts is a business-related decision, and as I said was one immediately after training, so that everyone earlier, I looked at the losses that we were facing in had to standardise, and then every 80 scripts there cycle 1 and could see only continuing losses in cycles was a stop and check as to how folks were doing vis- 2, 3, 4 and 5. This year we have lost £50 million as a a`-vis a quality standard that was applied to every result of this contract. We are a not-for-profit one. organisation: we have no shareholders and we are Andrew Latham: I would like to point out a not traded on a public stock exchange. We are not in significant diVerence from the past model of quality a position to absorb £50 million losses per annum, so control whereby the markers self-selected their own for us it was a business decision and that was the scripts up front. They would send a set at the very most expeditious one. beginning and they would send a self-selected script Chairman: Thank you. at the very end. They would send them to their own individual team leader, who would evaluate them on Q151 Paul Holmes: We have established that there an individual basis, whereas we were using a national were three parts to the contract—delivery, logistics standard, an objective set of scripts that were and quality control—and that delivery and logistics, marked by the most senior leadership in advance, so Processed: 17-07-2009 19:04:48 Page Layout: COENEW [E] PPSysB Job: 406183 Unit: PAG2
Ev 20 Children, Schools and Families Committee: Evidence
10 September 2008 Dr Philip Tabbiner and Andrew Latham everyone was being evaluated against the same Dr Tabbiner: That is correct. standard and if you did not meet that standard, you were not allowed to go on. It was much stricter than Q161 Paul Holmes: What parting advice would you has ever been used in the past. oVer to the new one-year contractor so that they avoid the shambles that you say was imposed on you Q155 Paul Holmes: But was that not in eVect what by the QCA and NAA? has always happened? I was a teacher for a long time Dr Tabbiner: I think, given the timetable that they and marked exam scripts for a long time. You always now face, that having a crisp, well-detailed end-to- had the team leader, an experienced examiner, who end project lined up and well documented is the only produced sample scripts and you all marked them way they are going to be successful at this juncture. blind and then compared what you had done. Why is what you have done any diVerent? Q162 Mr Chaytor: In terms of the number of scripts Andrew Latham: No, my understanding of the and the value of the contract is this the largest model that was used in the past for the NCTs was contract that ETS has taken on board outside North that the markers themselves would select the ones America? that they wanted to send in and then they would send Dr Tabbiner: Outside North America I believe that them in to their team leader. It was not the team to be the case, yes. leader who was selecting them. Q163 Mr Chaytor: And within North America how Q156 Paul Holmes: Well, perhaps I am thinking of does it compare to other contracts? the old days when they used to get together face to Dr Tabbiner: It is not our largest contract. face and do it round a table over two days. There is a huge discrepancy between what head teachers are Q164 Mr Chaytor: Do you think there are problems telling the NAHT and what you are saying about the in terms of the size and scale of the assessments that quality. How will we establish what the truth is? we have? Is there something intrinsic about the sheer Andrew Latham: That is what the reviews process is volume of assessments and scripts that we are trying for. If teachers or head teachers feel that the marks to manage in one contract? are not accurate, they send them in for the reviews Dr Tabbiner: No, I do not think so. process—to be re-evaluated. Q165 Chairman: Dr Tabbiner, you are in a sense an Q157 Chairman: Is a larger number of papers than outsider, coming from the United States, with all this normal coming back for review? experience, of, I think, 80 countries—I think Lynda Andrew Latham: I do not know the answer to that over-egged it a bit when she said 180; it is 80, is it not? because that is something that the NAA— Dr Tabbiner: It is 180. [Interruption.] Yes, we are no longer doing the Chairman: A hundred and eighty? reviews. Dr Tabbiner: That is correct. Chairman: Well, with all that experience, tell us—as Q158 Paul Holmes: Looking ahead to the next year, someone who has come to our country and whose someone else now has the contract just for one year contract has gone wrong, you have something of an while the Government rethink the whole thing. Will overview, from 180 countries; you have looked at the they use the same systems that you have set up—IT way we administer tests, and the way we test and systems, delivery, logistics—or will they start all assess: have you any general feeling about the kind over again? of testing and assessment regime that we run in this Dr Tabbiner: They are not going to be using the IT country? systems we have set up. Dr Tabbiner: Well, first I would point out that at ETS we develop and live by standardised testing Q159 Paul Holmes: Is that because you, under the each and every day, so I tell you straight out we are contract, are saying, “You’re not using our system,” proponents of standardised testing. I would also say or because they have said, “This system’s not that for the future of the tests we have, I think online working. We’ll start again”? marking—not just online training—would be a very Dr Tabbiner: I have no idea what they have said to viable way in which to do this volume and to have a the NAA. higher level of success and lower risks in the transit of materials and the training of individuals. Q160 Chairman: But Dr Tabbiner, others have told the Committee that you oVered them the technology Q166 Chairman: That is interesting; but when you and the stuV that you pioneered this year and they look at us, compared to the 179 countries you work did not want it. in, do we test and assess too much? Dr Tabbiner: When you say “they”, do you mean the Dr Tabbiner: That is really a public policy and other company? educational decision. Chairman: No; the QCA and the NAA. You oVered them the kind of technology and intellectual Q167 Chairman: But you are an outsider; you are a property that you developed over this first year, at a very experienced educationist. You have amazing price, and the QCA and NAA were not interested experience in the education world. Do you think you in that. can go to a system where you have too much testing Processed: 17-07-2009 19:04:48 Page Layout: COENEW [O] PPSysB Job: 406183 Unit: PAG2
Children, Schools and Families Committee: Evidence Ev 21
10 September 2008 Dr Philip Tabbiner and Andrew Latham and assessment, so that in a sense you teach to the to you that from a public policy or educational test and drive away from access to a broad policy perspective the Government have to curriculum? determine which suits the needs that they have got. Dr Tabbiner: We are large supporters not only of testing at a summative level, at the end, but also of Q170 Chairman: But that does not answer the formative testing, which is testing throughout the question whether this could be done locally. We process. could still have both kinds of tests but they could be delivered and dealt with much more locally, could not they? Q168 Chairman: So you do not think we test too Dr Tabbiner: You could certainly use a standardised much. test and deliver it on a local basis, and then Dr Tabbiner: No. accumulate it up from there. Q171 Chairman: What do you think, Andrew? Q169 Chairman: Lastly, do we have to have the sort Andrew Latham: Are you talking about the marking of bureaucracy we have had this year and for many of the assessment, because it is delivered at a local years? This is not company-specific. Could not the level in that sense? It certainly is a viable model and can be done. Of course, what you lose if you do that tests be delivered locally? Could we have a national is the concerns about national comparability of a test administered in schools where the teachers know purely objective set of markers marking the the pupils? The tests can be set nationally; why not assessments. mark them in the schools? Why do we have to have Dr Tabbiner: But if you then roll it up and it gets an environmental disaster of all these scripts running marked online, I think that you accommodate that. around the country? Would not it be better to do it Chairman: Right. We have had a very good session. locally, administered nationally, in a lighter way? Thank you very much for your full, frank answers to Dr Tabbiner: We certainly,as I said, believe in the use our questions. We look forward to keeping in touch of formative testing through process, but we also with you as we write up your report. believe in the value of summative testing, so I submit Dr Tabbiner: Thank you very much for your time.
Witnesses: Isabel Nisbet, Chief Executive, and Kathleen Tattersall OBE, Chair, Ofqual, gave evidence.
Q172 Chairman: May I welcome Kathleen Tattersall that was laid down. However, I must say, as an and Isabel Nisbet of Ofqual to our proceedings? You evidence-based regulator, that there were many are not as unfamiliar to the Committee system as the processes in the national curriculum tests this year previous witnesses, so we do not have to ride you that were successfully delivered. You heard about through the process. However, Kathleen, I will give some of them earlier. They included the delivery of you a chance to say a few words to open the session. the test papers themselves, the printing and delivery You have been sitting here, listening to the of 10 million of those papers to schools and the proceedings, and, knowing your background and administration of the tests by the schools that of Isabel, I know that you will have looked at themselves. So the pupils at that point had had a every bit of evidence that we have had in this inquiry good experience. I think that it is also important to already, so you will be well primed. Kathleen, may I note that the plans that were in place for controlling ask you to say a few words to open the session? the quality of marking were better than in previous Kathleen Tattersall: Thank you very much, Chair, years. However, the marking process is the heart of for inviting Ofqual to give evidence today. As you testing and examinations. Communications with know, we are the new regulator for examinations and schools and markers and the data entry processes all tests in England. In our interim form—that is, before went badly wrong. We have set up an independent inquiry, as you know, led by Lord Sutherland. He the legislation governing us is passed—we are part of will be looking at the whole of the processes, QCA. We began our operations on 8 April 2008, including our actions as the regulator. The DCSF with myself as the chair and Isabel as the acting chief has also asked Lord Sutherland to report to it on executive. Before Ofqual came into being, regulation areas outside our remit. We are continuing to was undertaken by the regulation and standards monitor the national curriculum tests and NAA’s division of QCA and Isabel was the director of that management of the review process. However, from division. This summer, we have monitored both the the processes that we have observed so far, we believe national curriculum tests and the A-level and GCSE that the quality of marking is as good as in previous examinations. We have had a very successful years. We will, of course, be looking carefully at the summer with A-levels and GCSEs. Sadly, that was reviews. Going forward, the priority is to regain the not the case with the national curriculum tests, confidence of teachers, pupils and the public in where there have been some very well documented national curriculum tests. To do that, we will be problems. We share this Committee’s looking carefully at the 2009 series and monitoring disappointment that the delivery problems were it carefully to ensure that pupils get what they such that many students are still waiting for results, deserve. We want to ensure, as far as we can, that the as was said earlier. It is unacceptable that pupils and problems that have been experienced this year do schools did not receive those results in the time scale not recur in 2009. Thank you, Chair. Processed: 17-07-2009 19:04:48 Page Layout: COENEW [E] PPSysB Job: 406183 Unit: PAG2
Ev 22 Children, Schools and Families Committee: Evidence
10 September 2008 Isabel Nisbet and Kathleen Tattersall OBE
Q173 Chairman: Thank you, Kathleen, but you have Kathleen Tattersall: The examination system is not said anything about what went wrong. You slightly diVerent. Certainly, the awarding bodies do heard the evidence that was given to the Committee not operate under the same contractual arrangement just now and you know about the evidence from the with a local education authority as ETS did. In terms QCA. Who do you think made things go wrong? of the contract, the events that were described Was it an ineYcient contractor or too much happened before I joined Ofqual, but my colleague interference from the NAA and the QCA? Isabel might be able to throw a bit more light on Kathleen Tattersall: The evidence that we have all the issue. heard today is the first time that I have heard ETS give its side of the story, so there is clearly something there to be absorbed. But from my point of view not Q175 Chairman: If that contract had been changed only as chairman of Ofqual, but previously as the dramatically, would you not have been shocked? chief executive of a very large awarding body—the Kathleen Tattersall: I would not expect a major part AQA—I know that at the heart of any testing and of the requirements to be changed unless it was clear examination process are the communications with that the process was not working as it should do, in markers, the management of those markers and the which case it would be very foolish to continue with management of the scripts. In terms of the problems whatever was there in the contract. There would that we experienced, the way in which markers were need to be in a place a fallback system to enable all managed and dealt with throughout the marking the processes to go forward. If online training caused process was very much at the forefront. It is diYcult a problem—with your permission, Mr Sheerman, I to recruit markers, and you always have to be shall ask Isabel to talk about that—there would have acutely aware that these are professional people who been no point in persisting with it because we might want to do a good, professional job, but that they are well lose the very people whom we wanted to retain doing that job over and above the job that they hold in the marking system. Throughout an examination as teachers, so you have to treat them in a very cycle, including the awarding bodies as well as the professional manner. Some of the stories that we national curriculum tests, things do go wrong and have heard would suggest that markers were very one has to be fairly nimble footed to change tack to upset by the way in which they were treated, so that accommodate that. Things do not always go as is one of the aspects. There are also the basic smoothly as we plan. As for online testing, I would logistics—the movement of scripts from schools to rather Isabel commented on that. the markers concerned. Again, there was some Isabel Nisbet: In February 2008, Ofqual, the evidence that the scripts in particular subjects did regulatory part of QCA, attended a meeting with the not make it to the markers of that subject and had NAA and ETS when they reported back on a pilot to be redirected. There was also evidence that some that they had been doing on diVerent aspects of scripts did not make it at all to markers. I would have quality assurance. thought that one of the basic requirements of any system of assessment and examination is that there is a checking process to ensure that everything is out Q176 Chairman: What date was that? with the markers. So the logistics around script Isabel Nisbet: It was 21 February 2008. On that movement—getting the scripts back to the occasion, one of the things reported back on was the warehouse and checked for marking—and the online trials of the online training of markers. The handling of markers are at the heart of the problems. response of the markers to that was poor. The But I would emphasise what I said in my statement, conclusion at the meeting was that it would be better which was that as far as the training of markers was to give them the option of face-to-face training. The concerned—regardless of what was said about regulatory part of QCA and I agreed with that whether it should be online or face to face—what we judgment. The reaction to the online training was observed was very good, and that has led us to the not good. In addition, the regulator was not view that the quality of the marking should be okay. responsible for the detailed scrutiny of the contract, but we were monitoring how everything was going Q174 Chairman: But Kathleen, with great respect, overall. A lot of things that went wrong later were you were the chief executive of the AQA and you not about training, but about parcels not being heard representatives of a similar organisation checked before they were sent out, for example. I giving evidence to the Committee just now, saying should say that, as regulator, we had great diYculty that they had signed a contract and that that getting good data on where all the scripts were. contract was quite dramatically altered, which made it diYcult for them to deliver on the basis that they thought they were going to deliver, particularly in Q177 Chairman: Who was responsible for the details terms of the training of the markers. Was it your of the contract, if you were not? experience as the chief executive of a major Isabel Nisbet: The contract was between the NAA examination board that that is normal? Would the and ETS. We were not supervising the contractual QCA or the NAA dramatically change the nature of aspects. Our duty, as set out in our regulatory a contract after you had signed it, when you were framework, was to look at the quality and to make actually running the operation? Did that happen to sure that the tests were delivered in a way that would AQA? command confidence. Processed: 17-07-2009 19:04:48 Page Layout: COENEW [O] PPSysB Job: 406183 Unit: PAG2
Children, Schools and Families Committee: Evidence Ev 23
10 September 2008 Isabel Nisbet and Kathleen Tattersall OBE
Chairman: Thank you for those opening responses. wrote to the Department, it was clear that we had real concerns that we were not going to meet the Q178 Mr Pelling: I shall ask some basic parameter deadline. questions. What was the role of the regulator in the actual procurement of the contract with ETS? Kathleen Tattersall: The procurement process was Q181 Mr Pelling: So what do you feel are the most long before Ofqual came into existence. Again, I ask significant causes of the delay in delivering the Isabel to comment on that. results? I have heard some clear diVerences in nuance Isabel Nisbet: As regulator, my part of QCA did not between you and the earlier witnesses. have any formal part in the contractual process. The Kathleen Tattersall: Yes. I only today heard ETS proprieties of the contracting were overseen by the speaking directly about this, and I do not OYce of Government Commerce, which reported particularly wish to comment on that or get on it several times. Our responsibility was to make involved. From our point of view, the marking was sure that our regulatory requirements were met. slower that it ought to have been, scripts did not They are that the levels are properly set; the move around as quickly or eYciently in the system standards are properly made; the quality of the as they ought to have, and the checking of the marking is good; and the tests are delivered in a way marking did not appear to be as eYcient as we would that meets the standards of previous years. Clearly, have wished. Isabel may be able to add some other a lot of the things in the contract were to do with points from her detailed operational side, but that is that, but the actual contracting was not overseen my view as the chair and as someone who has been by us. involved in examinations. Isabel Nisbet: One other factor is that the Q179 Mr Pelling: I presume that, if the processes of complicated nature and the logistics of what the the procurement were so limited, there would have markers were being asked to do on the system was been no oversight role for the regulator. just too heavy a load to bear. They were having to Isabel Nisbet: It is fair to say we oversaw the whole enter every item separately—millions and millions of delivery of the tests, but we were not monitoring the items—and that system was too complicated and too contract. We were not a contract checker. Our burdensome for them. I know that that is a requirements are set out in our regulatory conclusion that NAA has also come to. Expecting publications, and those are the issues that we were markers to make an entry for every single question, checking against, not the terms of a contract. every single item, on a system that they found quite Kathleen Tattersall: It should be said that our diYcult to work with just imploded. That is one of regulatory role was in respect of NAA. That is where the major burdens on the system. our regulation falls. NAA managed the contract with ETS. That is the line of accountability. Q182 Mr Pelling: So the emphasis on good quality Q180 Mr Pelling: When did you, as regulator, training is important? become aware of the problems with the testing cycle Isabel Nisbet: Yes. in 2008? Kathleen Tattersall: The testing cycle itself is clearly between May and June: students take the tests in Q183 Mr Pelling: You stated that you had concerns May and the marking follows. Certainly at a fairly back in February. Do you feel that there should have early stage in that process, towards the end of May, been earlier intervention on the contract, as you were we were very aware that there were problems that aware in February—having looked at the pilot with were likely to slow down the marking process: the online approach—that the training process was getting the scripts to the correct markers, getting on in trouble? with the marking, and getting the scripts back. At Isabel Nisbet: I think that what we concluded was the time that all those concerns were raised, we had assurances that the date of delivery of results back to that even at that late stage NAA was right to ask for schools was not in question—that it would be met— the option of face-to-face training. I do understand but by the beginning of June we were very concerned that that would have meant setting up a lot of things about the resources required to deliver the tests on quite quickly and that it would have been a risk, but time. Isabel wrote to David Gee—I think it was on the other risk would have been the roll-out of the 3 June—about the delivery of tests, and as a result eVects of the pilot, when a lot of the markers were NAA made available more resources to back up the not reacting well to the online training. What they ETS operation. If you go back to earlier points in the were being trained to do was quite burdensome, and, cycle, when we expressed concerns about with hindsight, one lesson for the future is that item- communication with markers and the resources level entry is not really practicable. available—this is going back to January 2008— again, NAA put in a lot more back-up to help the ETS. So we played a role as a regulator. So NAA was Q184 Mr Pelling: The timetable for training, in very responsive to many of the concerns that we had. terms of altering the approach and putting proper In answer to your specific question, it was around training in place, was likely to have been a three- the beginning of June that we were really voicing month run-in, rather than one month, which is those concerns, and indeed by 16 June, when we something that we heard earlier. Is that correct? Processed: 17-07-2009 19:04:48 Page Layout: COENEW [E] PPSysB Job: 406183 Unit: PAG2
Ev 24 Children, Schools and Families Committee: Evidence
10 September 2008 Isabel Nisbet and Kathleen Tattersall OBE
Isabel Nisbet: Yes, I cannot speak for the details, but the contract by NAA. NAA was responsible for the I could certainly look into that further and come contract and the word on the street was that it was back to the Committee if you would like us to look made at a much lower price than any of the other at that.4 bidders. Indeed, we heard—astonishingly—that the American team lost £50 million on the contract this Q185 Mr Pelling: We would be grateful for that. So year. The NAA appointed these people and quite the QCA is blameless in all this, is that right? soon worried about whether they could deliver, so it Kathleen Tattersall: I really find it very diYcult to is not surprising that it put a lot of support in early. answer that question, which is one of the reasons we The two things cannot be separated, can they? have asked for an independent inquiry. Kathleen Tattersall: ETS has a very good reputation Chairman: This is an independent inquiry. internationally—I have to say that. Certainly, it was Kathleen Tattersall: Sorry, Chair. I do appreciate an organisation that I was aware of as a chief that and I appreciate the authority of this executive and it has enormous experience: not just in Committee, but in terms of who did what, it is the states but, as it made clear in its evidence, across somewhat diYcult to unravel the process. There are the world. So I rather suspect that it was somewhat questions that we would ask regarding who did what of a surprise that, in this particular contract, things and when, and what our own role was in that. As to began to go wrong. Again, however, it may be that whether anybody comes out of this blameless, it is Isabel, from an operational perspective, has a better quite possible that there will be some responsibility idea than I have about the answer to that question. borne by all the parties concerned, because Isabel Nisbet: The first year of a new contract for any everybody had a responsibility for diVerent aspects organisation is a heightened risk, and it was made of what went on, both in terms of the contract and explicit between us and NAA that we would expect the management of the ETS. NAA—part of QCA— them to take extra care in the first year of a new is responsible for that management. For our part, contract to ensure that it went well. If that meant regarding the regulation that we applied, what we outposting people to get the right results for pupils, want to know and what we want an independent then that was the right thing to do. There were person to judge us on, and you may yourselves make similar problems, tensions and high risks in the first this judgment, is: whether we asked the right years of other contracts in history. questions at the right time; whether we evaluated in a proper manner the answers that we had; and, lying Q189 Lynda Waltho: You quite rightly said earlier, behind that, whether the management systems for Isabel, that the regulator does not have a direct role giving us all the information were appropriate to the in procurement. However, Ken Boston did say that task. That in itself might be one of the problems. there is a role to advise, support and warn. I am wondering why—perhaps you have some ideas on Q186 Mr Pelling: Do you have a view on whether the this—the regulator waited until February 2008 to degree of change that was required within the advise, support and warn that markers would not contract is something that is experienced elsewhere, like online training, and also why they did not internationally? advise, support and warn about how diYcult it was Kathleen Tattersall: That is something that I cannot to recruit markers in particular areas. Why do you comment on; I am sorry. think that that advice did not come sooner, rather than later? Q187 Mr Pelling: To the extent that you feel that the Isabel Nisbet: There had been regular exchanges NAA would have had a more hands-on role in this between the regulator and NAA about aspects of the process, do you think that it is inevitable that if there tests. Those exchanges did not start in February; is culpability, the NAA is going to carry more they go on right through the year, including culpability than the QCA? exchanges about aspects of the tests to do with Kathleen Tattersall: The NAA is the body that was markers. There was a clear set of pilot exercises that responsible for the management of the contract. In NAA, with our agreement, had asked ETS to terms of the response to our regulation, as I conduct, about the quality of marking. There were a indicated earlier, the NAA put in an enormous number of things that ETS was trying out. One of amount of resource to back up the ETS to deliver the them was this online training of markers. That issue contract. That was at an early stage, as well as later came to its culmination in February, but it had been in the process. From our point of view, the NAA has going on for several months before. It was the report done all it can to avoid the problems that happened of that pilot scheme that was presented to the in June and July. Clearly, what it did was not enough meeting in February, which I referred to. So things to prevent the problems becoming manifest, but it had been going on before February. I was surprised was very responsive to the problems that we to hear this morning that our guests from ETS were highlighted and it put in the resource where it could. surprised at the issues around recruitment of markers because, rather like Mr Chaytor, I felt that Q188 Chairman: Okay. A cynical outsider might say, that was a well-known background fact about the “The NAA would do that, wouldn’t they?” system. Everybody was surprised when a new player from America, with no experience of the UK, was given Q190 Lynda Waltho: It just feels all a bit muddy to me, Chairman. For something so important, there 4 See Ev 31 just seems to have been a very big delay in knowing Processed: 17-07-2009 19:04:48 Page Layout: COENEW [O] PPSysB Job: 406183 Unit: PAG2
Children, Schools and Families Committee: Evidence Ev 25
10 September 2008 Isabel Nisbet and Kathleen Tattersall OBE and agreeing the basis of a contract in which online here: the deliverer, the body managing the deliverer training and marking was presumably specified, and and the regulator. We have to think very carefully as then realising that the markers were not going to like to whether that enables us to regulate eVectively. Let it. It just seems to me rather unprofessional or even me compare it with what happens in awarding irresponsible that there was such a gap. I am finding bodies. Awarding bodies might have contracts for it hard to absorb that. With all of these some aspects of their work, but the heart of what conversations and e-mails that were happening happens there—the marking process, the training of before, do you think that they were at a level that was the markers, the taking of the scripts and getting the almost unnoticed, and therefore the concern was not results back to the schools—is undertaken by the going up the chain to the people who really should body that is regulated by Ofqual. There are some know? I just cannot work out why there was such a lessons that we need to think about, such as what the gap. process of regulation should be in national Chairman: It does all seem a bit messy, Isabel, does curriculum tests. it not? Isabel Nisbet: Yes, I think that there are issues about Q193 Mr Slaughter: And who is policing whom. the number of layers in this system and whether they Kathleen Tattersall: Yes, exactly. are really working well. In what was my division, I had a team of five people looking at national Q194 Mr Slaughter: We will be coming on to quality, curriculum tests. They had day-to-day exchanges which you will have more to say about. While we are with the NAA, whom we regulate against these on the operational side, you heard—I am afraid that documents. Now, if there were concerns coming up, I missed part of the ETS session—that the changes they would report those up the line. We monitored in the contract, post-letting, were so great that it was risks, as we do in Ofqual, and there would be a time unable to cope. I am not entirely sure whether that when we would take a judgment that this matter is was simply the process of change in a short time, or really so seriously wrong that I need to raise it with that the requirements were so onerous that it was not David Gee, for example, and there were several resourced or equipped to do that. Can you comment occasions when I did that. However, I understand on that at all? Can you say whether you think that is what you are saying and we are learning lessons from right and whether that is the main explanation, and this situation, too. even if it is, do you think that the contract was under- Chairman: What we are getting the feeling of is that resourced for what ETS was originally asked to do, it was all a bit cosy. When we did the Building or whether it was the right sort of contractor and had Schools for the Future inquiry, in a previous the capability to do what was asked? Committee, we learned one real lesson: if you are Kathleen Tattersall: If I can answer the last part of going to build a decent school, you need a good that first. As a huge organisation, as we know ETS client. So it is all very well to blame the operator— to be, with a great reputation in America and the person who is building the house—but if you are elsewhere in the world, I would have thought that it not a good client, you do not know what you want, was the sort of organisation that could manage a you do not set the parameters and you do not do all contract of this size. that well, then you can end up in a bit of a mess. I hope that when you contribute to this whole evaluation, you will not spare the client as well as the Q195 Mr Slaughter: That seems to be an important operator. point: all the things that went wrong, ETS was capable of getting right. Kathleen Tattersall: I would have thought that any Q191 Mr Slaughter: Do I understand that you are experienced organisation dealing with examinations saying that you are too remote from the procurement and tests should, to use an earlier phrase, be nimble- process to answer questions about ETS’s footed enough to accommodate changes. No employment and performance? examination series goes to plan. Things go wrong Kathleen Tattersall: I do not think that it would be and they go wrong at the last minute. You have to right for the regulator to be part of the procurement put in all the resources at your disposal to get things process itself. It would be proper for the regulator to right in order to meet the deadlines. In my own have expectations of the nature of the services that experience, I have been there. You have 48 hours would be required in any given contract, but it would before publication and you put in everything in not be right for us to be part of the actual order for people to get their results. procurement process. As Isabel said, the OGC oversees that process. Q196 Mr Slaughter: So unless it was out of character, what happened? Is it because it was under- Q192 Mr Slaughter: In terms of commenting now resourced or because it was asked to do something and helping us to understand what went wrong, is it that it was not prepared to do under the terms of this reasonable for us to be asking you such questions? contract? Kathleen Tattersall: We find it very diYcult to Isabel Nisbet: I cannot speak for the changes to the comment on the procurement process for the reasons contract because the regulator was not involved in that we have given. In terms of what we can those processes. What I can say is that a lot of basic comment on, it is the aspects of the tests that we have things did not work, not just about markers but regulated. As I said to Mr Pelling, the regulation falls about systems, management information, the fact on the NAA. As Isabel said, there are several layers that it has been so diYcult to know where all these Processed: 17-07-2009 19:04:48 Page Layout: COENEW [E] PPSysB Job: 406183 Unit: PAG2
Ev 26 Children, Schools and Families Committee: Evidence
10 September 2008 Isabel Nisbet and Kathleen Tattersall OBE scripts are, even to today, and the fact that the Q201 Mr Chaytor: But you were present or some communication was so poor. I would make a staV from the regulation and standards division judgment that those were systemic things that went were present? wrong throughout. They are not all a matter of Isabel Nisbet: Yes, we were. If you ask me whether resource. the NAA reported to us that ETS had said that this would lead to a long delay, I do not recall that, but I will check.5 Q197 Mr Slaughter: But they do not just happen. Isabel Nisbet: No. Q202 Mr Chaytor: My next question is, did you or your representatives or the NAA ask what would be Q198 Mr Slaughter: What we are trying to get to is the time implications of scrapping the initial plans this: whose fault is it? for online training and suddenly moving to face-to- Isabel Nisbet: I cannot say that those were all the face training, given we are talking about— result of contract changes. I have no evidence for Isabel Nisbet: I do not recall that we did. that. I cannot comment on those. If you ask whether the problems of data, not opening parcels and poor communication with the markers were because of Q203 Mr Chaytor: Can I ask about the regulation lack of resources, that might have contributed to and standards division? The functions of Ofqual are them. All I know, as a past member of the executive essentially the same as the functions of the division. of the QCA, is that the rigour of the procurement It has just been taken out of the QCA and made process, which I was not involved in, was of a very more independent. I am interested in the relationship high quality and was overseen by the OGC. between the regulation and standards division and the NAA, because in terms of Ofqual, you have been Q199 Mr Chaytor: Can I come back to the meeting in existence only since April 2008, but the work of of 21 February, when the results of the pilots for the division has been going on for a number of years online marking were reported? It is agreed by both and the NAA has been formally in existence since sides—ETS and the NAA—that that was a key April 2004. What is the exact relationship between moment when the decision was taken to scrap the what are essentially two divisions of the QCA over online training and move to face-to-face training. the last four years and particularly in terms of ETS’s whole defence seems to hinge on the fact that considering the standards implications of the it was told by the NAA to scrap the online training processing or the progression of particular and that lost it between three and five months in the contracts? At what point, as the head of the whole process. It did not explain to us earlier this regulation and standards division, were you able to morning that the reason why it was told that was the say to the NAA, “We think there are standards poor response of markers to the online training. At implications in the way this contract is being the meeting of 21 February, did ETS point out that implemented”? it would cause huge delay to the project if online Isabel Nisbet: When regulation was passed to the training was scrapped in favour of face-to-face QCA, there was a framework that was set out, that training? Did either you, as the then head of the we regulated the NAA as if it were an awarding body, regulation and standards division, or the NAA make but within the QCA; so on particular issues of any comment on that in terms of the likely impact on standards, like, for example, the level setting, what meeting the 8 July deadline? would happen is that there would be a lot of Isabel Nisbet: The answer to that, Mr Chaytor, is discussions between my staV and the NAA about the “Not to my memory.” I was at that meeting and it technical parts of level setting. The NAA would was a presentation by ETS of the results of the pilot. make a proposal to Ken Boston, then I and my staV There was some discussion with the NAA and would come in afterwards and say what we thought ourselves, and then we as a regulator went away to about it from the point of view of the regulatory decide what we thought about it. There were some division, and then an overriding QCA decision other things as well as that point about the online would be made, Ken having listened to the advice training of markers. I do not recall that being said, from my people and from the NAA. The diVerence but we will go back now and check the records to see now, with Ofqual, is that we are doing the same whether there is anything I can say to— functions, but with quite diVerent governance and accountability; so now that does not happen that I report to Kathleen and we make our separate Q200 Mr Chaytor: Okay; and do you recall any judgments and communicate them separately. subsequent occasion when ETS was absolutely specific about the scale of the delay that would be caused if online training was replaced by face-to- Q204 Mr Chaytor: In terms of the standards face training? implications of quality of marking, would you have Isabel Nisbet: I have just been passed a note. In fact, been expected to report your concerns about that to ETS was not at the particular meeting that I was the chief executive of the QCA? If you had concerns talking about. I am sorry—I misled the Committee. about marking or the training of markers, would It was the NAA reporting what it had said to the that have been an issue for you? NAA, so it was a meeting largely with members of staV of the NAA and us. 5 See Ev 31 Processed: 17-07-2009 19:04:48 Page Layout: COENEW [O] PPSysB Job: 406183 Unit: PAG2
Children, Schools and Families Committee: Evidence Ev 27
10 September 2008 Isabel Nisbet and Kathleen Tattersall OBE
Isabel Nisbet: Yes. I would first of all normally have lots of exchanges of e-mails about particular issues notified David Gee if things were escalated to me and the responses to those particular issues. It was from the team. I did that on several occasions on not as if there was a great big hole and a great big diVerent matters, and they took action each time. If silence at that point. I think that the importance of I felt it was getting more serious than that, the next late May and June is that we were now in the period step would be for me to raise it at the top of the QCA. when the pupils had taken the papers and it was getting to the marking period, and the getting of the Q205 Mr Chaytor: But coming back to the timeline scripts to markers, and so on. That is when things and the 21 February meeting when you gave a report began to escalate, and the problems really did begin back of the concerns about the online training of to pile up. Earlier, we heard the reference to the markers, you said you wrote to David Gee on 3 June helpdesk and the number of e-mails that were being about the delays. There is a long time between 21 received about particular problems. February and 3 June. Was it not until 3 June that you were seriously concerned about the possible delays Q209 Mr Chaytor: How long did it take to conduct because of what you had seen in respect of the the face-to-face training of all markers? Presumably, training of markers? it did not start until well after 21 February. Isabel Nisbet: There were other exchanges with the Isabel Nisbet: I do not know the answer to that, but NAA from the regulator about issues including, for I can provide it to the Committee.7 example, some of the communications issues with markers, which we wrote to the NAA about in January. Also there were some aspects about the Q210 Mr Chaytor: That seems an incredibly short load that the data system for marking was having to time scale, given that markers were being trained bear. As a result of each of our interventions the only after 21 February for a set of assessments that NAA did things: for example it put some extra had to be done and dusted by July. resource in on the communications side in January, Kathleen Tattersall: Can I just make a general point, and did some load testing that we had asked for. So although the note that is being passed to Isabel in each case there was an exchange when something might have the answer? Generally, when we talk happened. What happened in June was that after the about training the markers, we are actually talking young people had sat the tests and there had been about training them on the live scripts, as well as any reported delays in getting to mark them there began preparation that might be done for that; it is the to be accumulated reports of things going wrong. As point when markers are standardised, when they are the chairman said, in mid-June we took the view that there with live scripts and they know what the there was overall now on balance a high risk that the standard is. It is not as though the training had to results would not be delivered on time. That is when take place at the end of February; there is a longer we did ask Ministers. period than that to regroup your resources and look at how a diVerent system might work. But Isabel has Q206 Chairman: Just to nail this down, can we have an answer that has been given to her, which might a list of the number of interventions you made, and very well contradict what I have just said. when you made them? Isabel Nisbet: Not at all. I am advised that the bulk Isabel Nisbet: Yes, we can send that.6 of the training was in early May. My own staV attended 18 of the meetings—a lot of them were on Q207 Chairman: Because what is worrying me, as diVerent subjects and happened concurrently. There someone who is listening to both lots of evidence, is was one day per session, so markers would go to a that there were reports in the press that things were one-day session. A lot of that was at weekends, and not going well quite early on. You as regulator, we we observed 18 of those meetings. would have thought, would have been on the case. Kathleen Tattersall: If I can just draw on my Isabel Nisbet: Yes. previous experience, Isabel has just confirmed that when you get all your markers together—Mr Q208 Chairman: It seems that it is in the May–June Holmes has been a marker, so I think he will period that suddenly everybody realises it is all going understand what I am saying—you get the scripts in to hell in a handcart—Dr Tabbiner flying over in and you get all your markers organised into groups May, and by June saying, “We want to walk away with senior markers. They go through a session of from this contract.” You as regulator should surely marking the scripts against the mark scheme and have known that by then. they have a discussion about the mark scheme. They Kathleen Tattersall: In answer to Mr Chaytor’s come away from that able to understand and apply question about what happened between February the mark scheme. As was described earlier, you then and June and your request for a list of the exchanges, have an exercise on the process of the marking, you will find that there were almost weekly which enables you to be sure that the markers are exchanges about problems, but that those problems keeping to the standard. In the three months, or were isolated problems that we dealt with. It was not whatever it was, between the stopping of the online something that sat around and suddenly appeared marking and the point where markers were being again in June. Things were being handled on a day standardised, it was possible for the alternative by day basis by Isabel and her staV and there were arrangements to be put into place.
6 See Ev 32 7 See Ev 31 Processed: 17-07-2009 19:04:48 Page Layout: COENEW [E] PPSysB Job: 406183 Unit: PAG2
Ev 28 Children, Schools and Families Committee: Evidence
10 September 2008 Isabel Nisbet and Kathleen Tattersall OBE
Q211 Mr Chaytor: There is an issue about the marking. We were very comfortable with some training involving the standardisation of marking, things, but on the question of hope over experience, but was there not also an issue about the training when we began to observe what was happening with dealing with the new software? Isabel, you expressed the slowness of the marking and with the movement concern about the sheer volume of input that was of the scripts, our experience said, “We are really required and you indicated that the software was concerned that we are not going to meet that date.” over-ambitious. Isabel Nisbet: Yes. Q215 Paul Holmes: These test results are crucial for schools. Teachers get their performance-related Q212 Mr Chaytor: So when was that training done? pay—“professional pay” or whatever it is called At what point were markers trained in the handling these days—on the basis of results. Schools can be of what you consider to be over-ambitious software? praised or threatened with closure on the basis of Was that done in early May as well? results. Ofsted, when it turns up to inspect, has Isabel Nisbet: I believe it was at the same time, but I number-crunched all this and come along with will check on that and come back to the Committee.8 preconceptions on the basis of these results. Are you confident that the results are accurate this year, or, as Q213 Mr Chaytor: Finally, with the benefit of even ETS claims, better than in previous years? hindsight, do you think that Ofqual should have Kathleen Tattersall: We have said that we believe issued a warning notice earlier than it did about the that the quality of marking is at least as good as in potential danger of not meeting the deadlines? previous years. That is our current position. The Kathleen Tattersall: If I may, I will answer that. Up judgment is based on the training, which we have to the point where the pupils took the tests, things mentioned, on the level-setting, all of which was went relatively smoothly—there might have been observed by Ofqual staV, and on a realisation that problems, but they were addressed. So we are talking the markers themselves are not new people, but about the approximately six-week period from the people who have been in the system for quite some point where the pupils have taken the test to the time. New people will, of course, join the system, point where results come out. It is about three weeks but, generally speaking, we are talking about people into that period that the concerns that we expressed who have marked before. We are very aware that begin to manifest themselves. That was an teachers are making other statements. I have had appropriate time for us to begin to say what we did; meetings with various teaching unions. I am aware it would have been quite inappropriate to make a of the great concerns that schools have, and I very prejudgment at the very beginning of the marking much sympathise with them. However, we, as process that things were going to go wrong, because Ofqual, have to distinguish the diVerence between we did not have the evidence at that point. It is when the quality of the marking and the quality of we have the evidence that the scripts are not getting everything that surrounded it, which is what we are to the markers and that the markers are bogged trying to do. We will monitor very closely the review down by the itemised material that they have to process, which began in July, because those schools record that we begin to say, “Are we going to make that had their papers and results back at that point 8 July?” At the beginning, we were being given would have put in their request for a review. Schools assurances that we would, but by the end of May, that are getting them back still have the time to do beginning of June, we began to ask, as we did of so, too. We will look at what goes on, and we will be David Gee, “Are we going to make that date?” We very interested to know the proportion of level got some further assurances, but we were suYciently changes, because it might lead us to a diVerent view. worried to put in place an accountability meeting However, we want to keep ourselves to the evidence, with the NAA on, I think, 3 July, which I chaired. and, at the moment, the evidence is that the quality That was an indication of the seriousness with which of marking is at least as good as last year. we were taking things. In June, we made those noises to the NAA and to the Department, and Q216 Paul Holmes: But why are you so confident, unfortunately our fears proved to be well founded. when the delivery and the logistics were a shambles; However, we were given assurances that the deadline when the training got oV to such a bad start; when would be met. It was a much later stage when other there were lots of stories in the educational press people began saying, “No, it is not going to be met.” about experienced markers resigning, saying, “I’m not having any part of this”; when people had to be Q214 Mr Chaytor: Was it the triumph of hope over recruited at the last minute in the summer—after 8 experience, given everything that you had observed July, when everything should have been done and over the previous few months with the operation of dusted and they were dragging people in to catch up ETS? on the backlog; and when unmarked papers are still Kathleen Tattersall: This is a situation in which some turning up? It beggars belief that the quality will be things were good. We observed some things that as good as in previous years. were good, and in spite of what has been said about Kathleen Tattersall: As I have said, we are trying to the organisation of the training, the training itself draw a fine line between all those surrounding was good. We were also impressed by their online problems, which caused a lot of anguish for teachers benchmarking and the standardisation of the and others who were handling the situation, and the quality of the work by people who were in the 8 See Ev 31 system. Those people in the system, whether they Processed: 17-07-2009 19:04:48 Page Layout: COENEW [O] PPSysB Job: 406183 Unit: PAG2
Children, Schools and Families Committee: Evidence Ev 29
10 September 2008 Isabel Nisbet and Kathleen Tattersall OBE were recruited later or at the normal time prior to the do that by working with the NAA as we do with the taking of the tests, will have received the same awarding bodies in GCSE, A-level and vocational quality training. Our belief is that that quality areas to identify the risks and agree between us the training and the systems to weed out markers who management of those risks, and we also need to did not continue to meet the national standard, consider the systems underpinning the delivery of namely the benchmarking exercises, are of good the examinations. If we consider the documentation quality and will have assured the system that the that Isabel held up earlier—the codes of practice and markers who mark are good markers and that the so on—the emphasis is very much on the quality of marking should be trusted. We might find problems the test, the quality of the marking and so on. We at the review stage. If there are, clearly Ofqual will have to broaden that, and we do not need legislation look at them, and we are open to revising our to do so. We need to do that anyway, regardless of judgment, if we have to. when the legislation comes through. As a regulator, we expect to have greater powers to direct an Q217 Paul Holmes: We have a non-scientific sample awarding body to take a particular course of action from the National Association of Head Teachers. Of in the interests of maintaining standards, and in the 26 reading papers, 22 were marked incorrectly; all interests of the pupils and the students who do the English papers were wrongly levelled; 14 out of 29 tests and the examinations. At the moment, we do papers were returned; and so on. Is there a not have those powers and my expectation, as chair, qualitative assessment at that stage to show whether is that we would be given powers of that kind, so that more complaints and more papers are being returned we could exercise them in the interests of pupils and than at the equivalent period in previous years? students. Kathleen Tattersall: My understanding at the Lynda Waltho: Thank you. moment is that we are not in a situation in which Isabel Nisbet: We have not waited for anything, but more papers are requested for review, but I would we have already asked for extra resources from the have to ask Isabel for the detail. Department to beef up our resources for looking at Isabel Nisbet: I am advised that normally in previous the next year of the NAA’s management of the tests. years about 1% of the papers come back with It will clearly be a new contract, probably for one requests for review. So far, it looks like being a bit year. There is risk there, and we need to have people above that, but we will not know until about a to look at the systems. The other thing is that at the couple of weeks down the line, because all the end of July I put in place weekly accountability September ones are coming in now. The NAA has set meetings, which I chair with the NAA, to keep track up a new database. We have asked for weekly reports of exactly where things are moving, and I propose to on the outcomes of the reviews, particularly the continue those for the foreseeable future. That is an changes, and we expect to see them this week. escalation. That is not a normal regulation, but these are not normal times, and so we will be continuing Q218 Paul Holmes: So there appears to be an with that. increase already. Isabel Nisbet: A slight increase, yes.
Q219 Paul Holmes: But a number of the heads in the NAHT report said that the new system this year Q221 Chairman: Kathleen and Isabel, when did you discourages schools from sending as many papers make the decision to call in Lord Sutherland to back. Under a new system that discourages sending conduct an inquiry? papers back, there have been more than previously. Kathleen Tattersall: It was following my meeting on Kathleen Tattersall: I understand that from my 3 July with the NAA, when it was apparent that the 8 meeting with the NAHT, and we have raised the July date was in great jeopardy. It seemed to me that issue with the NAA. All “artificial barriers” to given the importance of the tests to pupils—and also requesting reviews have been removed, so the NAA to teachers, schools and parents—that this was an responded to that. issue that was so serious that we needed to try to bottom out all the issues, including our own role. As Q220 Lynda Waltho: All that has happened so far a new independent regulator, we want to learn from clearly has implications for you, as a regulator. Once the experiences of this year and be able to apply you are on a statutory footing, what might you be ourselves in the future in ways that will be very able to do to avoid a similar situation next year or in eVective in all regards. So we felt it best not to have subsequent years? What have you learned? just an internal inquiry about what had gone wrong, Obviously, I understand that we have to wait for but to call in an independent person of standing— Sutherland, but how you can avoid such problems in Lord Sutherland—to take evidence and to reach an the future? What could be your role? objective judgment as to where the balance of Kathleen Tattersall: If I can answer first, perhaps responsibility lay for particular parts of the process Isabel can then come in with some of the detail. I going wrong. We also asked him to say—as part of hope—this is regardless of whether the legislation is the brief—whether Ofqual had acted in a timely in place—that we can look at how we manage the manner. Did Ofqual ask the right questions? Did it regulation of the NAA in a more systematic way evaluate the answers in the right way? Could Ofqual from next year. Isabel referred earlier to treating the have done more? That was why we did that, and that NAA as an awarding body. I think that we have to was the timing. Processed: 17-07-2009 19:04:48 Page Layout: COENEW [E] PPSysB Job: 406183 Unit: PAG2
Ev 30 Children, Schools and Families Committee: Evidence
10 September 2008 Isabel Nisbet and Kathleen Tattersall OBE
Q222 Chairman: What were your communications to be inquired into, too. That demonstrates a with the ministerial team, before you made that strength on our part, because we are willing to judgment? subject ourselves to public scrutiny as well as having Kathleen Tattersall: Before we made that judgment, others subjected to that public scrutiny. we did not have any communications with the ministerial team. Clearly we informed Ministers, Q227 Chairman: You did not think that this because we have always taken the view that there Committee had the ability to do that? should be no surprises, and it would be wrong for us Kathleen Tattersall: At the time that we set up the to do something so momentous that it became a inquiry, there was no suggestion that there was also surprise in the public domain without alerting going to be a Committee inquiry about this. people. But we were very conscious that that was our decision and that we were going ahead with it. Q228 Chairman: You must have known that the Isabel Nisbet: I certainly told oYcials that that was Select Committee on Children, Schools and Families what we had decided. As the Committee may know, was bound to investigate. after we had made that decision Ministers decided Kathleen Tattersall: I am very pleased that this that they too would like to ask Lord Sutherland to Committee is looking at the issue, but I still think report some conclusions to them and so, with Lord that it was right and proper for us, as the regulator, Sutherland’s agreement, the approach taken was a to ask a person of such standing as Lord Sutherland, single inquiry with two reports. from outside the system, to look at the issues and to report to us. Q223 Chairman: There was communication with the Department, which would have informed Ministers. Q229 Chairman: Kathleen, Lord Sutherland might So Ministers knew what you were doing. be outside the system, but he does not have the Isabel Nisbet: Yes. powers that we have as a Select Committee. Kathleen Tattersall: Indeed not. Q224 Chairman: And they decided to join you. Isabel Nisbet: Yes. Q230 Chairman: Indeed, people can give evidence to us in a free and frank way here, that they cannot give Q225 Chairman: I take it that Sutherland is getting to him. You will know that, because the legal basis his teeth into it already. Is he not constrained by the of our relationship with the witnesses is diVerent. evidence that we had from the contractor, the ETS, Kathleen Tattersall: Yes, I understand that that it is still constrained by the contract and is not completely, but I think that asking Lord Sutherland fully liberated to frankly discuss with Lord to look at the issues and report to the Ofqual Sutherland what really happened? committee gives us a good basis to go forward, in Kathleen Tattersall: It was the first that I had heard terms of ensuring that our regulatory authority and of that, today, and I was really alarmed. It seems to actions are eVective in the future. me that in order to get a complete picture, Lord Sutherland—and the Committee—needs to have the Q231 Chairman: So you will be taking the report evidence from all parties. I hope that whatever the from this Committee and from Lord Sutherland very restrictions are, they will be lifted to enable ETS to seriously indeed? give evidence. Kathleen Tattersall: Indeed.
Q226 Chairman: There are great threats and Q232 Chairman: Well, when we come to discuss the opportunities around now for you, as a regulator. legislation that puts your organisation into a formal Were you conscious that when you, the new context, after the Queen’s Speech, the threat and the independent regulator, asked Sutherland to have a opportunity are that if your actions and reports are look at this that from the outside, a lot of people seen to be robust and independent, it will be a lot might see the QCA, the NAA and yourselves as easier to vote for the legislation. being a bit cosy? You have shared roles with the Kathleen Tattersall: We hope that what we have QCA—Isabel certainly has. It does look a bit cosy. demonstrated so far is that we are robust and Although you are only a shadow regulator in one independent. Indeed, I do not think that we would sense, because the legislation has not yet been made be talking about the national curriculum test in this to make you absolutely oYcial, the first thing that way, were we not in ourselves so robust and you have done as a tough, independent regulator is independent. ask somebody else to be independent. Kathleen Tattersall: In a strange way, that absolutely Q233 Chairman: But Kathleen, if we look at the demonstrates our total independence. As a regulator record, your opening remarks were very diplomatic that is quite separate from the QCA and the NAA, about the NAA. If there is not a proper and robust we believe that something quite serious has gone evaluation of not only what the contractor did, but wrong, and that we might be part of it, in terms of how the NAA and QCA operated as clients—were whether we asked the right questions, timing and so they too interfering, did they get it wrong, was there on. Bringing someone in to look at what went wrong something flawed in the process of choosing the demonstrates that we have the courage of our own contractor?—you know as well as this Committee convictions. There needs to be objectivity in that you will be dogged by a question mark over considering the totality of the evidence, and we need your independence as a regulator. Processed: 17-07-2009 19:04:48 Page Layout: COENEW [O] PPSysB Job: 406183 Unit: PAG2
Children, Schools and Families Committee: Evidence Ev 31
10 September 2008 Isabel Nisbet and Kathleen Tattersall OBE
Kathleen Tattersall: I fully appreciate that. That is Kathleen Tattersall: We are going to receive an why we want to really get to the bottom of where interim report in October and I would expect the responsibility lies for what has happened in the 2008 publication to be some time after that—perhaps national curriculum tests. We are taking this failure about November. very seriously, and we want to investigate that in the Isabel Nisbet: Lord Sutherland is reporting orally to manner that we have indicated and to place the the Ofqual committee in mid-October. So far, until Ofqual committee in a position to act. If that he sees all the written evidence that has come in, he involves taking up issues with the NAA and the has not committed himself to a final date, but I think QCA, we will do so in a very independent manner. that it will be within a few weeks after that. Q236 Chairman: Time is of the essence. Kathleen Tattersall: Absolutely. We really want to Q234 Chairman: Even if that includes robust impact on the 2009 tests, and the report should criticism of colleagues you have worked with for provide us with a basis on which to do that. many years? Chairman: This has been a very good session. We will Kathleen Tattersall: Even if that includes robust go away and decide whether we will have any more criticism, yes. sessions, and indeed, whether Lord Sutherland will come to the Committee. We look forward to meeting you again soon. Q235 Chairman: Right. When can we expect the Kathleen Tattersall: Thank you all very much Sutherland report? indeed.
Supplementary memorandum submitted by Ofqual On the further points required:
Timetable for the marker training and the impact that the face-to-face rather than online had on the timetable
Marker training has had a fairly fixed cascade for several years, culminating in the majority of markers being trained during the first two weekends in May. For 2008, the earliest formal schedule of which Ofqual had sight of is from mid-November 2007. It clearly indicates that the final marker training meetings were scheduled for 10 and 17 May 2008. Whether switching to a new online training model, or remaining with the established face-to-face model ETS would have been aware of the need to meet the dates outlined in this schedule, and indeed in the Proof of Concept Pilot—Design document (May 2007), section ix details the ETS contingency “in the event the pilot proves it does not validate the acceptance criteria” would be a series of face-to-face training sessions.
How long did it take to conduct the face-to face training of all markers?
As in previous years the majority of face-to-face training sessions take place on a single day,for 2008 either the 10th May (for key stage 3 markers) or the 17th May (for key stage 2 markers). The vast majority of markers (around 5,000 per key stage) are trained on these days at a variety of training locations around the country (around 50 diVerent venues in total). This is the culmination of a five month cascade where by the Marking Programme Leader develops the training materials with her deputies (meeting 1, 2 and 3), who then train the senior markers (meeting 4 and 5), who then train team leaders (meeting 6 and 7) who then train markers (meeting 8). Supervising markers (seniors and team leaders) get two days of training. One day of training on the mark scheme, and the second day of training for their role as supervisors. All training days usually run from 9.30 until 16.30/17.00.
Did the NAA reported to us that ETS had said that the change from online to face-to-face training would lead to a long delay?
At the meeting with NAA on 19 Feb 2008, the regulator was given reassurances to the opposite at the meeting with NAA to discuss the results of the proof of concept pilot, and as early as January, NAA had assured us that venues were already being sourced as a contingency in line with the steps detailed in section xi of the Proof of Concept—Design document. Processed: 17-07-2009 19:04:49 Page Layout: COENEW [E] PPSysB Job: 406183 Unit: PAG2
Ev 32 Children, Schools and Families Committee: Evidence
At what point were markers trained in the handling of what we considered to be over-ambitious software? Was that done in early May as well?
Markers were supposed to have completed an online training module before they arrived for their face- to-face training in May. As this was to be conducted from home, this wasn’t something we could observe, though we have heard reports that markers had considerable problems with this system. There also seemed to be no eVective method of confirming exactly who had and who hadn’t completed this module before markers attended their face-to-face training. The training module did not go live in time for supervising markers to have experienced the system, meaning they were unable to answer queries on the system at the training meetings in May. Any queries on the system had to be answered by ETS representatives, who weren’t always present at the training meetings. This left the majority of our observers unconvinced that markers had been given enough training on the new ETS software systems.
Also for clarification
Markers entered “question level data” rather than “item level data” onto the OMC system.
Q206—number of interventions The Regulation & Standards Division of QCA had no formal involvement in the process of awarding the test operations contract for 2008–12, although Ofqual’s Acting Chief Executive, Isabel Nisbet, was a member of the QCA Executive at the time the contract was awarded. From the summer of 2007, the National Curriculum Assessments monitoring team (“NCA monitoring”) was invited to comment on planned changes which set out the ways in which markers were to be supervised and quality assured. NCA monitoring raised some concerns at that early stage with NAA. On 2 November 2007, NCA monitoring provided feedback to NAA on marker training for the marking pilot which said that communications in the broadest sense gave rise for concern. For example, how contacts with markers were managed; ease of access for venues. ETS carried out a “proof of concept pilot”, examining the eVect on the quality of marking of four new approaches to on-line standardisation and quality assurance. In February 2008, NCA monitoring, together with the Director of Regulation & Standards, Isabel Nisbet, attended two presentations on the outcomes of the pilot. On 21 February, Isabel Nisbet wrote to the Director of NAA (David Gee) expressing the regulator’s support for the four approaches which had been trialled, but conveying two “significant concerns”. The first was a systems—based concern that the volume of marker use at key points in the marking process might compromise the ability of ETS to operate eVectively. This was to be tested through load testing and NCA Monitoring asked to be kept informed of the outcomes. The second concern was that the delays experienced in delivering the pilot and its report could indicate that the resources being assigned by ETS [were] insuYcient to meet the required deadlines and standards of quality. David Gee replied on 6 March, agreeing to provide updates on load testing and sharing the regulator’s concerns about the management of the pilot, but commenting that it was managed by a diVerent team in ETS than would be involved in the 2008 operational cycle. On 21 April, in response to a further request from the NCA monitoring team for an update on load testing, David Gee wrote that significant work was done by ETS over the previous six weeks, and concluded that the data would provide assurances that preparation is well in hand. In April 2008 Ofqual was set up within QCA. On 28 April Kathleen Tattersall, newly-appointed Chair of Ofqual, had an introductory meeting with Ed Balls, Secretary of State for Children, Schools and Families. In a general overview of the issues facing Ofqual over the summer period, Kathleen Tattersall mentioned, among other things, the heightened risk of there being a new contractor responsible for the delivery of the National Curriculum tests. The NCA monitoring team observed a sample of the ETS marker training events and, following a training session on 10 May 2008, reported concerns to NAA about attendance of markers and the rigour of their selection. During May, conversations with markers alerted the team to delays in the delivery of scripts for marking and to problems in communications with ETS. On 3 June 2008, Isabel Nisbet wrote to David Gee to seek reassurance from NAA that marking of National Curriculum tests will be completed and the scripts returned to schools by the deadline of Tuesday 8 July 2008. Specific reassurances were sought about aspects of marking, communications and administration on marker training. The letter also informed NAA that Ofqual would hold a formal accountability meeting, chaired by Kathleen Tattersall, with NAA on 3 July.The decision that Ofqual’s chair should chair the meeting with NAA was made on the basis of risk. Processed: 17-07-2009 19:04:49 Page Layout: COENEW [O] PPSysB Job: 406183 Unit: PAG2
Children, Schools and Families Committee: Evidence Ev 33
David Gee replied on 11 June addressing each of the specific questions raised in Ofqual’s letter of 3 June His letter stated that the NAA was heavily focused on ensuring the delivery of results to schools by 8 July. Despite a number of challenges created by inadequacies in the ETS delivery process system David Gee had extensive reassurances that this would be achieved. However, in order to reduce any risks further he dedicated significant additional NAA resource to support ETS in meeting its contractual obligations. On 11 June, the NCA monitoring team took up a long-standing invitation to visit the distribution centre in Dewsbury. Following that meeting the NCA team raised concerns with NAA about the apparent absence of control processes in the management of scripts. In mid-June, DCSF asked Ofqual for a short briefing note to show to Jim Knight, Minster of State for Schools and Learners, before a Ministerial meeting with NAA about test delivery. Ofqual’s note stated that it was our clear impression at that time that there is a high risk that all schools will not receive their results by 8 July as a result of marking not being completed and problems with the ETS marking and distribution systems. The note went on to report problems with preparations for the following week’s level setting meetings because of insuYcient marks on the ETS data system. On 2 July 2008, David Gee wrote to Isabel Nisbet notifying Ofqual that NAA would be in breach of one requirement of the Code of Practice because some schools would not receive all their data by the published date. The accountability meeting took place on 3 July. Following the meeting, Kathleen Tattersall decided that Ofqual should set up an independent inquiry into the regulation and delivery of the 2008 tests, and on 4 July she wrote to the Secretary of State informing him of that decision. Her letter also gave some reassurance about the quality of the marking of the tests: “While results will be delayed and I cannot predict the volume of reviews that schools will request this year, from the processes we have observed, the quality of marking is at least as good as previous years and justifies issuing the results.” On 4 July, NCA monitoring requested from NAA details of the marking panels that ETS needed to set up to complete the marking process and also details of the quality assurance arrangements for these panels. Visits were made to marking panels on 16 July. Starting on 25 July 2008, Ofqual has held weekly regulatory meetings with NAA (observed by DCSF) to monitor progress in identifying and marking unmarked scripts and in the review processes. Those meetings continue. A problem, particularly in the earlier meetings, was the diYculty of obtaining a precise estimate of the total number of unmarked scripts and assurance that they had all been identified and were being dealt with. On 28 July, NAA announced that it would take over from ETS the management of the review process in 2008. Ofqual publicly welcomed that announcement. On 30 July, Isabel Nisbet wrote to David Gee expressing concerns about the adequacy of ETS’s resources and quality assurance for marking the remaining unmarked scripts and asking NAA about their intentions regarding deadlines and charges for reviews. David Gee replied on all these points on 8 August. Concerns about data on the system were particularly relevant to the publication of Key Stages 2 and 3 data. These must be produced to the standards set out in the OYce of National Statistics Code of Practice and free from any political interference. DCSF’s Head of Profession for Statistics (Malcolm Britton) twice wrote to Ofqual seeking advice for him to consider when deciding whether to publish results for Key Stages 2 and 3 at national and local authority level. Ofqual’s replies focused largely on what evidence was available on the quality of marking and the confidence that could be placed in the outcomes. The advice sent on 4 August (about the Key Stage 3 results) included: “Ofqual recognises that the confidence of teachers, parents, pupils and the wider public has been damaged by the problems in delivering this year’s national curriculum tests…[T]he credibility of results published at this time will no doubt be challenged. However, in providing advice to inform your decision on publication, Ofqual needs to consider whether there is hard evidence to call into question the quality of the figures which you are considering publishing this year, compared with previous years, to the extent that would justify a decision not to publish or require publication with significant reservations. … [F]rom the processes that we have observed to date, there is no evidence of widespread problems with the quality of the marks at Key Stage 3 that would justify withholding publication of the provisional results at national level.” Similar advice had been sent out on 28 July in relation to the Key Stage 2 results. September 2008 Processed: 17-07-2009 19:14:49 Page Layout: COENEW [SE] PPSysB Job: 406183 Unit: PAG3
Ev 34 Children, Schools and Families Committee: Evidence
Monday 26 January 2009
Members present: Mr Barry Sheerman, in the Chair
Annette Brooke Mr John Heppell Mr David Chaytor Mr Graham Stuart
Witness: Lord Sutherland of Houndwood KT, gave evidence.
Chairman: Lord Sutherland, it is a pleasure to have recognise that the Committee’s work has been you here in front of our Committee. You know the integral throughout the sad period since June or July, rules, as you are a Member of the House of Lords, when things started to go wrong. The sessions you and in that wonderful group of people in this held before the school holidays were very important. country who do not have to appear before the You picked out a number of major issues, and I was Committee. able to build on that in the evidence I asked for and Lord Sutherland: Nobody told me that. in some of the cross-examinations that I carried out. Chairman: I think one of your colleagues at one I share what is evident: you, like me, were hugely stage had a rather interesting relationship with disappointed in the treatment of teachers, pupils, Gwyneth Dunwoody, when he was advising on blue- markers, parents and school governing bodies, and skies approaches in No.10, and refused to appear what they had to put up with. I had direct evidence of before her Committee. It is a pleasure for us that you this in a number of schools and from talking to head have agreed to appear before our Committee, and teachers of my own acquaintance. They were badly for someone who has known you and your work in let down this summer, and there is no excuse for it in education for quite some time—I think we first met the end. I hope that my report has been instructive in when you were Vice-Chancellor of London providing a description and an analysis of what went University— wrong, finding the problems that were encountered Lord Sutherland: That is right. That was a while ago. in the delivery. I hope too that there are messages for the future. This is not simply a matter of looking backwards. As you will have read, I identified a Q237 Chairman: It is a while ago. You know why we number of contributory factors—there were very are here. We have been looking at the mess that many of them, in fact, which was part of the occurred last summer. Indeed, I have to recount to diYculty. Had there been just one issue, one would you a story that I picked up in the first week of term have identified that fairly quickly and dealt with it in after the Christmas holidays at Almondbury High the report. But, in the end, one of the key questions School in my constituency, where I went, as one does that remains with me, and one of the key causes of in one’s constituency, merely to see how the school the diYculty, was the lack of end-to-end testing of was getting on. Out of the blue, they said, “We’re still the system. That was a major failure on the part of waiting for our last level 3 SATs tests to come back.” the Educational Testing Service, and a primary I wanted to make sure that that was absolutely cause, because if you do not test the system end to accurate, so I rang them this morning, and they said, end, you do not anticipate problems that you could “Mr Sheerman, you won’t believe this: the last 12 have anticipated—you would know that a diYculty scripts arrived this morning.” Now, there is quite a there will have repercussions further down the line. long gap between today and the date that we all ETS failed pupils, schools, markers and teachers, know they should have arrived in July. So, there we and it has primary, but not sole, responsibility for are: I just mention that as a little bit of background. what went wrong. There were a number of Lord Sutherland, we know what this is about. It is to significant management failings within the find out what went wrong and prevent it happening Qualifications and Curriculum Authority, again. Would you like to say what you found in particularly in its management of risk, and no doubt your inquiry? we will talk about that in due course. As I suggested, Lord Sutherland: If I could make a short opening it is not simply a backward-looking report: I hope statement, I would then be interested in taking that it looks forward. I made a number of questions and perhaps even having a discussion. recommendations, 19 in all, if I counted correctly, There are important things to work out. and I wanted to take a constructive and forward- Chairman: I do not think we can have discussions in looking approach. There are some specific this Committee. Hansard does not take kindly to it. recommendations that I regard as especially Lord Sutherland: Thank you, Chairman, for inviting important to improve the system for the future. me, and also for focusing on the report and on the First, there must be full end-to-end testing of the issues so promptly in July, as you did. I think you system, not just checking that the individual bits were the first above the line, making very important work within themselves, but that when you link questions plain and putting them into the public them up, they run as a single operation that does not arena. I found that a good starting point for me. I have the hitches we discovered last year. Secondly, I Processed: 17-07-2009 19:14:49 Page Layout: COENEW [O] PPSysB Job: 406183 Unit: PAG3
Children, Schools and Families Committee: Evidence Ev 35
26 January 2009 Lord Sutherland of Houndwood KT believe that forms of modernising the test delivery to receive support, as the lines that were meant to be process should be put in place, and I dare say that we available to them were not always functioning will look at that in your questions. We will have an properly; or if they are rushed in their marking important opportunity to legislate for the regulation because of poor administration and scripts arriving of tests on the same statutory footing as the late in the day or at the wrong time, that can aVect regulation of exams in the forthcoming children, quality severely. I want to stress for the future that skills and learning Bill. I hope that we make the most paying attention to the role, support and, indeed, of that opportunity. Taken as a whole, those three wisdom that markers have about the system is key recommendations, and the others that stem from important. The recommendations that I have made them, give the Department for Children, Schools focus on delivery issues—that was the job that I had and Families, QCA and Ofqual a clear direction for to do—but they are also pertinent to ensuring the the future. I know that the Committee has taken a quality of test results in future. One element of great interest in establishing the quality of the tests that—and I have referred to it already—is ensuring and the results. I recognise that that is fundamental, that Ofqual has adequate powers to carry out its and it is important that you continue to do so: you task, and that is a matter for legislation. That is are a continuing body—my job is oYcially done. I slowly being floated oVshore. The process began last believe, however, that the independent regulator, April, and it will be complete when the Bill that we Ofqual, performs a vital role in that respect. I believe all anticipate will be introduced goes through that the relationship between Ofqual and you is an Parliament. Equally, I stress the importance of important marker for the future, and it will be a having the voice of the marking community critical way for it to report to Parliament. It has the available to those administering and running the role of overseeing the tests to ensure that “pupils get system. If we put those in place, there will be the marks their work deserves”. Many people have significant practical steps forward. That is as much asked whether they can trust the 2008 results—it is a as I want to say at the moment, and I am happy to question that I have asked. Fundamentally, Ofqual pick up questions and comments. has to determine that, and I believe that a report is forthcoming. It is important that it is an evidence- Q238 Chairman: Lord Sutherland, thank you for based report—I anticipate that it will be. An your introductory remarks. Can I open the important part of the evidence, for example, will questioning by saying that we hoped this would be a look at the outcome of the reviews process, which is session with you for an hour or so—perhaps an hour not yet complete—you alluded, Chairman, to some and a half—and a session with Ken Boston? At first, of the implications of that in your opening remarks. we were given an indication that he, too, would It must look, too, at the content of test questions and appear before the Committee today, but he found he mark schemes, marker training—that is an was unable to do so. As you know, he tendered his important part of quality in the system—and, as I resignation, and it was not accepted, but he is said, the outcome of the reviews process, which is not currently on what I suppose most of us would regard yet complete. Ofqual has expertise in all those areas these days as gardening leave, and he is not available and, more importantly, it has the responsibility to to the Committee. Does that surprise you? consider that. Having said that the question of Lord Sutherland: I do not know if “surprise” is the quality is one that has to be finally decided in word, but it disappoints me and I hope he will be relation to last year, there are two elements to that, available. I am sure there is quite a lot you could which I came across in preparing the report and learn in a process such as this that would help the which I would like to share with you and comment development of the system in future. on later. First, the national curriculum test markers are a small constituency of dedicated professionals—they are often teachers or retired Q239 Chairman: It does slightly open up a problem. teachers. I want to pay tribute to them, because When you were asked to look into this matter, it was despite the diYculties that they faced—and they the Secretary of State who did so. Do you think it were severe and significant at times—they would have been a better arrangement, even if he persevered to mark 9.8 million scripts to the best of had initiated it, if your report came back to this their ability in increasingly problematic contexts. We Committee? have relied on these same people in previous years, Lord Sutherland: The first approach I had was from and I have no doubt that we will continue to do this Ofqual. Immediately thereafter, I was also year and perhaps into the future. They need to be approached by the Department. I pointed out properly supported—and they were not this year— something that I am sure that it already knew—that and that is fundamental to the quality of what goes nobody wanted two reports and it would be sensible on. I also believe, more than that, that they need to to have one. However, I had a reporting line to Ofqual which, from within the old QCA, properly be consulted about the delivery of the test system realised that there was something which would and how it can be modernised and the quality benefit from external scrutiny. That is why it got in improved. They have the experience—many of them touch with me. over a number of years—that would be highly relevant to ensuring that the best quality is identified and continues into the future. If markers find it Q240 Chairman: Do you think any of the characters diYcult to access training—some of them did, and who played a significant part in last summer’s drama that is itemised in the report; or if they find it diYcult have been badly treated? Processed: 17-07-2009 19:14:49 Page Layout: COENEW [E] PPSysB Job: 406183 Unit: PAG3
Ev 36 Children, Schools and Families Committee: Evidence
26 January 2009 Lord Sutherland of Houndwood KT
Lord Sutherland: Not to my knowledge. to the QCA chief executive, and he wrote to ETS to make it plain that there was no such embargo. That Q241 Chairman: Well, some of us have always is point one. Point two: we extended the period respected Ken Boston as a highly qualified and wise available to it to submit evidence. Point three: it person at the head of QCA. As I read your report, chose not to do so. Point four: the evidence was and reflected on the evidence that had already been available to us, through the QCA documentation— given to the Committee, I did wonder sometimes we had all the exchanges of correspondence and why he had been suspended from his role. Here is a contracts and so on, so I believe I had adequate talented man, presumably still drawing his salary. evidence. However, as you said in your report’s introductory remarks, the breakdown in the summer was a multi- Q245 Chairman: Is there not a danger, Lord faceted, multi-causal problem. I will ask you again: Sutherland? I do not believe in conspiracy theories— do you think it is absolutely necessary that Ken and I know that you chair a Committee similar to Boston should carry the can on this? ours in the House of Lords—but if you were sitting Lord Sutherland: In a sense, we have all been pre- in my place with my colleagues, you would see that, empted on that, because he resigned when he read first, we cannot get Ken Boston back to talk to the report. The QCA board, for its own reasons, Committee. Secondly, we cannot get the company which I am not directly aware of, decided it should after this time, after due reflection and after things be suspension rather than resignation. He made the have settled back. Those are two important sources judgment—it is much to his credit and honour that, of evidence for our inquiry, which is separate from if he felt responsible as a public servant, he should yours, with a slightly diVerent focus, and their stand aside. absence sits rather oddly when you are trying to get to the truth of the matter. Q242 Chairman: He is rather an old-fashioned, Lord Sutherland: I did not know the position over honourable gentleman, is he not? Ken Boston before you told me just now. Putting Lord Sutherland: Yes, I have high regard for Ken these two things together does not occur to me as an Boston and what he has done. obvious thing to do, but perhaps you have background information that I do not have. As for Q243 Chairman: Your report, as I read it, surprised the ETS thing, in the end, the company cut its losses me, in the sense that the number of failures are and left, repaying, I have to say, significant sums of surprising. It is like a road accident: it seems like a money. It accepted that it was not delivering, to the very simple thing, but when you analyse it, there are tune of nearly £25 million. For an American probably 10 diVerent causes of the most simple company, that is a big acceptance. It accepted that. accident. However, this one did look like something I suppose, in its position, it thought, “We are a long of an accident waiting to happen. The feeling we got way away.” Head oYce said, “Oh, just pull up the when we took evidence from other parties, was that drawbridge; we’ve had enough.” However, I did people totally over-estimated the capacity of the new have a later correspondence when my conclusions company to fulfil the function. Did you get the same were evident, and I wanted to be sure that the senior feeling as us that people here in the UK, who were oYcial within ETS was aware how critical some of part of the process of drawing up and awarding the my conclusions were. I wrote to him, giving him that contract, thought that this rather large and globally information—not sending advance copies or active company was going to come with all its anything, but just saying, “This is what we are going resources and knowledge and take on this contract? to say about ETS”. I had a very interesting and However, it started a new company as an oVshoot discursive letter in response. There were one or two with no experience and hardly any personnel. Was helpful details—he could put numbers, facts and so that not something of a surprise to you? on, in place—but there was no response of the kind Lord Sutherland: It is easy to be dazzled by expertise that would lead me to go back to square one and that is well presented. No doubt, ETS made a very start again. good presentation and went through the long-due process—an iterative process—of putting in a tender Q246 Chairman: Have we had a sight of that letter? for this work. It is evident, however, that it was not Lord Sutherland: I have no idea. up to it. I say that quite explicitly: it did not deliver on the contract that it signed, and therefore it did not Q247 Chairman: Could we have sight of it? keep the promises that it had made. Lord Sutherland: All the materials are deposited with DCSF. I am sure you know the routes to obtain Q244 Chairman: Why do you think that it decided information of that kind. I have not kept any of that it could not give evidence to you? these documents. They are the last thing I want lying Lord Sutherland: You would have to ask it, but let us around my flat, I have to say. hang on to that, as it is an important question. I have no doubt that there was adequate evidence available Q248 Chairman: So it is available. Lord Sutherland, to me to make the judgments that I believe are you can see me smiling, but I do want to ask you a definitive about the role of ETS in this report. It more diYcult question to conclude my opening believed at one point, in view of something that it questions. In a sense, when a government set up an said to your Committee—you were aware of this— inquiry, sometimes it is set up to take the heat out of that there was an embargo on it in some way. I spoke the situation and to take the focus oV the Secretary Processed: 17-07-2009 19:14:49 Page Layout: COENEW [O] PPSysB Job: 406183 Unit: PAG3
Children, Schools and Families Committee: Evidence Ev 37
26 January 2009 Lord Sutherland of Houndwood KT of State and the Government, who might be seen as sounded like just one more player rather than the top responsible for the debacle of last summer. As you of the pyramid and the driving force behind the very said, a lot of people were hurt and upset by that structures that you were describing. Do you think process. Do you feel at all that you were used by the that is a fair criticism of your report? Government to help them to evade accountability Lord Sutherland: I hope that I am clear on that. The for their role in this mess? DCSF set the policy, and it provides the resources— Lord Sutherland: I will say straight out, if they that is its responsibility. It has to ensure that delivery thought they were going to do that with me, they got is possible, which means it must have an appropriate the wrong man. The report is firm—as you say, it mechanism. It believed it had, and on the basis of the contains many criticisms, including comments on two or three previous years it did have, because procedures that relate to links in the Qualifications delivery was made. Below that, we come to QCA, and Curriculum Authority and the Department for which has a set of responsibilities, and so on, down Children, Schools and Families. That is firm and the system it goes. In my understanding, the DCSF clear. I have to say that there was no attempt to is responsible for policy, resource and setting up the interfere with the processes, and I made it plain at the overall mechanism. The kind of mechanism that is outset, that if there were any such attempts I would set up relates to how well it might work. go to the press immediately, and I think they believed me. Q252 Mr Stuart: Defining the Department’s responsibilities in that way is very convenient for Q249 Chairman: I was asking you a slightly diVerent Ministers who wish to push responsibility away; question, Lord Sutherland, from the one that you they could say they did it in the best odds. One could answered. If you look at what was going on in the take issue with the point you just made about the Department and in QCA at the time that we are previous three years. There was a failure in 2005, discussing, this was not a ship that was changing there were diYculties in 2006–07. It made it a rather course: this was a ship where the engine was being peculiar situation in that it might have looked like it changed during the voyage. Perhaps a navigational was working, but it was only just. It made it rather a system was being changed, to use the analogy or peculiar decision by the QCA to take such a push it a little too far. At the time, if you were Ken monumental risk for 2008. Boston sitting there as captain of the QCA ship, with Lord Sutherland: You mean to do it at all? all these fundamental changes going through—there was no legislation, but there were shadow Q253 Mr Stuart: Yes, and particularly to do it in the authorities, totally changing the nature of the QCA, way they did. Did you manage to find out to what changing its role, splitting it up, shadow this, extent that was being driven by Ministers, being shadow that—was that not rather diYcult for the driven by the Department? On the face of it, it is hard leadership and the captain of the ship in QCA, to see why the QCA would want to enter into such a having all that going on as well as a major new high-risk strategy if they were not being pushed by contract that had been arrived at only recently? someone else who was going to be able to Lord Sutherland: It is a big job—no doubt about commission an independent report and push the that. On the other hand, in a position like that, you responsibility away afterwards. need to be a big leader to carry it through. You get a Lord Sutherland: You have to ask QCA why they remit letter each year, and if at some point you say, accepted the remit they were given and they did. You “I do not have the resource or the capacity to do all have to ask why and what diVerences there were that of that,” that is the time to say it. However, if year. I do— someone accepts the remit and goes with it, the expectation is that you believe you can deliver this Q254 Mr Stuart: I wondered whether you did, year. because you did the inquiry. Chairman: Okay, let us open up the questions to Lord Sutherland: I do genuinely believe that they colleagues, or they will be very unhappy. Stuart. thought that in ETS they had a winner. The external Mr Stuart: Or Graham, even. signs were justification for the view they took. I am Chairman: I am terribly sorry—Graham Stuart—it critical of some aspects of that but they believed that is because I was talking to a Scot. they had a winner and that they could build on the previous experience of Edexcel. There was adequate Q250 Mr Stuart: Two Scottish Christian names in a handover and coaching arrangements and so on in row. Lord Sutherland, who do you believe is place. Are you questioning the very idea of there ultimately responsible for delivering proper exam being an arms-length arrangement with anybody? results across the country? Most of Government is doing this now and I will put Lord Sutherland: In the end, it is the Department and in brackets—this is not in the Report; it is a point I the Secretary of State. Below them, they have an am making to you—I believe there is so much of that arrangement where they issue a remit and resource now that it is time to pause and say,“Have we got the for QCA, as it currently is, to deliver that. best possible arrangement that allows such an allegedly arms-length delivery?”. It sounds grand, Q251 Mr Stuart: Do you think that your report is but the arm is a real thing and messages move up and explicit enough about that? I try to imagine a lay down. It is happening across Government. There are person reading it and being clear about it. In it, as choices such as whether you use the private sector to you laid out the various organisations, the DCSF deliver Government policy or not. I would be Processed: 17-07-2009 19:14:49 Page Layout: COENEW [E] PPSysB Job: 406183 Unit: PAG3
Ev 38 Children, Schools and Families Committee: Evidence
26 January 2009 Lord Sutherland of Houndwood KT surprised if it was not necessary in a whole range of compensation and they were not prepared to play areas, but that is a good reason to review how well it any more. Is it not fair to say, therefore, that the is working. That has been done in this particular version of events you gave just now is perhaps an case. One of the things that they had started doing over-simplification or do you think there is no merit and which it is important we hang on to is the in ETS’s arguments? process of setting up an independent regulator, Lord Sutherland: I can tell you on the point on which which was desperately needed—this is one of the I have direct evidence that I did not refuse an complications. If that regulator is given the powers extension. They were oVered an extension and that and the resources to do the job, many of the is on the record. The extension was quite adequate in diYculties could be avoided. terms of the seriousness of the matter. This was a £156 million contract. We had diYculty getting in Q255 Mr Stuart: You have put your finger on the touch with them—they had left their oYces in this exact issue, which is that there is this tendency to use country and were in the States. Getting the the private sector. The question is whether it is individual who would take responsibility was not appropriate in a democracy and for Ministers always easy, but they were given an extension—I can accountable to Parliament. In a democracy you are tell you that for sure. supposed to be able to identify those you can vote for who in turn take responsibility for what goes well or goes badly. When they find it convenient to have Q257 Mr Chaytor: May I ask about the procurement arms-length organisations doing the critical bits, process, which I was fascinated to discover from Ministers can wash their hands of it. I do not know your report was called Project Tornado? Somebody whether you intended that Ed Balls should be able to in the QCA had a sense of humour, I must say. You use your report to wash his hands of responsibility conclude, Lord Sutherland, that the procurement for last year’s fiasco. It took a long time and a lot of procedure was sound, but then you go on to list a parliamentary eVort to get him to say the word whole series of criticisms of the procedure, many of “sorry”. which seemed very substantial to most members of Lord Sutherland: I understand that he has said sorry. the Committee. So, are you absolutely confident, in I am not in constant communication with him. This the light of all the criticisms you make, that the is not a close relationship, you should be aware of procurement procedure was sound? that. Lord Sutherland: That is a point on which I could Mr Stuart: Fine. have been clearer. The procurement procedure was Lord Sutherland: It is about how you run your sound in this very clear respect: that they followed to country. You have opened up a big issue and, I warn the letter the procedures laid down by the OYce of you, I am a philosopher and we could go on a long Government Commerce. They went through that time on this. If the Government is going to run the process, and I have seen how they took evidence and country using private agencies—this has been the how they made their judgments. They made the track for 20 years—it is not good enough to say that judgments first on the basis of capability, without in this particular case it is an excuse. It is a much looking at the question of how much it was going to bigger issue. You are right, however, that you have cost. All that side was correct. What I then go on to to watch very carefully. I had the Secretary of State say, and I think this is a message for how these things in to give formal evidence. He did give formal are done, is that there is additional due diligence evidence and he was asked the same hard questions which I believe should have been carried out—I as everyone else. I had one of the other Ministers in think others have made that point as well—but it is to do the same, and the permanent secretary, so it not required. I want to make it plain again that the was a formal process of review, including evidence job that PricewaterhouseCoopers did in due from those individuals. I hear the words you use: I diligence was exactly what they were asked to do hope nobody is washing their hands of anything, within the Government’s procedures and guidelines, because the critical thing is to get it right next year which was to look at financial stability and whether and the years after. it was a viable company, and so on. But there were clearly reputational issues that were not discovered Q256 Mr Stuart: One further question, taking you and should have been. That I regarded as outside the back to your evidence a little while ago to the formal procurement process. I did not want them Chairman on the subject of ETS not giving evidence. coming back saying, “Of course you said so on and The way you told that slightly diVered from what we so on, but we followed all the rules.” Well, they did. have heard, or at least is a simpler version. ETS said I did put this to the board of QCA: if you were they did write. It took some time in the light of the oVering a company a contract of this size, would you closing date of your inquiry for QCA to respond. In not have done a bit more checking on the reputation fact they said that, in the meantime, because they of the body? They did not, and that was a failure. were asking for permission to give evidence to your inquiry, they wrote to ask you to give them an extension of the period. You refused to extend the Q258 Mr Chaytor: In light of what you just said, period, QCA took ages to come back to them and by would it not be more accurate to say that the the time collectively everyone came back to them to adherence to the procedure was sound or complete, give evidence they said they had given up—wrapped but the procedure itself had significant deficiencies, it up. They were paying millions of pounds in which you just referred to? Processed: 17-07-2009 19:14:49 Page Layout: COENEW [O] PPSysB Job: 406183 Unit: PAG3
Children, Schools and Families Committee: Evidence Ev 39
26 January 2009 Lord Sutherland of Houndwood KT
Lord Sutherland: That is a distinction I recognise. Q262 Mr Chaytor: Is it absolutely clear that not a The procedure was there; they followed it right single person in the Department for Children, through. It is a procedure that is laid down for every Schools and Families, QCA, National Assessment government contract. Agency or Ofqual went on to their computer and googled ETS? Lord Sutherland: I cannot vouch for what several Q259 Mr Chaytor: But in terms of your thousand people did or did not do. recommendations, based on the findings, you are actually recommending that certain changes are made to the procedure. Q263 Mr Chaytor: There is no evidence that anybody did that? Lord Sutherland: Yes, other things should have been Lord Sutherland: No, and that is my criticism: not done. I accept that. that nobody did it but that there were people responsible who should have done it. Q260 Chairman: So who should have googled ETS? Should it have been PricewaterhouseCoopers or Q264 Mr Chaytor: In your findings and should it have been QCA? Because, as someone said recommendations, some of your criticisms are to us, you only have to google to get all that press specific to QCA or the DCSF observers and so on, comment that was already there. and others are expressed passively, with more Lord Sutherland: PricewaterhouseCoopers did what reluctance to attribute responsibility: “Such-and- they were asked to do. They performed according to such a process should have been carried out”, for contract, which everyone had. So I would not lay the example. Is that conscious or just how the report is blame there. So I think it should have been QCA, written? Sometimes you are very clear as to who was and I did put it to their board. at fault and at other times less so, as if you do not wish to point the finger of blame. Lord Sutherland: The main driver of the report was Q261 Mr Chaytor: On the role of to produce an analysis of what went wrong. PricewaterhouseCoopers, your report says that they Sometimes it was very clear from that analysis who were asked to do the following tests. They produced should have done what, and sometimes it was a the Dun & Bradstreet rating for each company, they question of, “What was the arrangement operating examined the last three years’ published financial between NAA and QCA?” That is an uncertain area, statements, and then they did a review of press and interestingly, given that you suggest that information over the past 12 months. That is the key perhaps my report has not been tough enough, NAA flaw, because, had they carried out a review of press no longer exists. That is recognition that the information over the last three years, the same structure there was wrong or uncertain. That is why period for which they published the financial sometimes you could not say that it was person X in statements, that would have highlighted a number of NAA or person X within the broader context of press reports in serious publications in the United QCA. Do not forget that Ofqual was not a separate States and, I think, in the UK, that would have given body most of the time, and was only a quasi-separate a diVerent picture. Is it conceivable that a large body from 1 April 2008. I would have to see specific multinational management consultancy such as examples, but that is probably why occasionally the PWC would not have known of the activities of ETS criticism is passive. in the United States, and its track record prior to the 12 months for which it did the review of press Q265 Mr Chaytor: Regarding the problems that information? I would find that remarkable. were identified in the early stage of the procurement Lord Sutherland: I have two things to say on that. process, could you remind the Committee what your PWC is a professional body—it had a job to do and recommendations for change are, and whose it did it. There are two constraints on how much responsibility it should be to flag up problems? There testing you carry out and how far back you look. were capacity problems: ETS said that it could One constraint is the 12-month period and that was manage with 60 staV, which was a vast given to them, and the other is that you look at underestimate. contracts worth more than £20 million a year, and Lord Sutherland: You have to go a bit down the line that was also given to them.1 Within those that says who is awarding the contract. The contract constraints, PWC did what it was asked to do and so and its award is recommended by QCA. QCA is on I would say that it delivered on its contract. The the front line in deciding who gets the contract. The question is why, if we were dealing with such an question then becomes, what tests might it or should important company,was more not known about it in it have carried out that it did not? And is there the system, including in QCA and by those who anything else that should have been done further up would be working with it. Indeed, I would have or down the system? I have made some specific thought—this is the point of your question—that recommendations on that. One of those they would have had some international contacts recommendations is that I believe that Ofqual whom they could have asked, but they did not, and should in future be consulted. It is not its job to that is a severe failure. When you go back to where award the contract, but it should be consulted on the blame lies it is not simply ETS, it is the checking. nature and the details of it. The reason for that is that Ofqual needs to have certain kinds of information if 1 See Ev 47 it is to carry out its quality assurance processes, and Processed: 17-07-2009 19:14:49 Page Layout: COENEW [E] PPSysB Job: 406183 Unit: PAG3
Ev 40 Children, Schools and Families Committee: Evidence
26 January 2009 Lord Sutherland of Houndwood KT it was evident that it was not receiving that contracts. I may have missed this in your report, but information. Unless that is built into the contract what is the general advice that you give? In this case from the start, there will be diYculties. That is an it seems that there is a lot of reliance on the company. example of why Ofqual should be consulted when If you want a diversity of contracts, you cannot the details are being prepared. I do not believe that aVord for one of the small number of contractors the presence of DCSF observers on various that oVer the service to not be in business. committees is as clearly based as it should be. Lord Sutherland: There are real issues that this case Perhaps there is an assumption that if a DCSF has helped to illuminate. You and I will remember observer is there and does not say stop, everything the Student Loans Company, which went must be all right. We will be told that that is not the disastrously wrong a number of years ago. Its role of DCSF observers. Experience probably varies headquarters was in Glasgow. There are any number from one committee to another because there are a of examples, such as the Child Support Agency. This lot of them around. However, if somebody from the issue comes back to Mr Stuart’s question over the DCSF or the OYce of Government Commerce way in which contracts are awarded and whether this procurement agency is present, their role must be is a good principle to follow. I have no doubt that made clear. you need the reassurance that there is a good market with customers who will compete with each other. Q266 Mr Chaytor: Are you confident that the There must be choice in case a company proves not structural changes to the QCA and the to be adequate. To reaYrm the point that you make establishment of Ofqual as a separate body will about, I have looked in detail at the process, and all resolve such procedural diYculties in future the evidence I have—and it is strong evidence— contracts? suggests that the procurement process marked the Lord Sutherland: It is the right direction, but it is a companies on abilities and skills first, before opening tight run this year. In the longer term, I hope that the the envelope containing the bid cost. ETS came out structural changes will have an impact on future No. 1 on all the indicators and was also the contracts. I hope that how scrutiny should be cheapest—I am convinced that that was not a factor, allocated is much clearer in the future. I agree with your judgement on that.
Q267 Mr Chaytor: Finally, in terms of reputation Q269 Mr Heppell: I am fascinated by the fact that and the track record of checks, you do not the diVerence seems to be so great between what the recommend that procedure should change to require contractor and the QCA believed to be happening. a longer period for the press review: it remains as a ETS were saying that they needed 60 staV, yet at one review over the last 12 months. Would it not be a stage were employing about 400 staV. A lot of the sensible change to the procurement procedure to pilot tests went wrong, but there did not seem to be make it a press review over the previous three years? any communication that that had happened. As you Lord Sutherland: It certainly would be a sensible said, end-to-end testing of the system seemed to procedure. It is the experience of my working life disappear altogether. I think that ETS told us that that when you lay something out very clearly and the original contract was for online training, but the put it under the nose of those responsible, you are QCA say that that was not quite what it had said. able to scrutinise whether they have taken good That makes me wonder—it does not reflect well on account of it. They claim that they are going to. the QCA whatever way we look at it. They drew the contract very badly in the first place—with so many Q268 Chairman: But Lord Sutherland, on the ambiguities and holes in it that it was not going to procurement question, there was gossip going stand up—or was it the case that they changed their around when this all started that the QCA had taken mind about it as well?. I wonder about the online the lowest tender and that it was cheaper to take on testing—going through it and identifying that it had this new company. From what I have heard in this not worked very well, suggesting that they do Chair, I do not believe that. I think that it was based something else instead—because, obviously, more on a desire to not be too reliant on one supplier. enormous disruption and extra costs are caused for There are not many competitors in this area. You the contractor when this happens during a contract. might have picked up on this in the report and I Am I getting the sense of this right? might have missed it, but I got the feeling that it was Lord Sutherland: I think there was a not about the price. There were very few players in misunderstanding. Whether you use the word the area and the QCA wanted a diversity of supply. “genuine” or not—whatever that may mean—about As you know, there has recently been controversy online training of markers, ETS believed that that over education maintenance allowances. A diVerent was the form that the training would take. The QCA company is having trouble delivering those on time. rightly say that the contract did not say that that was What is it called? Liberata. Sorry, I could not the form it would take—it said that they would be remember the name. interested in looking at a proposal that was properly Lord Sutherland: That happens to me sometimes. piloted and showed that online training of markers Chairman: Is it not interesting that, again, there is a would work. There was a pilot and it was not judged great reliance on the company? In an inquiry on to be adequate. There was a disagreement—not only individual learning accounts that we conducted did ETS say that they thought there would be online many years ago when I was first Chairman, we training of markers, they did not have a reserve plan learned lessons relevant to all procurement and all either. They could not have deduced from the Processed: 17-07-2009 19:14:49 Page Layout: COENEW [O] PPSysB Job: 406183 Unit: PAG3
Children, Schools and Families Committee: Evidence Ev 41
26 January 2009 Lord Sutherland of Houndwood KT contract that there certainly would be online Q272 Mr Heppell: But you are saying that we do that training of markers. They did not have a plan B and with every Government contract. Am I not right? they rushed around setting up eye-to-eye, face-to- Did you not just say that that is the normal face training sessions, which went badly wrong. If procedure and PricewaterhouseCoopers did not go you look at the evidence from markers, they were beyond that because it did not need to? appalled at how they were being treated. People Lord Sutherland: That is right: 12 months and £20 turning up in Manchester were told that their million.3 training was in Birmingham; people were not given the right date or adequate notice of when the Q273 Mr Heppell: Every other company outside, training session would take place. ETS would say you are saying, would go further, but the that there was an ambiguity, but the glass is running Government’s procuring policy does not? against them considering their other delivery Lord Sutherland: I know of companies that do that failures. Whether or not there were other kind of probing, whether by telephone or googling. ambiguities, there is a very strong case for looking at If you google first, you will get the press cuttings and online marking. That is a diVerent ball game and I then you say, “Ah, yeah, that’s so and so—no, no.” am persuaded that it would produce a wide range of But that is a serious report. Perhaps we need to benefits—we might come back to that. There was a inquire further. That is what was not done. specific ambiguity—they said that they looked at it in detail, but I do not believe that that is sustainable. Q274 Mr Heppell: And you have a recommendation that that should be changed and become part of Q270 Mr Heppell: You mentioned the fact that there Government— was an assessment of each of the bidders before the Lord Sutherland: I have laid out the facts and bids were actually opened. It would surprise me—as sensible people like yourself can hang on to them and push them on this. we told them that—if we had had the facts in front of us. Is it not the case that the bids should have been reassessed as soon as they were opened? They must Q275 Mr Heppell: But could that not have been one have thought to themselves that the low cost did not of your recommendations? If this is the fact and all match with the services they were saying they could government procurement works under the same provide. Did that not ring some warning bells? system, should it not be changed? Lord Sutherland: You made the point that the Lord Sutherland: If you are asking whether we original estimate by ETS was that 60 staV would be recommended that it should be lower than £20 allocated—they ended up needing hundreds in million and before 12 months, there is a general recommendation that they should probe further into addition to that. That came out of the QCA and 4 NAA’s pockets; they both had to supply the support the companies that they are dealing with. in many cases.2. Certainly,that is one of the probings Chairman: Thank you, John. that did not take place and, at that stage—you are right—it should have been a warning signal, leading Q276 Annette Brooke: Referring back to your earlier to the question, if that is the cost, have they really got comment, Lord Sutherland, I think that one of our their numbers right? Committee’s advisers described this as an accident or a disaster waiting to happen, given the problems in 2005, which made it diYcult to meet the deadlines Q271 Mr Heppell: I am a little intrigued as to what for 2006–07. Why on earth, when changing the reputational issues there actually are, because you contractor, did we decide to make it even harder and seem to be saying that they did an assessment of all to change the goalposts so significantly in terms of the people and so on. What other things would they the timetable? have been looking for if they went online and, Lord Sutherland: That is a perfectly fair, appropriate perhaps, searched for ETS? Are we in danger of and penetrating question, and it is the right one to talking about gossip? That is my worry. ask. Why so many changes and why was the Lord Sutherland: That is where it takes a bit of company not probed? If you have complete probing. It could just be gossip, but if they have confidence in the company, perhaps you would do publicly failed to deliver on contract in previous that. years—I am not talking 1920 here, but earlier, within the current period, of five, six or seven years—that is Q277 Annette Brooke: Right. It also seemed that one of the questions that you would then ask, if you there was not enough time allocated. Is that QCA or had done that probing, to improve their delivery and is it ETS not demanding enough time? performance. Some scathing comments were Lord Sutherland: For? published in reputable journals and newspapers in Annette Brooke: For picking up from the previous the States that may or may not have been justified, contractor and then picking up this year. but again those were suYcient to make you want to Lord Sutherland: I do not believe that that is so. I ask questions. I do not know of any major company believe that there was a long run-in period. The that would not have done that kind of probing if process of preparing to award the contract was they were bringing on board a relatively early, taking place in autumn 2006, which means that the completely new player into the game. 3 See Ev 47 2 See Ev 47 4 See Ev 47 Processed: 17-07-2009 19:14:49 Page Layout: COENEW [E] PPSysB Job: 406183 Unit: PAG3
Ev 42 Children, Schools and Families Committee: Evidence
26 January 2009 Lord Sutherland of Houndwood KT companies bidding for it were thinking about these Q282 Annette Brooke: Should there be a named things in 2006. The contract was awarded at the person responsible for the monitoring, so that it is beginning of 2007. That gave 19 months before the not just with the chief executive? Should it be clear, actual delivery. That should be adequate. with someone allocated the job of monitoring such a timetable? Lord Sutherland: My understanding was that that Q278 Annette Brooke: Right. So the time is there, was the responsibility given to the NAA, which is but presumably things did not happen. So what sort eVectively a division within QCA. You are quite of things did not happen during that period? right, the Secretary of State, the permanent secretary Lord Sutherland: I think this is one of the diYculties. and the director of QCA cannot have every detail There were, clearly, late deliveries on the pilots for constantly in front of them, so the responsibility training markers, for example. There were requests was there.5 to ETS for an outline of the details of the project and whether testing had been carried out. Again, that Q283 Chairman: I am looking at some notes and, was not delivered on time, or in good time, as it actually, the National Curriculum Assessment should have been. There was a whole series of events monitoring team—NCA monitoring—was doing like that. ETS was being questioned, and indeed the monitoring. From the account that I have here, money was held back for late delivery of some of the it was constantly monitoring. A lady called Isabel details in June 2007, a year before the disaster Nisbet is, very early on, flagging up “significant happened. All those were warning bells, which is concerns”—on 21 February—or “two concerns”. why I am allocating significant responsibility to She and her team seem to have been going at it all the those administering the contract. time, but were being rebuVed by NAA. Lord Sutherland: The group you are referring to became Ofqual and were expanded. Q279 Annette Brooke: Did the DCSF know that Chairman: Yes. there were problems in 2007? Lord Sutherland: Ofqual and the NAA had their Lord Sutherland: Those responsible—the National system within the structure—you referred to it in Assessment Authority of the QCA, which was your own evidence-taking as a fairly “cosy” responsible for withholding cash, for example, and relationship, which can cut various ways—but they did—certainly knew. In part, they had a existed within the same organisation, the QCA. The responsibility, if they thought the issue to be serious, lines of arm’s-length—dare I say—accountability to refer it upstairs—or escalate it: a piece of civil were not apparent at that time in the way that they service jargon that I have learned to say.If they think should be. Isabel Nisbet flagged up—we have given it is going to cause major problems, they have a the evidence about that—a number of points. There responsibility to refer it upstairs. The essence of are two aspects, one of which is that she did not much of the diYculty was that each problem in itself always get the evidence from NAA that she seemed to be manageable, but not when you strung wanted— them all together—that, and that, and that, was a Chairman: They refused to give it on occasions. problem. If you are late in delivering here, there will Lord Sutherland: I wanted to say in relation to the be a consequence further down the line, and that was question of Annette Brooke that, additional to the just not taken on board. So, the management of the points we discussed about contracts, I would want contract in the QCA fell down. built into a contract the management information necessary if such monitoring is to be carried out. Ofqual in the future will know that the management Q280 Annette Brooke: Should there have been an information it needs will be presented on such and individual responsible for having an overview? such a date and it will be of this kind and this quality. Surely there was such a person? That is a very important point about contracts in the Lord Sutherland: There was—the chief executive and future. That is one aspect. The second aspect is that the board of the QCA. That was their job. very often, Ofqual then—the nascent Ofqual, pre- Ofqual—was passing the messages back to the Q281 Annette Brooke: That was their job. Given that NAA, whereas in the future the right thing to do is you have told us there was plenty of time for a pass them directly to the director.6 I have even handover, do you think it necessary in the future to suggested that, in the legislation, Ofqual in the be more specific about the timetable—X should future should have a responsibility to report to the happen after so many months, and so on? We cannot DCSF and to this Committee if it feels things are say it is just a matter of time here, because so much going awry and if the “We’re not co-operating” did not seem to take place on time. attitude seems to be breaking out. Lord Sutherland: That clearly was part of the contract, and there were target dates, which is why, Q284 Chairman: May I push you a bit further? What for example, the money was withheld in June 2007. was happening with the NAA? As I read this That was an early warning shot. There is more than evidence, it seems that the NAA is a problem, a that and it has to be laid out. Clear dates have to be 5 specified in the contract, so that those monitoring Note by witness: The responsibility was assigned to the NAA. and managing the system know when lines are being 6 Note by witness: The reference to “director” should be to the crossed and when there are potential failures. QCA Chief Executive. Processed: 17-07-2009 19:14:49 Page Layout: COENEW [O] PPSysB Job: 406183 Unit: PAG3
Children, Schools and Families Committee: Evidence Ev 43
26 January 2009 Lord Sutherland of Houndwood KT blockage; it refuses to co-operate and give answers later. We have not checked how that happens. You to questions. Has it gone native? What has gone do not troubleshoot in advance. It was too late when wrong? Did it get too close to ETS? the problems appeared down the line. Lord Sutherland: Essentially, the status of the NAA was ambiguous. It looks like a separate organisation Q288 Mr Chaytor: So end-to-end testing should be that awards a contract. The NAA is not in a position part of any future contract? to award a contract; it is not a legal entity. It is Lord Sutherland: That is absolutely fundamental. eVectively a division of the QCA, and divisions within these organisations often do not have the high Q289 Mr Chaytor: In terms of the delays in reporting profile that the NAA had. Of course, if you are or flagging up the problems that were identified over basically saying, “NAA, if it’s your job, get on with many months, where does the key responsibility lie? it,” the wisdom begins to accumulate with the NAA Can you put your finger on one issue or is it a and does not move out into the wider organisation. collective failure? Lord Sutherland: I have recommended that there Q285 Chairman: It gets its own culture? should be a thorough look at the risk-management Lord Sutherland: Yes, the wisdom does not move out processes in all these organisations: the DCSF, the into the wider organisation in the way that the wider QCA and any body to which this is contracted. It organisation needs if it is to carry out its own would have been within the NAA, because that was scrutiny in the light of its own responsibilities. I an extra step on the way, and the more steps you think the issue was the ambiguity. Perhaps the have, the less sharp some of these problems appear position should have been made plainer to the further up the system. Do not forget: the QCA has a individuals involved. I am not sure whether ETS huge remit and this is just one element of it. realised that the NAA was not a separate organisation. I am not sure about it going native. Q290 Mr Chaytor: May I just pick you up on that. The diYculty was that it was used to fixing things. If Under the section of the chapter on risk there was a problem, it had a person who knew, and management entitled “Analysis and Findings”, you they would go and fix it. There were just too many do not make any reference to the DCSF and risk things to fix this time round. If there was a problem, management, and nor do you make any it tended to take the view, which is very admirable in recommendation about the DCSF in the two specific some ways, “What are we going to do to sort it out?”, recommendations. That interested me, because you rather than go shouting at the next stage up the line. commented on the DCSF substantially, but you There is something admirable in that, but when it seem to have let them oV the hook when it comes to gets this big, you have real problems. drawing things together and making Chairman: People in this Committee, I know, believe recommendations for change. that there were some extremely good people working Lord Sutherland: Let us be clear, the fundamental in these organisations who did their best even role in this context of risk management lies with though things went wrong. QCA. They have to deliver, with their chosen Lord Sutherland: Yes. contractor, whatever the mechanisms they put into place. The role of the DCSF is, from their position, to be sure that there will be an outcome, but that is Q286 Mr Stuart: Could you give us a quick insight not the detail of risk management and dealing with into what has happened to the people in the NAA? individual problems. An issue should not go up to As you said, it was a division, but it had its own chief the Department unless it really is major, but the executive, so perhaps it was the fact that it was in an problems were not being escalated to the ambiguous position that made it all the more Department. It asked all the right questions but was defensive culturally to try to assert itself. What has given reassurances from QCA, which was given happened to the chief executive of the NAA and 7 reassurances from NAA, which was given other leading people in it? reassurances from ETS. When you are that far from Lord Sutherland: I understand he has been the action, if you have had three bodies who are suspended, and I do not know the outcome of the apparently dealing with it, it is very diYcult to do QCA board deliberation on that. He and Ken otherwise. Boston were the two who were suspended. Chairman: Can we move on? David, you will take us through risk management. Q291 Mr Chaytor: You draw attention to the lack of challenge of the DCSF observers on the supervisory board for the project, but you do not make an Q287 Mr Chaytor: Do you think that if the system explicit recommendation as to how that problem had been tested end to end, all or most of the could be dealt with in the future. problems could have been avoided? Is that the key Lord Sutherland: Yes, there are two things there. One flaw, the key missing piece? is that the role of observers is not a generic one—it Lord Sutherland: It would have been a very diVerent varies according to seniority and which committee position. There may be other diYculties, but this was they are on and so forth. I recommend that that right at the centre of the problem. What happened in should be reviewed and I have put that firmly to the autumn 2007 would inevitably aVect what comes Department in relation to this evidence. This is perhaps going beyond, but I believe that a model to 7 Note by witness: David Gee (NAA Managing Director) look at would be for each observer to have a set of Processed: 17-07-2009 19:14:49 Page Layout: COENEW [E] PPSysB Job: 406183 Unit: PAG3
Ev 44 Children, Schools and Families Committee: Evidence
26 January 2009 Lord Sutherland of Houndwood KT desk instructions as to what their job is, as observer new to the game who did not have the training they on that group. That would be good management ought to have had. Sadly, alongside that, they did without necessarily becoming over-bureaucratic. not have adequate resources and support—someone The observer needs to know, “Am I just watching? they could telephone and automatically get the right Am I taking part in the decisions? Am I required to kind of discussion and answer. There were real pass up the line, and if so, what? Because one up failings there, therefore. Because of the timing of from me will not like it if every time there is a minor some of this, there was a cultural problem—ETS did niggle I race to the oYce.” A very clear desk not know the significance of a bank holiday instruction is needed. weekend. The significance is that that is when many school teachers have some time to do some marking. Q292 Mr Chaytor: Should that be a specific Unless they deliver the scripts in time for that, they recommendation? are behind and the individuals will be back in the Lord Sutherland: There might be better ways of classroom in another week’s time. It is a very doing it, but it is now on the record, and if somebody practical point, but the acculturation did not take asks me how we should deal with observers, that is place and so people were being pressed to take on the kind of response I would give them as a way of marking. Someone would say, “I agreed to mark, doing it. The role of observers caused a long provided I got it by 20 May,” or whatever it was, confusion. The QCA and Ken Boston had some “and it is now 28 May and I have lower sixth, or beliefs about what they were doing and where they primary A, or year 12,” or whatever, “waiting for me were. I am sure that you have been on committees on Monday morning.” They were put under that where there are observers. Do you know what they pressure. That can be resolved. How far was it are doing? Unless you have the desk instructions too, resolved last year? You raise the specific point of 18- it is not evident that you can assume how much of year-olds and so on, and there are a lot of rumours what you say automatically goes back to base, and about. When I saw that I asked, “What evidence do how much does not. we have about who did the marking?” The first point of evidence is the ETS charts of who was doing the Q293 Mr Chaytor: Given your earlier comment marking. There was no diYculty there. QCA looked about the ultimate Secretary of State responsibility, at these in great detail and said, “We do not see any is there nothing at all in respect of the risk evidence of someone who is not an appropriate management procedures within the Department person doing the marking and being listed and other than this particular point about a clearer brief getting paid for it.” That is the evidence. You will for observers on external committees that needs to have to assess this. Some of the evidence from ETS be changed? Are you confident that the existing risk was not as strong as it should be. I would not say the management procedures within the Department are same of QCA because by that time they were deeply adequate? worried and they would have carried out a check like Lord Sutherland: I have said in the report that the that very rigorously. The question is whether they DCSF should, at points, have challenged. I itemised had adequate evidence and paperwork. They believe some of these places in the report. Eventually the they did; that is the only evidence I have, and I share Minister of State asked good and pertinent it with you for what it is worth. We now have to wait questions, to which he did not initially get adequate for Ofqual’s report on what happened in the system answers, so he called in people. Some of the and what the reviews were, in other words, the challenging could have been done by the DCSF referring back of papers that teachers and head prior to it reaching that level. teachers thought had not been adequately marked. Chairman: Annette, you are going to look at the They will say,as they have said every year, “There are quality of markers and marking. problems,” or, “You didn’t get this one right,” or, “This group of children are better”—or in some Q294 Annette Brooke: I think it is the same process. cases—“worse than this”. We will have to await the I want to look at the markers, follow through to the outcome of that. I agree that this is a fundamental quality and then look at whether the children get the problem that we have to pay close attention to when correct results. It is important that they all that report comes out. interrelate, the last one being the most important matter. You said quite a bit about markers in your opening address but there were clearly lots of Q295 Annette Brooke: I seem to recollect that, in rumours abounding: for example, 18-year-olds answer to one of my questions, Ken Boston said that being brought in to mark. In fact it was not every marker had received adequate training. I do necessarily the usual band that got the marking and not have that evidence in front of me, so I say that I heard of people who were asked so late and were with some caution. I am absolutely sure, however, asked to go to the other end of the country for that he said that to me. It is clear, though, from early training that they thought it was not worth it for the evidence that not every marker had the training that money. Did we therefore have a dedicated band of they should have. markers with previous experience? Lord Sutherland: The arrangements were so Lord Sutherland: We had a very large number of disrupted and disruptive of the lives of individuals markers who did, as far as I could see, a good job, as that I would be very surprised if the quality of they have done in the past. Because of the failure of training of the year before was replicated this year. the administrative arrangements, there were some This was for the reason I have already given of Processed: 17-07-2009 19:14:49 Page Layout: COENEW [O] PPSysB Job: 406183 Unit: PAG3
Children, Schools and Families Committee: Evidence Ev 45
26 January 2009 Lord Sutherland of Houndwood KT people not being notified of the right place to go and quite convinced that that can be done and that it not having the support online that they reasonably would be a helpful way of dealing with the pressure, had hoped for. which I wholly understand. Our grandchildren will be going through it very soon. Lots of folks have Q296 Annette Brooke: Again, it is the degree: you good reason to feel strongly about it. could make the claim that every marker had had Chairman: We are coming to the end of the sitting, training, but it might not have been the same quality. but we are going to cover a few more topics. Graham Lord Sutherland: That may be so. will take us through corporate governance.
Q297 Annette Brooke: I also seem to recollect that Q299 Mr Stuart: I shall try to gallop through this Ofqual said at a very early stage that they did not quickly. We have touched on it already, but to what have any concerns about the quality of the marking. extent do you think that the relocation of QCA to Perhaps because it was a relatively new organisation, Coventry and, at the same time, the loss of its it seemed a rather premature statement. regulatory function to Ofqual contributed to QCA Lord Sutherland: I remember that being said—it was taking its eye oV the ball at board level? said to this Committee, was it not? Lord Sutherland: I have thought carefully about Chairman: It was. that, because I heard what was said. In the end, if Lord Sutherland: When I read your evidence I you run an organisation like that—going back to a thought, “Careful, what are the words there?” As I point I have made—you get a remit letter and a remember, the words were, “We had no evidence that resource allocation that you will argue about and the standards would be lower than previous years.” discuss, and those two have to line up, and you either There is an interesting episode of “Yes, Minister” in sign oV on it or you do not. You might think that too which the words “We have no evidence” are analysed much is being asked of you. I have moved in great detail. However, as a result of the reviews, organisations and it is a big issue, especially if they have gone back to look at whether that is true. relocation of staV is involved, but if you accept that it is doable then you have not just sent a message, you have made a commitment. I understand that Q298 Annette Brooke: Finally, in this section, I am there are additional pressures that we all suVer from, sure that you took evidence from schools and the and if this building had to be relocated for six media. We had the impression that the schools were months, we would all be under pressure in the same totally dissatisfied with the results, not only because way, but if you have signed up to it, that is a they had gaps, but because they were not in line with diVerent matter. teachers’ assessments. Would you say that from what you have seen there were concerns about the quality of marking? Q300 Mr Stuart: So you think it is a meeting of Lord Sutherland: There are concerns. One piece of equals? It all depends on the culture, does it not? The evidence that there are concerns is that a much Ministry might want something to happen and an higher percentage of papers have been referred back individual might think that it will cause diYculties this year. I think that it is over 5%, whereas it was and say, “I don’t think we should do this, Minister,” under 2% in previous years. That is a big shift, which and the Minister might say, “Well, I’ve thought is not surprising in view of the noise that there has about it and I think you should.” Are you saying that been. There is discontent out there, but having said they really are in a position to say “No, I’m not that, you have to wait for the analysis of the doing it”? evidence. I confronted one group of head teachers, Lord Sutherland: You can make a public statement. with whom I have regular meetings for other The evident one is to say “Well in that case, I’m not reasons, and asked what it was like for them. They going to continue in this job.” When I was head of are all primary school heads in very diYcult schools the inspectorate, that was always an option open to in diYcult areas, here and in Walsall, and to my me. If there had been pressure on me from Ministers surprise, one of them just said, “No problems: all the that was unacceptable and unsustainable, that was results were in, we were fine; no referrals, and they’re an option. The similar one I made this time was that okay.” So there are diVerent stories about in the if there was any pressure I would go to the press, and system. I had to challenge the head in question and I would. ask “Are you sure?” and she said, “That’s what happened in our school.” There was a group from Q301 Mr Stuart: That comes back to the heart of probably eight schools there at the time and there this responsibility issue which we started with. If we was a variety, and that is why it is so essential to have rely on the fact that people will resign their job every evidence. That is also why I have proposed, and time they are pushed by Ministers to do something indeed recommended, that in future there should be that they should not do, we will not have many access to stakeholder or user panels, which should be people left in public service. The last thing I would set up in advance, so that as the system unrolls do is suggest that people involved in public service through the year, the Department or QCA or this are anything other than people of great integrity and Committee does not have a sudden explosion of genuine commitment. The reality is that people do things happening on the internet or on our website, not routinely resign. Again, to suggest that it is their set up by TES or the BBC. There should be a fault for agreeing to it is to get it completely the mechanism for having groups to whom reference can wrong way round. Did the Ministry push this hard? be made, whom we can ask to tell us the reality. I am Short of Ken Boston saying, “I resign,” which looks Processed: 17-07-2009 19:14:49 Page Layout: COENEW [E] PPSysB Job: 406183 Unit: PAG3
Ev 46 Children, Schools and Families Committee: Evidence
26 January 2009 Lord Sutherland of Houndwood KT prima donna-ish over such a thing—well, I do not Chairman: That was between 1992 and 1994. This know. I am just trying to tease out whether you got pertains particularly to this inquiry.We now have the it the wrong way round. power to interview the applicant for the job of chief Lord Sutherland: I hear what you say. There is no inspector, the head of Ofqual and so on. How do you doubt that that would be the nuclear option that we react to that power of a parliamentary committee? should never rely on—but it is there. There are ways Lord Sutherland: We are in the Wilson room, and in which you can make a clear and straightforward this is a Harold Wilson-like comment. I remember point about this. Ken made his claims about the back in 1979, when Norman St John-Stevas organisation before and how well it would operate proposed that there should be select committees with and how well it would not, but I did not even see a real powers, I said to somebody, “That is going to letter back from QCA saying, “You have asked us to change the way in which the House of Commons do all of this; it is too much.” works.” I am glad to say that it has. I think it is very important that this Committee has a range of powers. This is a pattern that seems to work very Q302 Mr Stuart: Fair enough. You have said that well in the USA. I do not see why it should not work you welcome Ofqual being put on a statutory well here. footing, yet you also say in your report that it was reactive rather than proactive in its scrutiny. Do you think that Ofqual was created too hastily and Q306 Chairman: So you would have applied for the without proper thought to the consequences? job if you had known that you would have to face us? Lord Sutherland: It is always a problem that if you Lord Sutherland: That is a diVerent question. are going to do something new, you will do it in the Chairman: The present chief inspector says she middle of something else. If it had not done it last would not have applied for the job if she had known April, it would be doing it now when it was waiting that she was going to be interviewed by this for the outcome of your response to my report or Committee, so that was a slightly mischievous whatever. That is always a diYculty. I have seen question. some legislation going through in other areas where Lord Sutherland: I was interviewed for the I have wondered about setting up shadow principalship at King’s College many years ago; organisations and appointing chief executives and so there was a student on the committee. I was on. All that has been done elsewhere. It is not as if it interviewed for University of London; there was a is unique. The sooner it got on with it the better. student on the committee. Having a student on the committee often means that it is like holding it in public, which is the issue here. I had no objections Q303 Mr Stuart: Do you thing that being on a to that. statutory footing will make it a better organisation? Chairman: Let us look at the future of national Lord Sutherland: Yes, it will give it powers and if it testing. David. does not use them it will rightly be excoriated. Q307 Mr Chaytor: Looking forward to 2010, there Q304 Mr Stuart: One more question, if I may. An is a question mark over the shape of the tests. There issue that I do not think we have touched on directly will be no Key Stage 3 tests in 2009 or 2010. Do you is that your report is comprehensive in setting out think it wise for the Government to proceed with a what was done, what was not done, by whom and testing system at a higher level of intensity than the when, but it is less consistent in setting out why current one? If the pilots for the single level tests are something happened or was not done. Can you given the go-ahead we will have tests twice a year explain why your report is not more consistent about and every 10 and 11-year-old will be sitting them. Is explaining the reasons behind the failures? that a sensible way to proceed? Lord Sutherland: That is outside the report. Lord Sutherland: The reasons why people do things are many and various. I remember how once in a senate meeting when I was a junior lecturer my Q308 Mr Chaytor: Are there any lessons to be professor of philosophy tried to explain to the vice- learned from your inquiry that might inform future chancellor the diVerence between a reason and a policy about the way Key Stage 2 is assessed? motive, a matter on which there is extensive Chairman: We are trying to get value for our money, literature in philosophy. He failed. The mistake he you see; with all your experience we thought we made was to assume that everything is done on would throw a few other questions at you. rational grounds. It is not. People have motives and Lord Sutherland: Complexity is always an issue. emotions that move them. They are sometimes in First, if you make the system more complex there are entrenched positions. Sometimes they feel additional risks of things slipping up to guard threatened. There are all sorts of reasons. You would against. Secondly, there is a group of able people need a better amateur psychologist than I am to looking at this at the moment. One will read, with chance putting that sort of judgment in a report of fascination, what they have to say. Thirdly, I believe this kind. it is important to have some sort of national testing system. One of the diYculties of the current one, apart from its intensity, is that it tries to satisfy too Q305 Chairman: As a former chief inspector— many diVerent things. What is the point of national Lord Sutherland: I knew I should not have testing? One is that it lets parents know whether or mentioned that. not their children are meeting a certain standard that Processed: 17-07-2009 19:14:49 Page Layout: COENEW [O] PPSysB Job: 406183 Unit: PAG3
Children, Schools and Families Committee: Evidence Ev 47
26 January 2009 Lord Sutherland of Houndwood KT is regarded as, at least, acceptable throughout the Q310 Chairman: Did you have the moment, when nation. Having that in place is important, whatever you were in the middle of this inquiry, that I did form it takes. Another is that it ensures that if there when I looked at all this? There were 9.5 million is a failure in reaching national standards, it is not scripts whizzing around the country—an because of the particular way a school or a local environmental disaster, let alone anything about authority carries out its procedures. That has to be education. There was a vast contract worth tens of checked as well. Also, people do not make enough of millions and people coming from all over the the point that national testing is a way of deciding country to be trained in centres. You can have a whether Government policies and expenditure are national testing system delivered locally. It seems to delivering the goods. They pin their policies and me that you can have a system, where the testing and expenditure to meeting certain targets which are not questions are done by a perfectly respectable group just for the teachers but for the system that they are of people, which is administered and marked locally. planning and funding. I would want any system that Is that not a possibility? develops to take account of those three very Lord Sutherland: I think there are a lot of options. I important tests—or checks—on what our national was reassured by various decisions taken this year testing system should do. that the magnitude of the task has been reduced. There is no Key Stage 3 for a start. That will be very significant considering the timescale we have for this Q309 Mr Chaytor: Do you think, though, given your year. In reducing the magnitude of what is going on, support for the principle of a national testing system, I think this is a trend in the right direction. There are that each of the functions you described can be diVerent ways of doing it. This is not really all that adequately represented by a single form of test? Or relevant but I will say it anyway. I chair the is the logic of your remarks that there should be Associated Board of the Royal Schools of Music. A diVerent forms of test to fulfil the requirements of primary core function that they carry out worldwide each function? is to set standards for musical performance. Now, if Lord Sutherland: I am not in a position to go as far you can do it for that, across the world, it must be as that yet, because I am still thinking about it. I am possible to check across the country whether our reassured, however, to know that people who are children can read, count, spell and write. However, more expert on this than I am are considering it. I the current system has had such diYculties that I will test what they say against those principles, but would be surprised if there were not significant yes, that is a possibility. All teachers assess the changes. children in front of them, if they are good teachers, Chairman: In the words of quite a well-known 20th- so assessment goes on. The issue is that if it is century philosopher, I think we shall pursue that pedagogically important, is it important in relation intimation, with interest. That was a good session. to these national criteria? Thank you, Lord Sutherland.
Supplementary memorandum submitted by Lord Sutherland Thank you for inviting me to participate in your inquiry into the National Curriculum tests by providing evidence on 26 January 2009. I am writing to clarify two points from the evidence which I gave. In answer to question 270, I indicated that the costs for increasing ETS staV capacity were incurred by the QCA. In fact, it is generally the case that ETS incurred the costs for its additional staV—this will have accounted for part of its reported loss on the project (as described to your committee on 10 September 2008 by Dr Philip Tabbiner, Senior Vice President, ETS). In addition, QCA incurred a smaller proportion of costs relating to an increase in the number of its own staV and temporary staV assigned to the project. The QCA Chief Executive indicated that the additional costs incurred by QCA were around £580,000 which is detailed on page 54 of my report. In answer to questions 261, 272 and 275, I would like to clarify that the parameters I referred to were set by the QCA. They were not set by the OYce of Government Commerce, PricewaterhouseCoopers or another body. The parameters set were: — the Pre-Qualification Questionnaire (first stage of procurement) asked for details of previous contracts worth more than £20 million a year (this is detailed on page 25 of my report); and — the press review commissioned from PwC as part of financial due diligence covered a period of 12 months (this is detailed on page 28 of my report). My report recommends that more extensive and additional checks are made in the future in relation to potential suppliers’ knowledge, capacity, experience and track record. 11 February 2009 Processed: 17-07-2009 19:14:49 Page Layout: COENEW [E] PPSysB Job: 406183 Unit: PAG3
Ev 48 Children, Schools and Families Committee: Evidence
Wednesday 22 April 2009
Members present: Mr Barry Sheerman, in the Chair
Annette Brooke Mr Andrew Pelling Mr David Chaytor Mr Graham Stuart Mr John Heppell Mr Edward Timpson Paul Holmes Derek Twigg Fiona Mactaggart
Letter to the Chairman from Dr Ken Boston AO As you are aware, I resigned as Chief Executive of the QCA on 12 December 2008. On 31 March 2009 the QCA Board was given Government approval to accept the resignation. During the interval of more than three months I was under so-called “suspension”. With the belated acceptance of the resignation, the public silence imposed by that particular form of house arrest is over. I now hope again to contribute constructively to the national discussion of issues relating to curriculum, assessment, qualifications and skills, although regrettably from outside the system rather than from within. There is however one matter which I believe it is proper to raise first with the Select Committee, rather than in the media. In his prepared statement to the House of Commons on the release of the Sutherland Report on 16 December 2008—and in his answers to Michael Gove and David Laws—the Secretary of State for Children, Schools and Families set out to demonstrate (1) that I had been complacent in my management of the delivery of the key stage tests; and (2) that I had repeatedly been pressed for answers by Ministers on numerous occasions, the most recent being 17 June 2008, and had given them strong reassurances that the tests were on track. He drew heavily on paragraphs 4.92 and 4.93 of the Sutherland Report (page 77) and on paragraph 4.137 (page 85). The Secretary of State and the Schools Minister, Jim Knight again relied extensively on these three paragraphs in their evidence to the Select Committee on Children, Schools and Families on 4 February 2009. Paragraph 4.92 quotes evidence given to Lord Sutherland by Jim Knight, who says that he met with me and David Gee, the head of the NAA (the assessment division of QCA) on 17 June. Knight’s evidence implies that I was complacent and disengaged at the meeting, and left everything to David Gee. Paragraph 4.93 refers to DSCF’s notes of the meeting, at which I am alleged to have been present. This is fiction. I was not at the meeting Jim Knight arranged with David Gee, nor had I been asked to attend. The DCSF note of the meeting does not in fact list me as one of the attendees. Further, there was no meeting between Jim Knight, David Gee and me, on any date during the period covered by the events into which Lord Sutherland was asked to inquire. Paragraph 4.137 (page 85) is about the escalation of risk by QCA to DCSF during the test delivery period up to the time of the failure, which became apparent on 25 June. It was quoted in full by the Secretary of State in the House of Commons, and again two months later in his evidence to the Select Committee and reads as follows: “In practice what happened in 2008 was that DSCF observers escalated their own assessment of risks to the DCSF ministers on a number of occasions. On this basis, ministers usually pressed QCA’s Chief Executive for answers. At this point, because information was not being escalated within QCA eVectively, ministers were given strong reassurances by QCA that all was on track. As late as 17 June when the Schools Minister met QCA’s Chief Executive and NAA’s Managing Director, they provided reassurances.” This too is fiction. Not only was I not present at 17 June meeting, but until the delivery failure at the end of June I had had only two meetings with the DCSF Ministers in 2008. David Gee was present at neither of them. On 18 March the QCA Chairman, Sir Anthony Greener and I met with Ed Balls and Jim Knight— at QCA request—to discuss the future of the QCA. The national curriculum tests were mentioned only briefly and in passing, and no specific delivery issues were raised by Ministers. On 2 June, the Chairman and I met with Ed Balls and Jim Knight on a range of matters, including the tests: as I advised Lord Sutherland, I did indeed on that occasion reassure the Secretary of State that earlier problems with marker recruitment, marker training and the distribution of scripts had been overcome, and I did so on the basis of evidence which I believed to be sound. I was not asked to meet directly with the Schools Minister in the months leading up to the delivery failure at the end of June, including the critical marking period in the final eight weeks. Nor was I being “pressed” by Ministers for answers on the telephone or by email. There was a flurry of meetings once the delivery failure occurred (2, 3, 24 July and 6, 14 August), but these meetings were about recovering from the failure process, conducting the review of scripts returned by schools because of problems with marking, concluding Processed: 17-07-2009 19:14:49 Page Layout: COENEW [O] PPSysB Job: 406183 Unit: PAG3
Children, Schools and Families Committee: Evidence Ev 49
the contract with ETS and tendering for a new supplier for 2009. They were not about the causes of the delivery failure and the events leading up to it, which were the matters on which Jim Knight was asked to give evidence, and on which Lord Sutherland reported. During the test delivery period, QCA was of course closely monitoring the blogs and the many reports in the media about problems with ETS, and working around the clock to resolve every issue which arose. We were in close contact with DCSF oYcials, who were also monitoring the situation. Many of the problems were successfully addressed by ETS, but some quite major ones could not be resolved. Many allegations proved to be false, such as the assertion that ETS was employing a first year undergraduate to mark Key Stage 2 papers: he had in fact been employed by a school to assist in the review of marked returned scripts. The flawed evidence on which paragraphs 4.92, 4.93 and 4.137 of the Sutherland Report is based has been used to portray me as complacent, disengaged, and constantly beleaguered by Ministers with questions I was unable to answer. This is far from the truth; it was not corrected by Ministers or DCSF oYcials at draft report stage; and it has been used by Ministers to my serious disadvantage. All this had no bearing on my decision to resign. Clearly however, the record should not be allowed to stand. The Secretary of State and the Schools Minister owe an explanation to Lord Sutherland, to the House of Commons, to the Select Committee, and to me. And the explanation cannot be that Jim Knight was simply mistaken about the date on which he thought he met with David Gee and me, because there was no other meeting with which it could reasonably have been confused; nor that there were occasions other than 2 June when I was pressed by Ministers for answers, because that is simply not true. 15 April 2009
Witness: Dr Ken Boston, AO, former Chief Executive, Qualifications and Curriculum Authority, gave evidence.
Q311 Chairman: Can I welcome Dr Ken Boston to terms of reference for Sutherland, because I think our proceedings. Ken, we have been waiting a long there is a lot to learn from that. Thirdly, I have a time to have evidence from you. Your interpretation couple of points about the role of observers on the of your contract of employment was that when you QCA, which I think has some lessons for Ofqual. Let were under suspension you could not appear before me begin by saying, in relation to the letter, that I the Committee. One member of the Committee may have resigned and I am not seeking to mitigate that ask you a question about that. You will know that in any way. Even if Lord Sutherland had been able we broke all previous records by holding a meeting to get to all the causes of the failure, and he was not of this Committee in the summer recess last year, able to get to all the causes because of his terms of such was the urgency and importance, we thought, reference, I would still have resigned because of my of the tests situation at that time. So you know it is part in it. So let there be no question that what I have of concern to this Committee. You know that we to say is attempting to go back over that ground. The have in a sense wrapped the issue up with the letter I sent to you sets out what I think are two problems that we faced with education maintenance major flaws in the evidence—quite incorrect and allowances, which are totally separate and not part unsound evidence—which has been used against me. of your remit at all. But there were two systems, both At 2 o’clock this morning on the radio I heard that of which seemed to fail. Independent companies that the Department had in fact put out a statement were hired to do a job did not seem to be able to saying that Jim Knight had corrected this evidence deliver satisfactorily. We hope that this evidence some time ago. I was not aware of that, but I think session will allow us to come to some conclusions any correction raises more questions than it answers. and to write up our report, so there is a lot hanging The whole point of my letter is the last sentence, on this meeting because we have been waiting for which seeks an explanation as to why paragraphs such a long time. This is the only Committee that 4.92, 4.93 and 4.137 in the Sutherland report are actually got evidence from the American company without foundation. There is no fact to them, and yet that failed to deliver. That company did not give they were used against me, to my disadvantage, in evidence to the Sutherland inquiry, but it did give the House of Commons and in this Committee. I evidence to us, so we have a particular responsibility want an explanation for that, and the explanation in this matter. Ken, we have known you for a long cannot be simply that Jim Knight was mistaken time in your role. We have had a valued relationship about the date on which he thought he met with with you, so we start on that basis, but our job today David Gee and me, because, as my letter sets out, is to find out what went wrong and why it went there was no other meeting with which it could wrong. As with all witnesses, I am going to ask you possibly have been confused. Secondly, there were if you want to make an opening statement. no occasions—other than the occasion on 2 June— Dr Boston: Thanks, Barry. I would like to make an when I was pressed by Ministers for answers. You opening statement, which touches on three things. I will recall that in the House of Commons there was think it might be a useful start. One is the letter that some banter about what “usually pressed” means, I sent to the Select Committee.8 The second is the and the Secretary of State clearly used it to mean “pressed on several occasions”. There was one. I 8 See Ev 48 want to know why the document was so flawed; why Processed: 17-07-2009 19:14:49 Page Layout: COENEW [E] PPSysB Job: 406183 Unit: PAG3
Ev 50 Children, Schools and Families Committee: Evidence
22 April 2009 Dr Ken Boston the flaw was not discovered sooner; why the DCSF spotlight on the QCA and ETS, prevented some oYcial, who took the notes of the meeting, did not major causes of the failure from being identified and draw the mistake immediately to the Minister’s compromised the independence of Ofqual and its attention when the transcript of the evidence came in capacity to determine its own aVairs. No doubt the a few days later from Sutherland, when the error Government wanted to avoid any risk of a public must have been clearly there; and why evidence given inquiry into the purpose and use of the Key Stage on 14 October last year was not corrected, to my tests. But I think that Ofqual should have been knowledge, until 2 am this morning. I think that trusted, on that point, as a regulatory authority reasons are required, and an explanation is needed as rather than a body for policy review. It is beyond to what that is all about. And what does Lord question that it could have been relied upon to give Sutherland think? He was given, in good faith, false Lord Sutherland terms of reference which focused evidence, which he had published. If I were him, I broadly on the programme. I have no doubt that would be furious, and I would be seeking an decisions made by Ministers and oYcials in 2006 had explanation. On the second point, on the issue of the a marked impact on the failure in 2008, most notably terms of reference of the Sutherland report, I believe staying with manual marking rather than moving to that one of the lessons that we have learned— the phased introduction of onscreen marking, as because we need to look ahead—is that Lord with GCSEs and GCEs. Four years after it was Sutherland should have been given a remit to recommended to Ministers by me and others in the examine the role of all three partners involved in the QCA, onscreen marking has become one of Lord development and the delivery, which are the Sutherland’s principal recommendations. The Government, the QCA, and the contractor, ETS. further contributory factor, of course, was the His inquiry, as we know, was restricted to the QCA question level data being captured in the marking of and ETS alone. Yet the Government, through the the 2008 tests—Lord Sutherland’s references to DCSF, determine the nature of the tests to be several thousand clicks being required for each procured, the pupils who are to be tested, which bundle of papers, you know. My final point, briefly, subjects they are to be tested in, how and when they is that I think the appointment of DCSF observers are to be tested and how much money has to be spent to the QCA board, and other QCA bodies, has on testing. The interval between the date of the test undercut the authority of the QCA and will undercut and the date for results—the eight weeks’ interval— the authority of Ofqual if we are not careful. By is set by the Government. They also determine reducing the formality of the relationship between whether the tests are marked manually or onscreen; the two organisations—the Government and the which results data are collected and how they are QCA—it leads to negotiated compromises from collected; the form in which the data are required; time to time, which erode public accountability. The and how the results are returned to schools. In those role of observers on boards and committees has been core elements of test development and delivery, to take part in discussions, to provide advice to neither the QCA nor the supplier has any Ministers separately from the QCA and to relay discretion—decisions are made by the Government ministerial feedback. It is in that third area where the alone. The development and the delivery of these problem arises. Observers typically advise in tests are not at arm’s length from the Government; committees and boards that Ministers would be the Ministers and the oYcials are at arm’s length “minded to” or “not minded to” agree with this or only from the detail of the test questions, and from that proposal or “content to” or “not content to” the marking and level setting. I believe that the agree to a particular recommendation, or even that, failure in the delivery of the Key Stage tests in 2008 “If I put that idea to the Minister it would be laughed presented Ofqual with its first opportunity to out of court.” Now certainly,prior consultation with establish itself as a truly independent regulatory the DCSF is important to inform subsequent authority. If the Government had genuinely wanted decisions by the QCA on what advice it should oVer Ofqual to assume such a role, and to get to all the Ministers, but too often boards such as that of the QCA are put into a position where it is expected that root causes of the failure, it would have asked Ofqual they will seek to negotiate that advice in advance. I to establish terms of reference for Lord Sutherland think that is a pernicious process that compromises which encompass the role of all three partners in test integrity and independence, and if we are not careful, development and delivery. Consistent with in relation to Ofqual, it will cause real diYculties Government rhetoric, it should have been beyond there. I do not think that for Ofqual the runes are question that Ofqual, with Sutherland as its agent, propitious at all. The reality of true independence for would account to Parliament through the Select Ofqual will need to be asserted, defended and won. Committee, not to the Secretary of State, and report QCA, from time to time, has been seen as far too on the roles of all three partners in the delivery independent, and that is one reason why its wings are failure. I think that in giving Lord Sutherland being clipped as the QCDA. The greatest risk for specific terms of reference to look at the QCA Ofqual at the moment is of being cynically management of the ETS contract, which inevitably outmanoeuvred by Government. I think we have to would have been the critical part of any Ofqual be very careful about that. inquiry, and in asking Lord Sutherland to report to him—not Ofqual—on that matter, the Secretary of State narrowed, not broadened, the scope of the Q312 Chairman: That is a very powerful survey of inquiry. His intervention put a protective fence how you see the situation, and we will now start around the DCSF and Ministers, focused the some questioning, but can we just clear one thing up. Processed: 17-07-2009 19:14:49 Page Layout: COENEW [O] PPSysB Job: 406183 Unit: PAG3
Children, Schools and Families Committee: Evidence Ev 51
22 April 2009 Dr Ken Boston
You talked about Jim Knight’s response. We have should be delayed, and I agreed to provide him with given you a copy of a letter received by the advice on the length of that delay. That is in fact Committee—I am not sure whether you saw it; from another element of the Sutherland report that is what you said, you did not.9 Jim Knight admitted wrong. It reports—I do not recall the exact words— that he had made an error in evidence given to Lord that I advised the Minister to extend the date, but I Sutherland; his letter is dated 9 February. Then you did not. My advice to the Minister was to stick with have Lord Sutherland’s reply, saying that he did not the date, and when he asked how long it should be think that that made much diVerence to his report. I extended if he insisted on doing so, I gave him the just thought that we should clear that up before we figure of a week. I remember the 2 July meeting very go on to the substance. well. It is beyond the terms of reference of Dr Boston: This is the first time, Mr Chairman, that Sutherland, and the description of it—it was quite an I have seen this letter. I think that it is disgraceful intense meeting—has no relationship to the that, if I was misrepresented by a Minister in characterisation of my performance or behaviour in evidence in a public report, I have not been the meeting which is set out in paragraph 4.92. contacted by the Minister. Q315 Chairman: Can I take you back. On the one Q313 Chairman: So you have never seen this letter? hand, the Committee wants to find out what went Dr Boston: No, I have not. I have not been given any wrong. You, as the QCA, hired this American information on this at all. I think that is outrageous. company, and all of us agree that it failed to deliver If as, just glancing through, I see in the last in the way we anticipated it should, in terms of our paragraph on the first page, the Minister is confusing hopes for the students taking the exams. Before I it with a meeting that took place on 2 July, and if that bring you back to that, let us be clear that you are meeting is the subject of paragraph 4.92, that is also very angry about how your role in this has been wrong. The Minister knew on 26 June that there was characterised by the Government. a failure. The meeting on 2 July was put together at Dr Boston: Yes. his request to talk about where we would go from there, with regard to releasing the results. It was the Q316 Chairman: That is very clear, and I understand first meeting I had with the Minister along with that and you have had three or four months to think David Gee, and others were present. about it. On the other hand, would not the Ministers say to us that they had asked the QCA to hire an Q314 Chairman: That was with Jim Knight? independent company to provide value for money Dr Boston: Yes, with Jim Knight. We had before us and deliver the test results but that did not happen? a set of data from ETS that was absolute nonsense. Whatever we say about which Minister met you and It showed very clearly that it was going to fail, but when, the fact is that the QCA hired these people and even from ETS we were getting some confusing they failed to deliver. What do you say to that? signals. The president of ETS in America sent an e- Dr Boston: I say that is exactly true. That is why I mail the next day, 3 July, as you might recall from the resigned. I was managing an organisation that had a evidence in Sutherland, saying that he thought that diYcult task ahead of it—we can talk about aspects they were still going to make it. All the evidence we of that—but it failed and I resigned. What I resent is were getting from the ETS people locally was that evidence against me being sexed up in a report by they were not. Jim Knight knew on the previous Lord Sutherland, on the basis of false evidence given Friday that this was not going to succeed. The by Ministers, to characterise me as something I am meeting on 2 July was to sort out where to go from not. I failed to deliver and I resigned. That is it. there. Therefore, that was in my view beyond the terms of reference of Sutherland. Sutherland was Q317 Chairman: In terms of the broader brush, looking at the reasons why failure occurred and the though, in your introductory remarks you suggested whole run-up to the failure. At that point, on 2 July, that there was something much deeper at the heart failure was evident and known. The key argument of the problem, and that is that the QCA has been we had at that meeting, which was very intense, was abolished—we now have two organisations—and whether to release the results. Jim Knight asked me the Ofqual that is emerging is weaker than it for formal advice on delaying the results day by a should be. week, and I refused to give that advice. My advice Dr Boston: We were first advised of the changes was that the results available on 8 July should be being made to the QCA by Ed Balls on 19 September published, and as events turned out, 93.5% of Key 2007—that he wanted us to split. I support Stage 2 results and 85% of Key Stage 3 results were absolutely the notion of a regulatory authority available on the results day of 8 July. A large number reporting to Parliament rather than to Ministers. of schools, a minority,would still not have got results That has always been a diYculty with the QCA: it is for all three subjects at the one time, but my responsible for maintaining the assessment argument was that letting the results go out would at standard, the height of the hurdle, yet reports at the least allow the majority of schools to have results, same time to the Ministers who want to drive up the rather than penalising the whole lot so that they performance standard—the number of people who would not be able to complete their evidence. The leap the hurdle. There is a real inconsistency there. In Minister stuck with the decision that the results my view, and this was my view way back in 2003 and 2004, it would have been much better either to have 9 See Ev 77 hived oV the operating arm, the National Processed: 17-07-2009 19:14:49 Page Layout: COENEW [E] PPSysB Job: 406183 Unit: PAG3
Ev 52 Children, Schools and Families Committee: Evidence
22 April 2009 Dr Ken Boston
Assessment Agency arm, to an awarding body, or to Dr Boston: Yes, indeed. have set it up as a separate authority, or even to have Chairman: So the relationship between the abolished it and given an awarding body that Department and the QCA is something that you function and the contract, and put the whole of the have thought about and reflected on since. I am rest of the QCA, including its regulatory side and its trying to tease out when the unhappiness that you curriculum side, in a statutory authority that had about that relationship and the changes that reported to Parliament and not to Ministers. That were mooted in the QCA emerged. I am trying to ask would have been a better solution. Regarding the you whether these are the words of a bitter man who notion of now taking the statutory authority role of feels that he has been wronged and is picking a fight the QCA away, there has been agitation there, and with the Government, or have you had these views that became very clear in the meeting we had with Ed for some time? Balls and Jim Knight about the future of the QCA Dr Boston: I have had them since the decision to set on 18 March. One of the great sources of agitation up Ofqual, which I strongly support, but I have had has been the way in which I have played my role and real concerns about the related decision to dismantle allowed other senior people to play their roles, the QCA. The meeting on 18 March was very tense. including particularly perhaps Mick Waters, the I was not the only one involved; the chairman and Director of Curriculum. The QCA was set up as a another board member from the QCA were also statutory authority under the 1997 Act not to be a present. It was very clear that the Secretary of State critic of government but to be a constructive and the Minister thought that we were handling our job in the wrong way. It has been diYcult from then participant in educational discussion nationally, and on. In fact, in the whole run up to the test delivery I have seen it lead part of that discussion. I have seen series to the end of June, I had only two meetings how it has not been bound by a set of speeches that with Ministers—in the entire year—as my letter says, were basically constructed around the Government one of which did not even cover anything about the press releases of the day, and how it has never national curriculum tests. When I go back through publicly criticised Government but has been my diaries for previous years, that is very much at prepared to shoot the breeze about other ways of odds with the previous pattern; during the crucial doing things. Notably I, and Mick in particular, last eight or 10 weeks I would have met with have done that on curriculum and indeed on changes Ministers four or five times in previous years. to assessment and modernisation. Now, the Chairman: It is certainly true that when you gave QCDA—the big debate has been about the right role evidence to our inquiry on testing and assessment of the word “authority” rather than “agency”—will you shared some of your concerns at that time. Can not have that capacity. It will be virtually an arm of we hold that for a moment David. the Department, and my personal view is, why not incorporate it within the Department? What use is it going to be as a body that is at arm’s length in one Q320 Mr Chaytor: There are two separate issues sense but not able to do anything in terms of having here. One is the relationship between yourself and a public role? the QCA and Ministers in the context of the appropriate degree of independence for the QCA and its successor bodies. The other is the Q318 Chairman: This is the last question from me. In responsibility for the events that led to the failure of your view, the terms of reference of the Sutherland ETS to deliver the tests on time by 8 July. You are inquiry were too narrow and excluded looking at the trying to conflate the two and to suggest that there Department’s role in all this. was a hidden agenda that led the Sutherland inquiry to focus only on your responsibility in not delivering Dr Boston: Yes. the contract to time. In retrospect, do you not accept Chairman: But what about the overall independent that there was a responsibility on the QCA and the scrutiny of Sutherland? Sutherland, as you know, is, NAA, as the agency directly responsible for like you, a highly respected person in the educational supervising the contract, to inform the Government world. Do you not believe that he made a fair at an earlier stage of the scale of the risk that you had appraisal and did a fair job, given his limited remit? identified? You said in your evidence last year, and Dr Boston: I think he did a very fair job, and I have you say again in your letter,10 that there had been said that on the public record several times. I think weekly meetings from September with ETS and that it was a very good report, but he was able to daily meetings from the beginning of May. Surely look at the role of only two of the participants, not that is exceptional—if there was no problem or if the all three, and he was fed some information that was problems were being dealt with, why was there a wrong. If I were him, I would be furious that it was need to have daily meetings from 1 May? wrong, given that he had given two opportunities Dr Boston: No, that was not exceptional. That was through the process for it to be corrected. It now in fact the plan, and it was very good management stands on the record erroneously. If I had been the and was similar to previous years. The daily inquirer, I would be very angry about that. meetings in the final few weeks are the norm. It was not because there was any particular crisis. As I have said in evidence before, right up until late May and Q319 Chairman: You had a good and quite the start of June there were some problems with harmonious relationship with the ministerial team for five or six years. 10 See Ev 48 Processed: 17-07-2009 19:14:49 Page Layout: COENEW [O] PPSysB Job: 406183 Unit: PAG3
Children, Schools and Families Committee: Evidence Ev 53
22 April 2009 Dr Ken Boston getting papers out to markers. There were problems We wanted to introduce onscreen marking for with distribution and return. They were fixed, and by security, better quality and greater validity of results. 6 June 50% of the marking had been done. We were The Department was pushing continually for a trial on target, just as we had been in previous years. The that would have had some papers marked manually signs then were very good. It was not until, in my and some marked onscreen and, provided that the case, 25 June that it became clear that we were not results were the same, onscreen marking would be going to make this. gradually phased in. The whole point is that the results would not be the same because the onscreen Q321 Mr Chaytor: Given the scale of the problems results would be better. I do not know whether that described by yourself and by Sutherland and would make the league tables look better or worse, acknowledged by ETS in its evidence to the but the one thing you can absolutely bank on is that Committee, it seems incredible that as late as 24 June you will get a diVerent, better and more valid result there could have been a general acceptance that the with a process that identifies aberrant marking and target date of 8 July was going to be met. You refer, other errors early, does item level data collection for example—I have the transcript of your evidence automatically and eliminates all that manual stuV. last year—to this backlog of 10,000 e-mails that ETS You will get much better results, but they will be had not responded to. This led to an identification of diVerent. The compelling argument, finally, behind the support provided by the NAA. This is closed doors where all of this was thrashed out was, exceptional, is it not? Which organisation, operating “Well, it’s too risky—the results might look normally, has a backlog of 10,000 e-mails? diVerent.” Dr Boston: Yes. The Department was fully aware that that was the backlog, fully aware that we were Q323 Mr Chaytor: Can I ask also about the role of coping with it, fully aware that we had put in the DCSF observers, because your argument is that resources to deal with it. It was a problem, and we they were involved with the NAA and the delivery of were working to fix it. In fact, the backlog was the contract at every stage? They were present at the removed. It is not a matter of the Department not weekly meetings from September. They were present being aware of issues from day to day. The contact at the daily meetings from 1 May. Why did they not we had through the NAA and elsewhere was very report to Ministers earlier the scale of the risk and close. Department observers attended the meetings the likelihood of failure? we were having with ETS and others. The broader Dr Boston: I cannot answer that. I don’t know. But it question is that every year this has been a high-wire is true that they were present at every level, including act. It is to do with the extraordinarily large scope of board level. They were aware fully of decisions being the job, done manually,and packed into eight weeks. made and in fact were participating and contributing We failed in 2004 when Key Stage 3 was late. We to that. The ETS contract was not even signed by the were within hours of failing in 2005, even though we QCA chairman until we had the then Secretary of had extended the Key Stage 3 results date. In 2006 State’s approval for it to be signed. She was being and 2007 we were fine. In 2008 the whole thing briefed by oYcials on, presumably, where this whole collapsed again. The enterprise, the whole archaic thing was heading throughout. I am not objecting to nature of this thing, is incredible, and we are that, nor am I absolving the QCA from persisting with it for 2009. Lord Sutherland has responsibility. It was the management agent. I am made recommendations about end-to-end testing not saying that we should not have taken the rap on which simply cannot be met for 2009. Key Stage 3 is this. It happened on our watch, but it cannot be said not being run this year but that does not halve the that it came out of the blue or that the warning signs risk. It simply means that there will potentially be of continued briefings were not making it apparent more markers who will be able to do Key Stage 2. In when diYculties arose. the general qualifications where the major awarding bodies have applied technology, are doing onscreen Q324 Paul Holmes: I have two questions. First, in marking, producing valid and reliable results and the letter that you sent to the Committee on 15 identifying aberrant marking quickly and delivering April11 you say, “I was not asked to meet directly a top-quality product securely, an enormous with the Schools Minister in the months leading up achievement has been made. Key Stage 2 and Key to the delivery failure.” We have already talked a bit Stage 3 is still the cottage industry it was in 2002. The about that. Why were you not asking to meet the only change is that papers are now no longer left on Minister? Why had you not spotted that problems doorsteps and post oYce counters—they are were emerging? You have just said that it did not barcoded and tracked. That is the only change that become apparent until 26 June. How come you did has occurred in this period. not spot that at an earlier stage? Dr Boston: We were certainly spotting the problems Q322 Mr Chaytor: You said earlier to the Chairman and dealing with them. As I said in my evidence to that you had recommended four years ago that Lord Sutherland, it is a fraught process. There are onscreen marking should become the norm for Key diYculties the whole time. Our approach was to roll Stage 2 and Key Stage 3. What was the Minister’s up our sleeves and get on with solving it, rather than response to that recommendation? running round telling people that there was a Dr Boston: There was extensive discussion with problem. If I had had a meeting with Jim Knight Ministers and oYcials particularly over a long period of time about introducing onscreen marking. 11 See Ev 48 Processed: 17-07-2009 19:14:49 Page Layout: COENEW [E] PPSysB Job: 406183 Unit: PAG3
Ev 54 Children, Schools and Families Committee: Evidence
22 April 2009 Dr Ken Boston every week for the last eight weeks in the run-up to Q326 Paul Holmes: Is there a wider institutionalised the closure, it would not have made him accountable problem about lines of accountability in general for the tests. It was still my accountability. It would between the Government and diVerent arms of not have helped simply to put problems on his desk. business that are being hived oV? For example, in Our objective was to keep him and his people this case a private company, ETS, was brought in, briefed, but solving the problems and addressing and it was a disaster. We had a similar thing some them and bringing the tests in. We have done that years ago with the delivery of individual learning year on year on year, except in 2004 and 2008. It has accounts and Capita and so on. We get private worked before. Our anticipation was that it would companies, and the Ministers say, “It’s up to them work again, but the risks in this are so high. now—don’t ask us about it, ask them.” The Sutherland is critical of the QCA board having the companies say that it is commercial confidentiality tests at an amber risk on its risk register for 14 and it all collapses and is a disaster. That seems to months. Amber means that likelihood is possible happen over and over again. My second point and the impact would be very high. I cannot see these concerns quangos, whether it be the QCA or NAA tests ever having a risk rating less than amber. They or whatever. We have seen the Learning and Skills could always fail, given the eight weeks in which the Council make a total disaster of sixth-form funding delivery occurs when they are done manually. And allocations and of funding for college buildings. the impact when they fail is—well, we saw it in 2008. However, the Minister keeps saying, “It’s nothing to do with us, you’ll have to ask them.” Have we got an Q325 Paul Holmes: We will come back to the lines of institutional problem, which has developed over the accountability in a minute. Given that you are last 10, 15 or 20 years, about what the lines of saying that it is always on a knife edge because of the accountability are between the Government and the sheer volume of tests and the short time scales, why private sector and quangos, who are delivering were you not banging alarm drums much earlier, things further down the system? both last year and the year before, and the year Dr Boston: Yes, I think we do. From my experience, before for that matter? Both you and I were at an I certainly think that there is a problem. Looking educational seminar—this was in the early stages of forward, that is a critical reason why Ofqual must be the development of diplomas. When we were completely separate from Government and report to chatting, you said you were worried about what was Parliament through the Select Committee. The happening with the development of diplomas, and QCA’s problem has been that it is not at arm’s that you thought the Government were being too length—to use the Government’s term—yet it can complacent and that there was not enough feed end up carrying the blame for a whole set of through from those people developing diplomas to decisions over which it had no control. In my the schools and employers. You asked me to urge the statement I pointed out all the things to do with Committee to look at that. I think you also talked to national testing that the Government determine, the Chairman separately and probably to other and it is virtually everything except what the people. The Committee looked at the matter at that standard is and responsibility for the marking. If one stage, and it did gee the Government up to get their goes further, and looks at awarding bodies, one sees act together a bit more. In that case you were really that the delivery relationship works eVectively on the ball. Because you were not getting the because awarding bodies are quite remote. They do response that you wanted from the Government, not work with Government even on a contractual you used the Committee to try and push things along. If you were very much on the ball in the early basis. They deliver a public service and are funded stages regarding concerns about diplomas, why if through schools to deliver that service, and it works. year after year the marking was such a problem were Distance, with Ofqual, is going to be critical. If it you not banging the drum year after year? If you does not get that distance, there will be a problem. were not getting anywhere with the Ministers, why As I said, I believe that Ofqual was made to stumble did you not raise the matter more widely? at its first hurdle. That is not an auspicious sign for Dr Boston: That is a good question, and it is one that the future of independent regulation. I have asked myself. I have frequently spoken publicly about diVerent approaches to testing and the problems with this particular approach and how Q327 Mr Stuart: As you have described it, the it was administered. I have also pressed privately for system eVectively has been specified by reform with Ministers and oYcials—there is no Government—they are the architects of the system question about that. Over the past few months I in almost all its components, except for the content have asked myself why I did not resign in 2006. At of the exams and the style of the marking. We have that point it was clear to me that I was not going to a constitutional system, where Ministers are always get Government buy-in to necessary reform. I responsible. You said that if you had met every week continued through 2007 and 2008 to run an with the Schools Minister, he would not have been organisation that was faced with close to an accountable—you would have still felt accountable. impossible task, which I had seen go belly up in 2004 To follow up the last question, do you feel that the and had the prospect of doing so again. Probably my constitutional convention of Ministers being greatest regret in all of this is that I was not more ultimately responsible for delivery, particularly honest with myself at that time and did not simply when the system is not genuinely independent, has say, “It can’t be done.” failed to be honoured in this instance? Processed: 17-07-2009 19:14:49 Page Layout: COENEW [O] PPSysB Job: 406183 Unit: PAG3
Children, Schools and Families Committee: Evidence Ev 55
22 April 2009 Dr Ken Boston
Dr Boston: Yes, I do. There comes a point where you within eight weeks using technology, giving us much have to accept your accountability. From my own more valid and reliable results than we get at the point of view, I accept the accountability and I moment. accept the responsibility. On matters of detail, I was not at fault, because I was not doing those detailed things, but I was responsible for the management of Q331 Mr Stuart: So the chief author of this failure, the people who were delivering them and for the in your opinion, was the Department, because of the overall result. It would seem to me that a time does specification that it set for the way that these tests come with all Ministers—over my career, I have were to be assessed. And, in the evidence to the worked with more than 40 Ministers—when a Sutherland inquiry, the Department failed to accept Minister has to accept accountability and go. that responsibility. Dr Boston: I certainly believe that, if we had been successful in introducing onscreen marking several Q328 Mr Stuart: The evidence that Sutherland years ago, 2008 would not have happened. quotes, particularly paragraph 4.137, says: “In practice in 2008 what happened was that DCSF observers escalated their own assessment of risks to Q332 Chairman: I have just listened to your replies the DCSF ministers on a number of occasions. On to Paul Holmes and Graham Stuart. Are you not this basis, ministers usually pressed the QCA Chief trying to have it both ways? On the one hand, you Executive for answers.” Is that an accurate account are complaining that the record is not straight and of what happened? that you hardly ever saw Jim Knight or Ed Balls. In Dr Boston: No, it is totally false. I was pressed, if other words, you were left alone to get on with the asking a question is being pressed, by Ministers—Ed job. On the other hand, you are saying that there Balls and Jim Knight—on 2 June, at the start of a should have been greater ministerial responsibility, meeting, as the first agenda item in a meeting that because when everything went wrong Ministers covered a series of agenda items. I am aware that in should have said, “Hands up, it’s all our fault, the House of Commons two Members sought because of decisions that we made, or failed to make, clarification of what “usually pressed” was, and the some years ago.” So let us get this straight. You are Secretary of State referred to that as meaning complaining: you said that Jim Knight made a “frequent pressing”—those were not his exact mistake, saying that you were at a meeting that you words, but that was the clear implication. were not at. It seems to me that someone could interpret what happened as the Government saying, Q329 Mr Stuart: So Ministers’ role in the system was “Get on with the job, Ken. Hire someone to do this to ensure, the Schools Minister told Sutherland, that job. You are responsible. You’re a good guy and “the reality clearly is that some of these tasks that you’ve got a good reputation, so get on with it.” they are doing are so mission-critical that we need Actually, Ministers do not really get involved until alongside that arrangement to be satisfied for everything seems to be going wrong. That is true, is ourselves that things are going well.” That was the it not? responsibility on Ministers. Is it your evidence to us Dr Boston: Yes, that is certainly true. Ministers were that they falsely maintained to the House of not getting involved in the detail of it at all. Look, I Commons that they had carried out that duty when don’t think I am trying to have it both ways. I have in fact they had not done so? very clearly fallen on my sword over this. I have Dr Boston: I am not certain about that, because I am taken responsibility,taken accountability,and that is not privy to the amount and detail of briefing that the end of it. Ministers were given by DCSF observers at our own meetings. I cannot really comment on that. Presumably, the briefings were sound. That is why I Q333 Chairman: But I am talking about the find it so curious, for example, that the senior DCSF ministerial responsibility,Ken. On the one hand, you oYcial who took the note of the meeting of David are saying that the Ministers were never there, Gee and Knight on 17 June did not draw the because they left you to get on with the job, and now Minister’s attention to this mistake a few days later you are saying, “But we need clearer . . . ”, and we when the transcripts came back from Sutherland. need Ministers to stand up and say that they got it The relationships between Ministers and DCSF all wrong. oYcials over this I cannot fathom. Dr Boston: If I had met weekly with Jim Knight, I think that this problem would still have emerged. I Q330 Mr Stuart: I know that it was a complex set of am not saying that his failure to meet regularly was circumstances that led to the failure of delivery. the cause of what happened. I take your point. However, do you believe that if the Government had However, I certainly do not think that I am, in any accepted your advice and brought in phased way, trying to diminish, reduce or explain away my onscreen marking rather than manual marking the own responsibility. I was foolish enough to continue deadlines would have been met? running an organisation doing this high-wire act, Dr Boston: I am certain of it. The evidence for that when I knew fundamentally that it was deeply is what we see with the awarding bodies and the flawed and that the only way that I could have made general qualifications. There is no question that it secure was to get ministerial agreement to do so these tests could be delivered eVectively and well several years ago. Processed: 17-07-2009 19:14:49 Page Layout: COENEW [E] PPSysB Job: 406183 Unit: PAG3
Ev 56 Children, Schools and Families Committee: Evidence
22 April 2009 Dr Ken Boston
Q334 Fiona Mactaggart: When did you first learn of and of the fact that there had been a statement until the public account by Jim Knight that was wrong, 2 o’clock this morning. I think, in all fairness and and what attempt did you make to correct it? Was it decency, that I should have been given at least this the letter that you wrote a few days ago to our letter and an explanation as to why I was so Chairman?12 misrepresented. Dr Boston: Yes. Q337 Fiona Mactaggart: That is a reasonable Q335 Fiona Mactaggart: Why? request, but one of the things that you have alleged Dr Boston: I believed that I was unable to do in your evidence to the Committee today is that anything prior to that. Let me say that I first became Ministers—in the phrase you used—“sexed up” the aware of this error—the flawed evidence—on 11 issue. I am slightly tempted to use the same phrase to December, when I had a pre-release copy of the describe the way that you have dealt with it, because Sutherland report to read in the Department, and you have made much of the “fiction” of Ministers. these three paragraphs just stood out as absolutely But as you now know—I am quite prepared to flawed. I resigned the following day. I would have accept that you did not know this when you resigned even if those three paragraphs had not been started—Ministers had made a correction to the there. On the following Monday, 15 December, the Committee and to Lord Sutherland. I am concerned day before the Sutherland report was released, I was that, although you have raised some serious issues then suspended by the QCA board at the proposal of before the Committee, they might be designed to, if the DCSF permanent secretary. I have contested you like, divert the attention from the very that suspension, or reserved a right to contest it, substantial criticisms of your role in the Sutherland because I believe that I simply should have been put report. on gardening leave, rather than receiving some sort Dr Boston: Let me say this: if there has been an edge of ill-defined suspension. I certainly was astonished to my remarks, there is a reason for it. I gave to be suspended before the report had even come out evidence to Sutherland on 18 September 2008. On 24 and before anyone had even read it. Once that September, which was six days after I gave evidence, suspension had occurred, I received a letter setting three weeks before Balls and Knight gave evidence out what a suspension meant. The letter did not and nearly three months before the Sutherland explicitly state, “You can’t speak to the Select report was published, which was certainly before Committee or write to newspapers,” but that would Sutherland had even put pen to paper, the clearly have been inconsistent with the spirit of the permanent secretary sought to negotiate my early letter, which set out that I was not to contact the resignation on the grounds that the Sutherland QCA or do a host of other things. I simply felt that report would be bad for me. The Ministers had not the proper thing was not to make a media issue or even given their evidence. The permanent secretary anything else out of it, but to come back to the Select said to me on that date that he wanted me out of the Committee in due course when the suspension was job before March 2009. I engaged an employment lifted and deal with the issue in the forum in which lawyer who wrote to the permanent secretary.I made the faulty evidence was used against me. That was it clear that my intention was to remain in post until my decision. the end of the contract and that I would be willing to consider departure only on the basis of the contract Q336 Fiona Mactaggart: So the correction that Jim being paid out. That was, in fact, the end of the Knight made in February to Lord Sutherland and to correspondence—once the permanent secretary had this Committee was not because you raised any replied to Lewis Silkin. The Ministers’ evidence query; it was just done by him.13 came after that—Knight on the 14 October and Dr Boston: I am uncertain about that. The QCA Balls on the 15 October. On the 11 December, as I board had to set up a panel to conduct a review in said, for the first time I became aware that there was light of the Sutherland report, and that panel met on a problem.16 I make no allegations or assertions, 14 January. On that date, I gave the panel, in a but it was very clear that, by the time the two confidential meeting, an account of my Ministers gave their evidence, the skids were under performance, as well as an analysis of it in relation me—or it was proposed to put the skids under me— to the Sutherland report. The QCA panel, and at Department level. That goes back to the meeting subsequently the QCA board, then became aware on 18 March and the whole issue of the future of that I had not been at that meeting.14 The panel then the QCA. contacted the Department to obtain a copy of the note of the meeting, which confirmed that I was not in fact present and had not made a contribution. It Q338 Fiona Mactaggart: So your contention is that seems clear to me that that action, and the QCA this is—I am going to use the word “plot”, but I am board getting on to the case with the DCSF, must not saying that you are saying it is a kind of secret have led to this letter being written.15 As I have said, plot—a plot devised well before the failure of the I have been totally unaware of this letter until today, curriculum tests to get rid of you from your role, with the failure of the curriculum tests being used as 12 See Ev 48 an excuse. 13 See Ev 77 14 Note by Witness: The meeting which took place on 17 June 16 Note by Witness: On the 11 December I read the final pre- 2008 between David Gee and the Rt Hon Jim Knight MP. release Sutherland Report, and for the first time I became 15 See Ev 77 aware that there was a problem. Processed: 17-07-2009 19:14:49 Page Layout: COENEW [O] PPSysB Job: 406183 Unit: PAG3
Children, Schools and Families Committee: Evidence Ev 57
22 April 2009 Dr Ken Boston
Dr Boston: I think that I was seen as a troublesome it. One problem that has emerged from time to time priest, and the failure of the tests was perhaps a is that Ministers have almost become arbiters catalyst seen to oVer an opportunity to allow those between the regulator on one hand and a body events to move forward. contesting something with a regulator on the other hand. I can think of various examples where that has Q339 Chairman: When would your contract have happened with subject associations and other run out anyway? groups. I think that is a problem for Ministers. They Dr Boston: On 30 September 2009. would be far better to stand back and stand well away. Q340 Chairman: David Bell said that he wanted you out by March 2009? Q342 Fiona Mactaggart: But is not that exactly what Dr Boston: Before March 2009. His argument was you are complaining that Jim Knight did? That he that he would prefer not to have me identified with did not keep pushing you and pressing you and so the delivery of the 2009 tests. He was seeking a on? That he actually did what you are saying resignation before or on the date of Sutherland’s Ministers should do? report. Dr Boston: No, I think I have said twice at this meeting that if I had met with Jim Knight once a week for the critical eight weeks of the delivery, the Q341 Fiona Mactaggart: From this Committee’s problem would still have occurred, the failure would point of view, we do not need to deal just with a spat, still have occurred. much of which has been properly dealt with by the Fiona Mactaggart: Absolutely. You said in your Sutherland inquiry. While you might contest some evidence to the inquiry that you were unaware of any things in it, actually much of its account is escalation between 6 and 25 June, which is a universally accepted. critical period. Dr Boston: I agree. Dr Boston: I am not saying that frequent meetings of Fiona Mactaggart: It is accurate, powerful and that sort, or Jim, could have solved the problem at teaches us some lessons that we need to learn. The that point. It was our problem to solve. issue that this Committee needs to focus on is Chairman: I am sorry but we have to move on. Other whether it is possible for Ministers to delegate their people need to ask questions. responsibility for mission critical things. One of the things that you have said about Ofqual is that it is Q343 Derek Twigg: When I was a junior Minister for critical that it has a further distance from Ministers. a short period before I moved to another I shall quote another piece of evidence that Jim Department, we discussed this issue of online Knight gave to Sutherland, when he said, “if you marking and the marking system, which you were looked at it theoretically, you would say that advocating at that point. As you have outlined, there ministers are delegating the scrutiny and challenge were diYculties in moving to that system. I will come function to the Board that we appoint. We have our back to that, as it is one of the key points you have observer on the Board in the form of a Director raised today. I want to get one thing clear. Are you General monitoring and taking part in discussions, accusing Ministers of lying? and then we have regular meetings with the Chair Dr Boston: No, I am simply saying the evidence in who we also appoint—and that is a major decision the report is wrong. for us—and that in many ways ought to be it. But beyond that theory, the reality clearly is that some of these tasks that they are doing are so mission-critical Q344 Derek Twigg: So, on the basis of the meeting that we need alongside that arrangement to be it could be a genuine mistake, but you find it diYcult satisfied for ourselves that things are going well.” to believe because surely a civil servant should have That is a way of saying that these arm’s length told the Minister? bodies, substituting people like yourself, with Dr Boston: I find it diYcult to believe because there is professional, long-term standing in the field of no other meeting it could possibly be confused with. examinations and assessment, for full ministerial responsibility, just don’t work, because people will Q345 Derek Twigg: That a Minister would actually always turn to Ministers. One of your contestations come to a Committee or submit to an inquiry and tell is that they have not done that enough. a blatant lie would be quite amazing, knowing there Dr Boston: I am sorry. They have not done what was no evidence. So it probably was a genuine enough? mistake, on the part of the Minister at least. Fiona Mactaggart: The public have not pointed the Dr Boston: Whatever it was, it was wrong. finger suYciently at Ministers. You are saying that giving Ofqual greater distance from Ministers will Q346 Derek Twigg: Let us get some dates clear: I am assist with this purpose, but Jim’s point and this sure it would not have made a great deal of diVerence discussion show that that distance will always be a to what happened but in order to get it on the record. fiction. The public will always require Ministers to You used a couple of dates. You said you first be accountable. became aware of the problem on 25 June and you Dr Boston: Yes. I think that is true. There are said that people—presumably in the Department— occasions when Ministers would be better to push knew on 26 June. Jim Knight’s letter says that “the back and stand back from issues and let bodies such first time QCA notified Ministers that ETS would as the QCA or Ofqual as the new regulator deal with not deliver test results on time was 30 June”. Which Processed: 17-07-2009 19:14:49 Page Layout: COENEW [E] PPSysB Job: 406183 Unit: PAG3
Ev 58 Children, Schools and Families Committee: Evidence
22 April 2009 Dr Ken Boston is it? On one hand, the Department is saying 30 June Q351 Derek Twigg: When did you have the final and you are saying 26 June. I am not saying it would conversation about this issue, in terms of a decision have made a great deal of diVerence to what by Ministers? You mentioned 2006 as being a key happened but it is useful to have it on the record. part for you, and a time when you could have Dr Boston: I am quite clear that after we obtained resigned. from ETS the formal letter it had to provide us with Dr Boston: I do not have the date of the final under the contract, which is called a duty to warn, decision, but it would have been late summer and which we received on 27 June, the director general certainly before the ETS contract was finally let. I Jon Coles was briefed on that day. say that because one of the things that we wanted ETS to do, if the delivery had been successful and we Q347 Derek Twigg: So it is 27 not 26 June now? You had kept ETS on contract, was the phased said 26 before. introduction—potentially—of onscreen marking Dr Boston: The 26th was the day we wrote to ETS over a period of time, if we could get ministerial and sent an e-mail saying it had to reply to us in agreement for it. It was actually built into the terms of this duty to warn. I understand that there contract that there would not necessarily be manual was a telephone conversation with the Department marking for five years; manual marking could on that day telling it that that had occurred. transform into other forms of delivery. Of course, ETS is a major international player in that field.
Q348 Derek Twigg: The 26th or 27th? Dr Boston: The 26th. Then on 27th there was further Q352 Derek Twigg: I just want to confirm that you contact when we had the letter from ETS. believe that that was the key to stopping the problem occurring? Dr Boston: The key to avoiding this problem in the Q349 Derek Twigg: So you are clear it was the 26th future is exactly what Lord Sutherland has not the 30th? recommended: end-to-end testing of systems of Dr Boston: Yes. onscreen marking.
Q350 Derek Twigg: You have rightly and openly set out that you accept responsibility, but actually you Q353 Annette Brooke: As you have been talking, want to take someone else with you, for whatever Ken, I have been reminded of the Foster report reason that may be. As far as I can tell, based on regarding the Learning and Skills Council, which what you have said, the key point—you will correct found fault with the Department for Innovation, me if this is wrong—is that there were always going Universities and Skills, as I recall, and also identified to be problems with the manual system, although we complacency and so on in the Department, as well as have had it for God knows how many years and we finding major faults with the LSC and the way that have not had this situation occur. However, you it had handled the capital programme and capital believe that moving to an online marking system was bidding. So it seems to me that the point that you the key point in terms of trying to solve this problem have made this morning is absolutely critical, which of uncertainty over delivering these results on time. is that everything follows on from the very limited Am I correct in saying that? terms of reference for the Sutherland inquiry—the Dr Boston: Onscreen marking. very fact that the Government were left out of that Derek Twigg: Onscreen marking, right. You alluded inquiry. I think that your evidence has confirmed to the point that the Ministers were somewhat that. I am not at all clear what these DCSF observers reluctant to adopt onscreen marking, on the basis had to do and I cannot see how they got left out of that it was a risk; 2006 seemed to be the key point, the equation, because I presume that they have been and you said you might have resigned at that point. hearing everything that has been going on. Could If you do not mind me saying so—again, correct me you give me a little more information on the status if I am wrong—there was a slight insinuation that we of the observers? Do they just sit and listen, and not were more bothered about the results not being as go and report anything back? I realise that that good if we went to the new system, and therefore question is a little diYcult for you to answer. None aVecting the league tables and the general the less, it seems to me that the nub of this issue is Government policy about improving education, that you are accepting your bit of the responsibility, than about those technical problems and the risk in but the Government and the oYcials have not moving to the new system. Which was it, in your actually had the questions really asked of them. view? Dr Boston: Lord Sutherland has also put his finger Dr Boston: It certainly was not the case that there on this, in that he is calling for clarification of the was risk in the system; this is proven technology and role of observers. In practice, what happens is that it is used widely throughout the world for school observers are there to represent Ministers and the marking. It is used here extensively, with the general Department. They take part in discussions. There is qualifications. It was not the risk. As far as I was an exchange of views and that is very helpful in the concerned, the clear stumbling block finally was the main. It is good to have that exchange and to have unpredictability of what might happen and the fact that interaction. The problem arises, as I have said, that the results would be diVerent. They might be when they are conveying ministerial impressions and better, or worse, but the one thing that you could be we get to the point where we are actually negotiating certain of was that they would not be the same. the QCA’s advice with them, rather than listening to Processed: 17-07-2009 19:14:49 Page Layout: COENEW [O] PPSysB Job: 406183 Unit: PAG3
Children, Schools and Families Committee: Evidence Ev 59
22 April 2009 Dr Ken Boston what they have to say, challenging it, pushing it hard such as “sexed-up”, because even though you have and getting everything from them we need so that we not said that the Minister lied, just as you did not say can then say, “Well, we will write this advice so that that he should resign, there is almost an implication the Ministers know what we think they need to hear, in the things you have said to the Committee that rather than simply what you think they might want would suggest that. I would like to clear up a couple to know.” It varies so much from committee to of things. Are you saying that DCSF observers committee and from role to role, and I am quite would report to you as well as to the Department? uncertain what the observers do when they go back Dr Boston: No, I do not think that they would report to the Departments. Do they sit down and brief to me at all, or to the QCA. On your first point, I Ministers and write notes for them, or do they brief agree, as I have said to the Committee, that had other sections of the Department, which would 4.137, 4.92 and 4.93 not been in the Sutherland probably be very helpful? In short, there needs to be report, I still would have resigned. There is no clarification of their role. Frankly, although Ofqual question about that. On the other hand, at the end has talked about having observers at various levels of a 45-year career in education, I do not wish to be on its committees, and indeed on its board, my portrayed by a Minister as complacent and as advice would be that it is making a mistake right unengaged—disengaged—when it is absolutely from the start. We do not want organisations that are untrue and unfair and based on absolutely false distant and apart, but a good exchange of evidence. I believe that needs correction and it needs information between them. But we do not want to be correction publicly. too cosy if finally Ofqual has to assert true public independence and accountability to Parliament. Q357 Mr Heppell: I think you are right. I think that you have a right to be annoyed that you never Q354 Annette Brooke: That openness, transparency received a copy of the letter that actually did correct and clarity seem to be really important, and there are that.17 That was not made public at the time, and I lessons to be learned. You have used the terms think that is something that most members of the “sexed-up evidence” and “flawed evidence”, and we Committee would accept, but I would like to move can play around with the dates and with who knew on beyond that if I may. One of the things you said what, but you actually go on to say that it was used to us—and it really worries me—was about walking to portray you as complacent, disengaged and the high wire act and that this is such high risk. I just constantly beleaguered by Ministers. If that is the found it impossible to believe that we need to accept case, it seems to me that the outcome is more serious the idea that, just about a week before the thing was than playing around with the dates on who said what to finish, you had not realised that it was going to fail and when. I would like you to confirm that you feel almost completely. You say, “Well, we were looking that, as a consequence of some misinformation at things, we were spotting the problems and we were being put into the system, you have suVered in the doing things to try to put them right.” But to way you described in your letter. actually say that you had not recognised that this Dr Boston: I find the remarks oVensive because they was going to fail when there was just a week to go— are untrue, but I am a resilient old bugger and at the I just cannot believe that. I mean if that is the case, end of the day it does not matter all that much, but we might as well not have you at all. Part of your role it needs to be corrected. is to identify where there is something wrong and, actually, the idea that the Minister was pestering Q355 Mr Pelling: You answered Graham Stuart by you—I have had these arguments before. People say saying that you felt the Minister should resign. Why to me, “Get in touch with us when you need do you think he should resign? something,” and I say, “Well, I won’t know if I need Dr Boston: Did I say that? something until you tell me when there is something Mr Pelling: That was the impression I got, but wrong.” So the Minister was not going to pester you perhaps you did not. until you knew something. EVectively, what you are Dr Boston: I did not say that. saying is that it was a week before the thing was Chairman: You did not say that in those terms. finished and you still did not know there was a Dr Boston: I do not have a view on that. People make problem. There must be a system where you could be up their own minds, and I made up mine. better informed than that. Presumably Ministers are also, from time to time, Dr Boston: We have gone over this in the Sutherland confronted with diYcult decisions. report and in our submission to the Sutherland report and so on. The management information we were getting was poor and it was conflicting. We had Q356 Mr Heppell: I am a little worried. I think that the situation where on 3 July, when we knew it had one of my colleagues said that they thought your failed on the 25th/26th, the president of ETS letter was intended to divert attention away from America was sending us an e-mail saying, “We are discussion of what actually happened—I am not going to make it. We are going to get there by the sure whether that was its intention, but it certainly 8th,” while his local guy was presenting us with data has diverted attention from what is relevant. I share which clearly showed that it was impossible that that Lord Sutherland’s view that it does not really make was going to be achieved. any diVerence to what the final result would be. I am not quite sure about your use of emotive language 17 See Ev 77 Processed: 17-07-2009 19:14:49 Page Layout: COENEW [E] PPSysB Job: 406183 Unit: PAG3
Ev 60 Children, Schools and Families Committee: Evidence
22 April 2009 Dr Ken Boston
Q358 Mr Heppell: I recognise what you are saying Dr Boston: I would simply say that the key things about conflicting stuV, but surely you should have that we need to look for are to move immediately had a system in place that actually spots that and sharply into onscreen marking and increasingly something is going wrong? into taking the tests by computer, onscreen— Dr Boston: The series of daily meetings we had on working with the tests, delivering the tests—and the detail of all of this, given the evidence that we with the sort of diagnostic feedback that onscreen were getting from ETS, indicated that we were on marking can provide to schools, which is immensely course. You are right to point to what seems—what important in shaping their further programmes and clearly is—a major problem: that right up until the interventions. That is the key thing that we should be end of June, 25 June, as the guy running the place, looking for. the information that I was getting was that we were Chairman: Ken Boston, we have had a good session. going to make it safely. I hope that you feel you have had a fair hearing. I Chairman: We have got to cut you there Ken, hope that you feel you have been asked some because Edward has not had a chance of a question. searching questions. I know that you intend to stay Sorry, John. in the education sector, making a contribution. This Committee would very much value a considered piece of work if you have the time to do it. Your Q359 Mr Timpson: I shall try to hold a positive note. original comment when you took over the job was It has been a very illuminating session and I think that we are still in a cottage industry using quill pens what has come out of it is that from the time that you and ink. We have not in those six years moved very were aware of failures within the QCA delivery, you far from that, as you said in your evidence today. have acted—from what I have heard and read— This Committee would very much like, before this honourably through that process and taken a lot of inquiry is finished and we write it up, some piece that the flak for the failure of many others, not just is not about what happened but about how you yourself. I know that you are also keen to continue would get us from that cottage industry to the system to contribute to the educational system, particularly that you would like. I would also like you to consider where it comes to qualifications and curriculums and that this Committee in its former report said that so on. To that end, what advice have you got for the there was too much testing and assessment, and one Committee on how the procurement process—for small victory we have had is that we do not have the delivery, for the service provider, for 2010 quite as much testing and assessment as we had last onwards, of testing—should take place to ensure summer. Thank you very much for your attendance. that we do not end up in this position again? Dr Boston: Thank you very much, Mr Chairman. Processed: 17-07-2009 19:25:09 Page Layout: COENEW [SO] PPSysB Job: 406183 Unit: PAG4
Children, Schools and Families Committee: Evidence Ev 61
Wednesday 20 May 2009
Members present: Mr Barry Sheerman (Chairman)
Mr John Heppell Mr Graham Stuart Paul Holmes Mr Edward Timpson Fiona Mactaggart Derek Twigg Mr Andy Slaughter
Witnesses: Rt hon Ed Balls MP, Secretary of State for Children, Schools and Families, Rt hon Jim Knight MP, Minister for Schools and Learners, and David Bell, Permanent Secretary, Department for Children, Schools and Families, gave evidence.
Q360 Chairman: I welcome the Secretary of State, my oYce to contact the oYce of Dr Boston to seek, the Schools Minister and the Permanent Secretary to and get, reassurance that marking was on track; on what will be a quite brief session. We have two 2 June at a meeting with Dr Boston; and on 6 June sessions today, both of which are important. when I wrote to a constituent and said that I had Secretary of State, it is customary for you to say personally asked Dr Boston to write to him, which something to open up the proceedings. Dr Boston did on 16 June. It is true that Dr Boston Ed Balls: I will do that. I start by thanking you, Mr did not attend the meeting on 17 June called at the Chairman, and the Committee for allowing us to request of the Schools Minister. Instead, his appear before you today, following our discussion in subordinate, Mr David Gee, did attend and February about the findings of the Sutherland provided similar reassurances to those that I inquiry and in light of Dr Ken Boston’s1 subsequent received directly from Dr Boston. Thirdly, Dr evidence. I wish to say from the outset that I am very Boston expressed concerns that the remit given to grateful for the public service given by Dr Boston Lord Sutherland for his inquiry was too narrow. In during his time as chief executive of the his statement on 22 April, Lord Sutherland said, “I Qualifications and Curriculum Authority. He made believe the terms of reference for my inquiry were a number of important contributions that we should drawn up in a way which gave me ample scope and recognise. I always had good relations with him every opportunity to examine the processes, roles during our time working together. As Lord and responsibilities of all the key players, and that Sutherland’s report made clear, it is also the case was not only QCA and ETS, but Ministers and that, while the primary responsibility for the delivery oYcials at the DCSF. I reject entirely any suggestion failure in last year’s national curriculum tests lay that my activities during the course of the inquiry with the contractor, ETS, there was also a failure on were constrained. I am glad that Ken Boston recognises that my report was fair. I was clear in the the part of the QCA to deliver its remit. Lord report that the prime responsibility lay with ETS, Sutherland was clear in his report that QCA but that there were significant failures in the QCA. I demonstrated insuYcient oversight of the stand by it strongly. I was informed in writing by Jim management and risks associated with the delivery Knight of the error in his evidence. As I said in my of its biggest contract. I wish to address briefly four response to his letter, the error did not materially issues raised by Dr Boston in his evidence. First, as aVect any of the key findings in my report.” Fourthly the Committee knows, the Schools Minister did, in and finally, Dr Boston told the Committee that last error, conflate two meetings with QCA management year’s failures could have been avoided if onscreen in his evidence to Lord Sutherland: a meeting on 17 marking had been used and that Ministers and June at which Dr Boston was not present and a oYcials had blocked its introduction in 2006. That is meeting on 2 July when he was. The error was first emphatically not correct. While it is true that Dr brought to the attention of the Minister on 6 Boston was clearly in favour of onscreen marking, it February after our appearance here. The Schools was the QCA itself that decided on the basis of trials Minister immediately wrote to you, Mr Chairman, that were carried out in 2005 not to proceed. Indeed, to apologise and to Lord Sutherland. The letter and in his statement about the onscreen marking trials the reply from Lord Sutherland were placed in the conducted during the 2005 test cycle, it was Mr Libraries of both Houses on 12 February. There was David Gee who said publicly, “Following analysis, I no attempt in any way to conceal or to mislead and, have concluded that it is inappropriate to introduce importantly, in his reply of 12 February, Lord onscreen marking for either maths at Key Stage 2 or Sutherland stated that that “clarification does not English at Key Stage 3 in 2006.” We have always substantially alter the points that were made” and been clear that last year’s failures let down teachers, “does not aVect my findings”.2 Secondly, Dr Boston parents, pupils and markers. We are determined to states that, contrary to Lord Sutherland’s findings, ensure that that does not happen again. That is why he was not pressed by Ministers on whether marking we have acted to ensure that all Lord Sutherland’s was on track. In fact, as I said in the House last recommendations are implemented in full. We are month, I personally raised the issue with Dr Boston very pleased to have the opportunity to come and on a number of occasions: on 19 May when I asked discuss this again with you today.
1 See also supplementary evidence to this session from Dr Q361 Chairman: Secretary of State, thank you. Can Boston: Ev 73–77 you give us some more background on the 2 See Ev 77–79 relationship that the Department and you had Processed: 17-07-2009 19:25:09 Page Layout: COENEW [E] PPSysB Job: 406183 Unit: PAG4
Ev 62 Children, Schools and Families Committee: Evidence
20 May 2009 Rt hon Ed Balls MP, Rt hon Jim Knight MP and David Bell personally with Ken Boston? Ken Boston was to do. And the new QCDA will be established on the viewed by many people not only as a very competent same non-departmental public body basis as the head of QCA, but someone whose reputation was QCA. As I acknowledged, there were past times broader than that. He was a well-liked public when there were diYcult words between Dr Boston servant. People thought that he had come into the and Secretaries of State over comments that had QCA at a very delicate time in its history. If you sometimes been made by Dr Boston’s senior staV remember, there had been a previous problem with a about aspects of Government policy. But I have to former Secretary of State and in came Ken Boston say that no such occasions arose at all while I was as a really rather refreshing force in QCA. He made a Secretary of State, and there was never a point at lot of friends, so people are asking what went wrong which I had to call, speak to or meet Dr Boston to when a leading public servant who had given great disagree with him over policy or the handling of it. service to education in this country falls out of But, of course, there was a significant management favour and ends up not only being treated in a very failure last summer by the QCA. strange way by being put into purdah for three or four months, but being very angry about his Q363 Chairman: Dr Boston’s view was that there treatment. What went wrong? was a specific meeting where the future of QCA, Ed Balls: In the 18 months when we worked QCDA and Ofqual was discussed where he believed together, I never had a cross word or diYcult that there was a disagreement between you and him. exchange with Dr Boston. From the beginning, we Ed Balls: That meeting was not attended by the worked closely together on issues around standards Schools Minister; it was attended by the Permanent in the run-up to our announcement on Ofqual, issues Secretary and me. I actually think that those who around the Williams maths report, which we were at that meeting will tell you that the commissioned at the time, and then through the conversation between the Chair of the QCA, Dr autumn, in particular, and the establishment of Boston and me was amicable and friendly, and that Ofqual. I never had any diYculties with him at all. I I said that there had not been a moment during my had a great deal of respect for the contribution he time as Secretary of State when there had been a made. It is certainly the case that there were past disagreement. I said that. times before I was Secretary of State when there were David Bell: That is absolutely correct. The some individual clashes over issues, but there was no discussion was about the detail of what was being policy issue over which there was a clash between me proposed, and, as the Secretary of State says, Dr or other Ministers or senior oYcials in our Boston was very clear about the value of the new Department and Dr Boston during those 18 months. independent regulator. He asked questions and It is also true to say that Dr Boston and the QCA sought clarification over the role of the QCDA, but were responsible for managing the clearly it would be wrong to characterise that as an unsuccessful contract with ETS. There were aggressive or confrontational meeting. significant management and oversight errors on that contract—the biggest—which were clearly documented in the Sutherland inquiry, and for Q364 Chairman: Let us move the personalities to one which Ken Boston has taken responsibility. The side and come back to what Ken Boston suggests is decision to suspend him was taken by the QCA a kind of systemic problem. The systemic problem board and not by Ministers, and a final settlement is that he seemed to be getting at was that it may have being reached with Dr Boston. I understand that he looked as though the QCA was independent, but all is not happy, but it very important for me to set the the time there were very powerful departmental record straight. presences at many of the meetings, and that was crucial. So what he is suggesting, in a sense, is that, up front, it might look as though you were allowing Q362 Chairman: But in evidence to this Committee, the QCA a high degree of independence, but actually we got the very strong opinion that it was your the men and women from the Ministry were at every arrival at the Department that sparked oV some of meeting, reporting back and pretty much keeping his discontent. Dr Boston believed that you thought control over what his staV were doing. that he was a sort of turbulent priest who wanted too Ed Balls: I understand that, but I feel no need to put much independence—a more independent role—for the personalities aside, because, as I said, at no point the QCA. He specifically mentioned the discussions did I have a personality disagreement, clash, angry about splitting the QCA into two new bodies and word or diYcult meeting with Ken Boston. whether Ofqual would really be independent. He Secondly, the Bank of England is independent of the gave this Committee the impression that you were Treasury and reports directly to Parliament. Ofqual, very worried about too much independence and that which we are establishing as an independent you did not like him standing up for a more regulator, reports directly to Parliament, independent QCA and, subsequently, a more independent of Ministers. However, the QCA is a independent Ofqual. non-departmental public body. It is not Ed Balls: That does not accord with my impression independent; it reports to Ministers, who are then of the facts, and I do not think it accords with the accountable to Parliament for the national evidence that Dr Boston gave to the Committee, in curriculum and the delivery of tests. Ken Boston was which he clearly said that he supports the not an independent decision maker or commentator establishment of Ofqual as an independent on Government policy. At no point did he seek to do regulator—he thought that that was the right thing so when I was Secretary of State, nor did any of his Processed: 17-07-2009 19:25:09 Page Layout: COENEW [O] PPSysB Job: 406183 Unit: PAG4
Children, Schools and Families Committee: Evidence Ev 63
20 May 2009 Rt hon Ed Balls MP, Rt hon Jim Knight MP and David Bell staV, although there had been issues in the past. online benchmarking. At no point in all my dealings What is clear, though, is that the QCA, on behalf of with Dr Ken Boston did he ever make the case to me Ministers, is responsible at arm’s length for for online marking—in writing or verbally. On 3 managing the national curriculum test process. It November 2005, David Gee, a subordinate to Dr would have been quite wrong for us to attempt to Boston, made a public statement entitled intervene in the detailed management of those “Statement on the use of onscreen marking: the 2006 contracts; instead, Ken Boston and his senior team, NCT cycle”. He said, “As I confirmed at an earlier with management expertise, were paid substantial NCT board meeting in October, the NEA has sums to deliver those tests to that remit. As Lord received the evidence from Pearson responding to Sutherland says, although primary responsibility lay the criteria set out in the reassurance trial. Following with ETS, Dr Boston and his senior staV did not analysis, I have concluded that it is inappropriate to succeed in meeting that remit. We are accountable to introduce onscreen marking for either maths at Key Parliament for that failure, but the failure was that Stage 2 or English at Key Stage 3 in 2006. The of the QCA. That is why Lord Sutherland made the primary driver of this decision was the impact on recommendations that he did. schools, in particular achieving the safe return of the original scripts to schools to time and to budget. I Q365 Chairman: It is refreshing to hear you say that can therefore confirm that we will continue with Ofqual will be as independent as the Bank of conventional marking for all key stage subjects in England. That is good news to the Committee. May 2006.” That was announced by the QCA in 2005. It I take you up on one particular point that you was the QCA’s decision, based on its analysis of the made—a factual one? Rather than it being the evidence. To my knowledge, there was no ministerial decision of the QCA board independently to involvement in that statement, but, as I said, since I suspend Ken Boston, we understand that it did so on became Secretary of State in the summer of 2007, at the recommendation of your Permanent Secretary. no point did Ken Boston raise with me or put to me Ed Balls: You will need to put that question to the the case for moving to online marking. Permanent Secretary and accounting oYcer. I am Derek Twigg: At any point? happy to say firmly on the record that at no point did Ed Balls: At any point. I ever seek, directly or indirectly, to make any Jim Knight: I think that predecessor Ministers had direction to the QCA board. Those were matters for some discussion about onscreen marking. the QCA board to decide. Derek Twigg: I did when I was a Minister. Chairman: That is a diVerent point. I think that the Jim Knight: But there was a very clear statement record will show that you said that the QCA made from David Gee at the QCA that that was its that decision independently. That is the information decision. that we have. Ed Balls: The QCA board? I did say that. David Bell: Obviously, I am not going to divulge all the details of that confidential board meeting. The decision to suspend the chief executive of the QCA Q368 Derek Twigg: Can we move on to the letter lay with the QCA board. issue? In your letter of 9 February to Lord Chairman: Right. Let us leave it there; and we will Sutherland, you stated that the evidence you gave to drill down on it in a moment. Derek, will you open the inquiry, “was absolutely correct in all but one the questions on the Sutherland inquiry? regard”, which was that Ken Boston was not at the meeting, as previously stated, on 17 June.3 Why did Q366 Derek Twigg: Just to be clear, if I heard you you not copy Ken Boston into that correspondence? rightly you said that the decision not to go on with Ed Balls: In retrospect, had we known that Ken onscreen marking was based on the QCA’s Boston was going to appear before the Committee recommendation. and make those points, we would have told the QCA Ed Balls: Yes. to ensure that he was copied in. At that time, the QCA had suspended Dr Ken Boston. We received a Q367 Derek Twigg: We had Ken Boston here; he submission from our staV, following, I believe, our clearly suggested that Ministers did not go ahead appearance before the Select Committee on 4 because it would have changed the results. That February, saying that they had seen the error. They could have been better or worse, which was a risk brought it to our attention on 5 or 6 of February. We that they were not prepared to take. He was very immediately decided that our priority was to tell the clear on that. You say that the organisation of which Select Committee and Lord Sutherland straight he was chief executive made that recommendation. away, and then to ensure that it was in the public Why do you think that he would come here and not domain by placing in both Libraries of the House say that? our letter and Lord Sutherland’s response. In our Ed Balls: It is not for me to comment on Ken view, it was for the QCA management and board to Boston’s evidence or motive. I cannot answer that pass it on to the suspended former chief executive. question. All I can do is tell you the facts. On 27 As I said, in retrospect, had I known, I would have March 2008, Ken Boston wrote to me, informing me told my oYce to ring the QCA and say, “You must of the way in which the QCA was going to proceed make sure that this gets to Ken Boston.” with the marking of the 2008 tests. It proposed online marking capture, online standardisation and 3 See Ev 77 Processed: 17-07-2009 19:25:09 Page Layout: COENEW [E] PPSysB Job: 406183 Unit: PAG4
Ev 64 Children, Schools and Families Committee: Evidence
20 May 2009 Rt hon Ed Balls MP, Rt hon Jim Knight MP and David Bell
Q369 Derek Twigg: The diYculty is that it was Ed Balls: Let me ask David to answer that in a material to him, therefore he was incredulous that he second, but let me say first of all that the first had not been copied in, and it seems strange that he Ministers knew that there was an error was when was not. Ken Boston’s subordinate, who works for him, came Ed Balls: I understand that, but from our point of to a meeting, which Jim conflated with another view the imperative was that Lord Sutherland’s meeting—the first we knew about that was on 6 report was in the public domain. We had responded February. We immediately did what we thought was to his recommendations and given evidence to you, the right thing to do, which was to make sure that and we wanted immediately to ensure that Lord Parliament and the Committee knew about it—and Sutherland and you knew, and that Lord Lord Sutherland, who has said that it does not aVect Sutherland’s view of it was in the public domain. his conclusions or findings. Any suggestion that That is what we did. “In the public domain” meant there was any attempt to mislead or conceal is available to all. In retrospect, I wish that we had also untrue. We immediately made this known to you and contacted the QCA board and said, “Make sure you then to the general public, by placing it in the get this to Dr Ken Boston,” but to be honest, our Commons Library in February. That we tried to assumption was that that was what they would do. conceal is not true and, secondly, the idea that this Our imperative was to make sure that it was made was in any sense a deliberate attempt to mislead public. Do you want to add anything, Jim? Lord Sutherland is untrue. Jim Knight: I think that Ken said in evidence to the Committee that he realised in December that I had made an error, and I regret that he did not tell me Q372 Chairman: Secretary of State, we are not then, and I could have corrected things sooner. saying that—we accept your explanation that it was Equally, I regret that, because we told Sutherland unfortunate that Ken Boston never got the letter. and every member of the Committee, and because it However, set in that context, why was he put into was on the Sutherland website and the QCA knew, suspension for three and a half months and unable we did not directly tell Ken. Naturally, I regret the to talk to anyone? error. In Ken’s evidence to you, it looks like he Ed Balls: That is a diVerent point, entirely thought that I was at the meeting on 18 March that unconnected to this particular issue of the report on was very tense, and I was not. People do make the 17 June meeting. I think you will find that the mistakes when recalling meetings. It is unfortunate, Permanent Secretary is happy to answer any and as soon as you know, you need to put the record questions you want to put to him about the whole straight. That is what I did. process.
Q370 Derek Twigg: In retrospect, if you had your Q373 Chairman: We never got a straight answer to time again, would you have sent him a copy of that the last question, Secretary of State, which is why I letter? asked you. Ed Balls: It is not clear that it was our responsibility David Bell: To be clear, the discussions at that to send it to a suspended member of the QCA board. Certainly, in retrospect we should have ensured, as confidential personnel-related board meeting are not well as coming to the Select Committee and making for public consumption. things clear in Parliament and in public, that the QCA board sent the letter to Ken. Therefore, I am Q374 Chairman: It is not confidential when one of happy to say— the people there gives evidence to the Committee about it. Q371Chairman: We did not get an answer from the David Bell: It is very clear. The QCA had a Permanent Secretary when I asked—not a direct confidential board meeting to discuss a significant answer—if he instructed the QCA to suspend Ken matter. What I said to you was that it was the Boston. Let me give you a parallel. Recently the decision of the QCA board to suspend Dr Boston. chief executive of the Learning and Skills Council Chairman: Listening to your advice. resigned. As you know, there was a problem over David Bell: I contributed to the discussion, but I did capital funding, and he went. Ken Boston’s not say, “You must suspend Dr Boston.” frustration was that you, and certainly the QCA, suspended him for about three and a half to four months, which made him unable to come to this Q375 Chairman: But you are the big boss, you are Committee and give evidence. He was in limbo and the Permanent Secretary. If you picked up the QCA could not give evidence or speak to anyone, because and expressed an opinion, are you telling me that it he was not on gardening leave—he was suspended. is not going to kowtow to you? We contacted him and said, “Will you come to the David Bell: This, in a sense, takes us back to previous Committee?” He said, “I can’t, I’m suspended.” discussions that Dr Boston referred to. I made it very Added to that, when he eventually comes to the clear that the decisions regarding Dr Boston’s Committee, he finds this letter that he had no employment, which include disciplinary matters, knowledge of. So, it does look like a bit of a rest with the QCA. They do not rest with the conspiracy, doesn’t it, when David Bell won’t really Permanent Secretary. That is a matter of fact. Any tell us whether he instructed QCA to suspend this decisions made by the QCA board are made by the man indefinitely? QCA board. Processed: 17-07-2009 19:25:09 Page Layout: COENEW [O] PPSysB Job: 406183 Unit: PAG4
Children, Schools and Families Committee: Evidence Ev 65
20 May 2009 Rt hon Ed Balls MP, Rt hon Jim Knight MP and David Bell
Q376 Chairman: In any other part of the world, in marking, because obviously we were concerned with any other business, in any other sector, if you were some of the feedback from the marking world. We sitting there making a decision about someone’s were told in terms by Dr Boston that any issues future, a top member of your personnel, and the big around training had been sorted out and that boss was there—come on, Permanent Secretary, we because ETS was doing things in a diVerent way,that weren’t born yesterday—if you were sitting there was bound to lead to some noise in the system, but and you articulated a view, are you telling me that things were under control. The Schools Minister they wouldn’t have said, “Yes, sir. That is the became more concerned in the next week or so by Permanent Secretary, you are suspended for three reports and therefore asked for a meeting with the and a half months.” QCA, but it was David Gee who attended that David Bell: No, it would have been quite meeting and gave the same reassurances. During inappropriate for the QCA to have made decisions that period Ministers regularly pressed for that were not its own. The QCA board had to make a reassurances and received them. It was not until 30 decision regarding suspension. In fact, I made it very June that the QCA position suddenly and clear in earlier correspondence that any decisions substantially changed, and it was on that basis that regarding disciplinary matters were for the QCA we moved into a diVerent phase. board, and the QCA board alone.
Q377 Chairman: But Ken Boston had oVered his Q380 Derek Twigg: So you stand by your statement resignation. Why didn’t you just accept it? that Ken Boston managed the test delivery process David Bell: It was not for me to accept. at arm’s length from the NAA? Chairman: But you were there. Ed Balls: Is that a statement from me or from Ken David Bell: It was the QCA board’s decision not to Boston? accept Dr Boston’s resignation, not mine. I had no Derek Twigg: You said it in your statement on 16 locus in that matter. December. Ed Balls: It is certainly the case that one of Lord Q378 Chairman: And you didn’t think it strange that Sutherland’s criticisms of the QCA management is taxpayers’ money was going to be paying a man for that they treated the NAA as a separate body, when nearly four months for doing nothing. in fact it was part of the management structure of the David Bell: Well, we have proper and well-accepted QCA. Therefore one of Lord Sutherland’s criticisms procedures that when you suspend individuals you is that Ken Boston did treat the NAA as more arm’s suspend them on full pay, pending the outcome of a length than was in fact the case. But I have to say disciplinary investigation. That is what happened that from my point of view Ken Boston regularly with Dr Boston. The QCA suspended him on full gave me reassurance, so, far from being at arm’s pay and then carried out an investigation into his length, he was telling me on regular occasions that conduct. That is entirely appropriate and you would things were on track. find that happening in almost all walks of life. Jim Knight: Let me just add, in terms of the sequence Chairman: Derek, sorry, I cut across you. of things from 30 June through to 2 July, which was the meeting that I got confused about and thought Q379 Derek Twigg: A key part of what Dr Boston was on 17 June—in terms of describing the character said, in terms, was that basically the evidence against of the meeting—on 30 June there were some phone him in the report was sexed up. He specifically said calls with Jon Coles, the then acting and now full that your statement, Secretary of State, was Director General for Schools, when they said there complacent and disengaged, and that he was were some problems with their projections that Key constantly beleaguered by Ministers with questions Stage 3 would be marked on time, and perhaps some he was unable to answer. I think he said that you problems with Key Stage 2 English. There are some stated that he, Ken Boston, managed the test e-mails that show that through the course of 30 June delivery process at arm’s length from the National that situation looked as if it was getting more Assessment Agency. What is your response to those accusations, in terms of you misquoting him, not serious. That is when I asked for the meeting with putting the correct evidence and trying to blame him David and Ken, which Ken said to you he for the whole thing? remembers very well and is beyond the terms of Ed Balls: The remit for delivering the national reference of Sutherland. I do not think it is at all curriculum tests was a remit from the QCA, of which beyond the terms of reference of Sutherland, because Ken Boston was the chief executive, and it was at that meeting, when, incidentally—looking managed within the QCA by David Gee, the through the notes that my private secretary took of managing director of the NAA. I think one of Lord it—David Gee contributed 24 times and Ken just Sutherland’s points is that ambiguity in the twice, we went through in a lot of detail exactly when oversight of Ken Boston of the work of David Gee the marking might be done. At that point, none of was an issue for the QCA. On 19 May these issues the technical problems that subsequently came to were raised in the House of Commons and I light, which were causing so much diYculty and contacted Dr Ken Boston’s oYce to seek delay in the delivery of the results, were coming to reassurance that the marking was on track, and light; so from 2 July onwards we still had a lot of received that reassurance.I had a meeting with Dr work to do in understanding the problem. It was in Boston on 2 June at which the first agenda item was that set of conversations that Ken was distant and Processed: 17-07-2009 19:25:09 Page Layout: COENEW [E] PPSysB Job: 406183 Unit: PAG4
Ev 66 Children, Schools and Families Committee: Evidence
20 May 2009 Rt hon Ed Balls MP, Rt hon Jim Knight MP and David Bell delegated everything to David. That informs the Lord Sutherland, who wanted to restrict nature of my comments that Ken took exception to publication, so the full board had not seen the full when I gave evidence to Lord Sutherland. report. Ed Balls: It was Jim’s comments. Q389 Mr Stuart: Thank you. So you had spoken Q381 Mr Stuart: Can I momentarily take Mr Bell extensively to the chairman. He decided to hold the back to the QCA board meeting on Monday 15 meeting on the Monday. It was a teleconference that December. Was it held at your behest? you attended. The other members of the board had David Bell: It was initiated by the chairman of the not read the report. It was decided to suspend—a QCA. rather unusual process—rather than accept the resignation of the chief executive. Q382 Mr Stuart: And was it held at your behest? David Bell: That was the decision of the board. David Bell: No. The chairman of the QCA called that meeting. Q390 Mr Stuart: That was of course a decision of the board on which they did not have the full facts—you Q383 Mr Stuart: You didn’t ask him to do so—you did and you attended. made no contact with him to suggest that should David Bell: That was the decision of the board. happen? David Bell: No. He decided to hold that meeting. Q391 Mr Stuart: Thank you. I think that makes the position extremely clear. Can I ask the Secretary of Q384 Mr Stuart: That was obviously his decision to State why you intervened when Ofqual was going to make, but he could have been advised by you. conduct the inquiry? Instead you intervened and David Bell: He talked to me quite considerably said that Lord Sutherland should report to you and about the run-up to the publication of Sutherland. specified the remit. I think there are two key points; Mr Stuart: So you did speak to him. there are nine bullet points, but eVectively you said David Bell: Of course I spoke to him. that Lord Sutherland should report to you on how Mr Stuart: Okay. So you spoke to him before the the QCA had discharged its remit from the DCSF meeting, and then the meeting was held, and as he is and how the QCA had delivered against its formal the chairman— success measures in relation to Key Stage 2. Apart David Bell: I had to speak to him on a number of from a reference to the appropriateness of the occasions in the run-up to the Sutherland DCSF’s arrangements to monitor, basically Lord publication. Sutherland was asked by you to report to you, not to Ofqual, and to do so on the basis of what the QCA Q385 Mr Stuart: This of course goes to the heart of had done with ETS rather than looking at the some of the issues here—that your Department is all broader picture. Is that fair? over the QCA, yet the QCA has the responsibility. Ed Balls: No. Was that meeting held when everybody was present, Jim Knight: No. or was it a teleconference? Chairman: Sorry, who is responding? David Bell: It was a teleconference. Ed Balls: I made very clear in my opening statement the view of Lord Sutherland, which is that he felt his Q386 Mr Stuart: And you attended? remit was very wide and covered the role of the David Bell: I attended along with one of my QCA, but also Ministers and DCSF oYcials. He directors general. says he rejects entirely any suggestion that his remit was in any way narrowed. The QCA does not report Q387 Mr Stuart: Had the rest of the board had the to Ofqual, it reports to me. I asked Lord Sutherland chance to read the Sutherland inquiry at that point? to look into aspects of this, but Ofqual also has David Bell: This is quite important: the QCA board responsibilities for national curriculum tests and meeting is a private one—it is a confidential standards. Therefore, in those parts of its remit, the meeting— Sutherland inquiry reported to Ofqual, and that was Mr Stuart: Had they had the chance to read the entirely appropriate, open, public and fully to the report? satisfaction of Lord Sutherland. David Bell: If I may say to you, Mr Chairman, I received minutes of the meeting that the QCA said to Q392 Mr Stuart: Ken Boston, as you know, gave me were legally confidential. evidence to us that the remit was constrained. If you read the remit, I think it is quite clear. I am going to Q388 Mr Stuart: Had they received copies of the move on, if I may. report by 15 December? It is nothing to do with Ed Balls: Mr Chairman? confidential minutes, Permanent Secretary. Mr Stuart: I am asking the questions. You answer Chairman: Graham, let us get some ground rules. Let them. I haven’t given you one, Secretary of State. I the witness give his reply, then you can come in will come to the next question in a moment. again. Ed Balls: I don’t think this is the way to behave, Mr David Bell: As a matter of fact, the QCA board did Chairman. not see an advance copy of the Sutherland report, Chairman: Let the witness come back, Graham. I although a couple of members of the board did. That give everyone a fair hearing on this Committee. was again in accordance with the instructions of Secretary of State. Processed: 17-07-2009 19:25:09 Page Layout: COENEW [O] PPSysB Job: 406183 Unit: PAG4
Children, Schools and Families Committee: Evidence Ev 67
20 May 2009 Rt hon Ed Balls MP, Rt hon Jim Knight MP and David Bell
Ed Balls: It is not fair or correct to say that the remit David Bell: This is about— was narrowed. Lord Sutherland in his statement on Mr Stuart: It is a simple question: why was an oYcial 22 April says, “I reject entirely any suggestion that from the Department seconded to the independent my activities during the course of the inquiry were Sutherland inquiry? constrained.” He also says he believes the terms of David Bell: There is a very straightforward answer to reference were drawn up in a way that “gave me that. It is very frequently the case that when ample scope and every opportunity to examine the independent inquiries are set up, the Departments processes involved and the responsibilities of all of that have commissioned them draft over oYcials to the key players, not only QCA and ETS but also carry out such work to support inquiries. That is Ministers and oYcials at the DCSF.” Those are not absolutely straightforward and happens not just in my words, they are the words of Lord Sutherland our Department, but right across Government. who stands fully by his findings. Q395 Mr Stuart: Right. So when you have an Q393 Mr Stuart: That is clear, Secretary of State. independent inquiry looking into, according to the The point I made was that Ken Boston, who was the Secretary of State, an apparently wide remit head of the QCA, doesn’t agree with that including the behaviour of the Department, it is assessment. appropriate for someone who works in the private Ed Balls: Dr Boston doesn’t agree with that oYce of the Permanent Secretary to be seconded to assessment but Lord Sutherland does agree—that is that independent inquiry to join the secretariat and his assessment. be part of the drafting team. Is that what you are Mr Stuart: You don’t need to say so for a third time, telling us? It sounds like an episode of “The Thick of Secretary of State. We have limited time and I would It”. You guys would make excellent characters in like to ask another question, if I may. that show. Jim Knight: I would like to say in response to you, David Bell: As I have already written to you this Chairman— morning, Mr Stuart, I pointed out to you that I Mr Stuart: Who is dictating this, the questioners or explained to Lord Sutherland that I thought we had the witnesses? an exceptionally able member of staV who was able Chairman: I am, as Chairman. I am going to give a to take part in this— fair crack of the whip to everyone. The Schools Mr Stuart: I bet you did. Minister wants to come in and then you are back in David Bell: Mr Chairman, I find there is a question the questioning. of my integrity here. The suggestion is that somehow Jim Knight: If I could add, in response to a question I put to Lord Sutherland a member of staV who from you, Chairman, Ken said, “I think he did a very would not— fair job and I have said on the public record several Mr Stuart: From your private oYce. times I think it is a very good report.” Ken thinks it David Bell: From my private oYce, who would not is a good report and it was a fair job. The person who be appropriate. The decision to take a member of wrote it says that the remit was wide enough for him staV from my private oYce, given the expertise that to cover the DCSF. He makes a few small criticisms they could bring, was Lord Sutherland’s. As I also of the DCSF along the way. I think the remit was pointed out to you in my letter this morning, Mr perfectly wide. Stuart, the final decisions—the findings and recommendations of an independent inquiry—rest Q394 Mr Stuart: Thank you. Ken Boston says that with the person who carries out the inquiry, not a it is a good report, given the limited remit that Lord junior member of staV who joins the inquiry Sutherland was given. But Lord Sutherland takes a secretariat. diVerent view, and it was his report, so I think you have that on the record. But this goes to the heart of Q396 Mr Stuart: That would be just as the QCA the issue—about independence, which often runs board was responsible for the decisions which you like a thread through so much of the work of the helped to lead it to come to? The next question, if I Department. Why was an oYcial from the may: could you confirm that the member of staV Department seconded to the Sutherland team? from your private oYce seconded over to this Ed Balls: Just to be clear, Ofqual is independent of inquiry played a part in drafting sections of the the Department. It reports directly to Parliament. report? The QCA is a non-departmental public body, as the David Bell: I don’t know that, because it is Lord QCDA will be. They are not independent—they Sutherland’s report. You would have to ask Lord report to Ministers who then report to Parliament— Sutherland that question. The final report is Lord but we set them an annual remit in which they Sutherland’s, not a member of the secretariat’s. manage certain aspects of their work at arm’s length. In this case it is the delivery of the national curriculum tests. Q397 Mr Stuart: I spoke to Lord Sutherland Mr Stuart: Could you answer my question? yesterday. He is a man of the utmost integrity. He Ed Balls: It is just as important to correct mistakes. told me that he took full and total responsibility for The issue of independence does not arise in the case the report, that everyone who worked for him of the QCA. It is not an independent body. produced the report to his requirements and that he Mr Stuart: I would quite like someone to answer my had no doubts whatsoever about the eYcacy and question. professionalism of his team. But I put it to you, Processed: 17-07-2009 19:25:09 Page Layout: COENEW [E] PPSysB Job: 406183 Unit: PAG4
Ev 68 Children, Schools and Families Committee: Evidence
20 May 2009 Rt hon Ed Balls MP, Rt hon Jim Knight MP and David Bell
Permanent Secretary, that sending someone from breach that confidentiality. It was entirely your private oYce to join the drafting team of a appropriate for me, as the principal accounting supposedly independent inquiry is not appropriate. oYcer, to talk to Dr Boston about his position, and Mr Heppell: Would that not be the normal process? yes, I did ask him to consider his position. However, Chairman: John. What’s the problem? when Dr Boston gave testimony to your Committee Mr Heppell: It just seems to me that in any inquiry he said: “The Permanent Secretary spoke to me and people are going to ask for people who have some I made it clear that I would be willing to consider knowledge to assist them. Is it not normal practice departure only on the basis of the contract being for somebody to be seconded? paid out.” In actual fact, that is not quite the case, David Bell: That often happens and I can cite a because when Dr Boston’s lawyers wrote to me, they number of examples when people who have been said: “Dr Boston should receive a payment working in the policy area go over to independent equivalent to 10% of his salary.”His lawyers went on inquiries. This is not just something that happens in to say— the DSCF, it happens regularly across Government. Mr Stuart: Chairman, that doesn’t answer my If you appoint independent inquirers to carry out question, and I’d rather he answered the question I reports, they need secretarial support. It is the put, rather than the one he would like me to have responsibility of the person carrying out the inquiry. put. You have just quoted Lord Sutherland, according to Chairman: Hang on. Graham, if he doesn’t answer your conversation with him, Mr Stuart, saying that the question, you will get another bite, but let him the members of staV concerned behaved with the answer first. utmost integrity, and the report and its findings were David Bell: You asked me why I had that Lord Sutherland’s alone. I am therefore struggling conversation, and I am referring to Dr Boston’s to understand what the problem is if that is the testimony. Dr Boston’s solicitors said on his behalf: decision Lord Sutherland has come to about the “All success measures appended to the remit letter support he received. from the Secretary of State have been met, other than in relation to delays over the Key Stage tests.” Q398 Mr Stuart: That may be a problem in itself. I So when Dr Boston said he only asked for his think any outsider would find it odd that someone contractual entitlement, that is not the case: he asked from the Permanent Secretary’s private oYce could for additional payment if he was even to consider be deputed. Is it possible that she could have written early retirement. draft sections of the report? David Bell: I don’t know. I had no contact with the individual concerned during the process. Q401 Mr Stuart: Why did you say that the inquiry Ed Balls: Does she work in your private oYce now? findings were going to be bad for him? David Bell: She does not work in my private oYce David Bell: You didn’t have to be a prophet to work now and she was physically out of my private oYce out that the inquiry was going to be bad. The other during the time this inquiry was carried out. thing I should say is that as the crisis unfolded during July and into early August it became apparent to me Q399 Mr Stuart: But she did return to work for your that Dr Boston did not have a grip of the detail of Department after her secondment to Sutherland? the issues. Therefore, I surmised that the inquiry was David Bell: Of course she did. going to be very diYcult for the QCA and for Dr Mr Stuart: So she worked for you beforehand and Boston. I had no contact with anyone in the inquiry she works for you again now. and I did not know what it was going to say. David Bell: She works for the Department, so yes, she works for me ultimately. Q402 Mr Stuart: You did have contact. You said you had contact with the person from your private oYce Q400 Mr Stuart: Permanent Secretary, I am sorry who had joined the inquiry team—only on for appearing cynical, but we are trying to get to the discussing administrative matters relating to the truth here. What we have is a person from your inquiry, but— private oYce who was apparently in charge of the David Bell: Precisely. recording equipment at the hearings and may have been part of the drafting team, and then, days after Ken Boston gave his evidence, weeks before Q403 Mr Stuart: But even after the appointment of Ministers gave evidence to the Sutherland inquiry, that person you were in contact with them on and months before it was published, you met with administrative matters. That person worked for Ken Boston and suggested—according to his your private oYce beforehand and was returning to testimony—that he should resign because the your Department afterwards, and you were in findings were not going to be good for him. Can you contact with them during the inquiry. No wonder explain that meeting and why you took that you were so confident. extraordinary step? David Bell: I will give you the administrative contact David Bell: Just before I answer that question, I we had. She said, on Lord Sutherland’s behalf, “Can think I should say that it was Dr Boston’s lawyers we set up an interview to take your evidence?” My who asked me to keep private and confidential all the reply was, “Yes, you can, and we will do it on this discussions I had with Dr Boston in September and date.” That is the administrative contact we had October. Dr Boston chose in this Committee to during the process. Processed: 17-07-2009 19:25:09 Page Layout: COENEW [O] PPSysB Job: 406183 Unit: PAG4
Children, Schools and Families Committee: Evidence Ev 69
20 May 2009 Rt hon Ed Balls MP, Rt hon Jim Knight MP and David Bell
Ed Balls: May I add briefly to that point? David Q407 Mr Heppell: Do you think, with hindsight—I wrote to the Cabinet Secretary on 22 April 2009 and know that such things are very useful—that it was a made it quite clear that at the time of the discussion good idea to be seeking resignation from Dr Boston he had in September with Dr Boston, when the before Lord Sutherland had reported and before the discussion with lawyers and the contractual Secretary of State and the Minister had even given payments arose, there had been no prior discussion any evidence at the inquiry? I know that you have at all with Ministers. That was a discussion that partly answered this. David had with Ken in his role as accounting oYcer. David Bell: Yes, I think it was appropriate. The The second thing I would like to say, and this echoes principal accounting oYcer has to take overall what Lord Sutherland has said on the record and to responsibility for relationships with other bodies Mr Stuart, is that in my view our Permanent and other accounting oYcers, and given that I had Secretary and our civil servants have at all times seen Dr Boston close up during the crisis and was operated with the highest standards of integrity and very concerned about the way he was handling it, I independence, and the insinuation that somehow think you would have been astonished if I had had there was some kind of complicit co-operation in no conversation with him about his future. I saw that trying to change the direction of the Sutherland and felt that a conversation would be appropriate in inquiry’s findings is absolutely untrue and the circumstances. Don’t forget that Dr Boston was unfounded in my experience. I think that in this scheduled under his original contract to go in Permanent Secretary we have somebody with the September 2009 anyway, so there was a conversation very highest standards of integrity. It is really that would be had in the autumn. I decided to initiate important for me to put that fact on the record, on that conversation and say that I thought that in the behalf of the two Ministers here. circumstances he should consider his position, but I was only ever prepared to accept that he would have six months’ notice. Q404 Mr Heppell: Following on from that, and to get this absolutely clear, on whose authority did you approach Ken Boston in September 2008? Q408 Mr Heppell: Just one final thing about David Bell: My personal authority. something Graham said earlier. Would the minutes of the QCA meeting that suspended Mr Boston not be subject to freedom of information? Q405 Mr Heppell: And when was the decision taken? David Bell: I think you will find that if it was a When did you make a decision to seek his confidential personnel board meeting they would retirement? not be, because you can utilise the arrangements David Bell: I first met Dr Boston to discuss that on regarding personnel and confidential details in the 24 September and that led to exchanges between me Freedom of Information Act. and the lawyers. I went back to the lawyers and quite explicitly said that it was not my responsibility to Q409 Mr Timpson: Secretary of State, you have used initiate any action against Dr Boston. What I was today as an opportunity to clarify a number of saying to Dr Boston, as principal accounting oYcer matters by referring to correspondence between to accounting oYcer, was, “I think you should your Department and Ken Boston. consider your position because it has been a very Ed Balls: My statement was clarifying diYcult year.” So I initiated that. correspondence between the Schools Minister, the Committee and Lord Sutherland. The Q406 Mr Heppell: So that was because of the correspondence that the Permanent Secretary impending report? Was it anything to do with the referred to is correspondence that we were not meeting on 18 March? That was what Ken Boston involved in and I have not referred to it in my said in evidence. He said he believed that Ministers evidence. had decided to put the skids under him—his words—following the meeting on 18 March. Q410 Mr Timpson: But it is coming from your David Bell: At no time did the Secretary of State, the Department and it is being co-ordinated with Ken Schools Minister or any other Minister say to me, Boston’s involvement in his role in the QCA. “You have to have a conversation with Ken Boston, Ed Balls: I have never seen any of the letters to which telling him to go.” They never said that to me. I had the Permanent Secretary has referred between the discussion as the principal accounting oYcer himself, lawyers and Ken Boston. I did not know of saying to the accounting oYcer responsible for the their existence. QCA, “You should consider your position.” Dr Mr Timpson: I am not suggesting you have. All I am Boston implied to you that he was only prepared to saying is that you have read that correspondence. consider his position if his contractual entitlements Ed Balls: No, I have not seen that correspondence. were met. Actually, Dr Boston was seeking a bonus and some other additional payments before he was Q411 Mr Timpson: Perhaps if I put the question to prepared to leave. I wrote back and said that it would you rather than anticipating where this might go, it be completely unacceptable to pay anything other might save us a bit of time. One of the periods that than his contractual entitlement. So Dr Boston was has been in dispute—the confusion about who was seeking more than his contractual entitlement if he at what meeting and what happened during that was going to have a negotiation with the QCA board period—is between 19 May when questions were about his future. asked in the House about the delivery of the tests Processed: 17-07-2009 19:25:09 Page Layout: COENEW [E] PPSysB Job: 406183 Unit: PAG4
Ev 70 Children, Schools and Families Committee: Evidence
20 May 2009 Rt hon Ed Balls MP, Rt hon Jim Knight MP and David Bell marking and 2 July when, as I think it was said to do is pre-empt the proper processes around the earlier, it seemed to move on to phase two: it was release of information. These are probably questions heightened. In order to help the Committee have that you should put to the Permanent Secretary for complete clarification and confidence in the answers an FOI point of view or that you should ask Lord that you have given, and in the spirit of transparency, Sutherland. do you not agree that it would be helpful to provide to the Committee the correspondence between yourself, the Schools Minister, your Department Q414 Mr Timpson: Well, can I put on the record a and Ken Boston, so that we can be absolutely clear polite request for that information? If I have to go as to what happened during that period?4 through oYcial channels, can I be given the Ed Balls: All the correspondence was submitted to information by which to do that? Lord Sutherland. In a sense, we have been through Ed Balls: It is not for me to make decisions about the this. Lord Sutherland has seen all this evidence in release of those kinds of papers. But we did say to compiling his report. Lord Sutherland that he had to have access to Mr Timpson: But this Committee hasn’t seen it. everything in order to do his inquiry, and he did. Ed Balls: The issues around the release of Mr Timpson: So that is a no. information that was put together for the Ed Balls: It is the same answer as I gave you a Sutherland inquiry are matters for Lord Sutherland, moment ago. It is not for me to make decisions and then the normal processes for release. That is not about the release of information in that way. But as something that we, ministerially, would be involved I said, I gave you a very clear answer. From my point in. In terms of the particular issues raised, on 19 May of view, full transparency is absolutely fine and the this was raised in oral questions. My oYce spoke to right thing. Dr Ken Boston’s oYce to ask for and receive reassurances. I then had a meeting with Ken Boston Q415 Chairman: Lord Sutherland has returned all on 2 June. At that meeting I also raised a the documents supplied by the Department, as I constituent’s issue. I wrote to my constituent on 6 understand it. June. Ken Boston wrote on 16 June. On 17 June the David Bell: Yes, he has. But to reinforce the Schools Minister had a meeting with David Gee. On Secretary of State’s point, there are proper 2 July I had a meeting with Ken Boston and David procedures regarding policy papers and other papers Gee. All of that is clear and in the public domain, that go through and flow through a government and all the papers that underpin that would have department. If Ministers are approached, either by been provided to the inquiry. It is not for me to pre- parliamentary questions or via freedom of empt processes around either the release of the information requests, oYcials have to consider those papers of the Sutherland inquiry or freedom of questions. They have to be considered under the information. There are proper processes for that. Secretary of State and by the proper procedures. We But those are not processes in which Ministers would don’t just put out all the papers on every policy area, ever be involved. because even under the Freedom of Information Act that would be considered inappropriate. But the key Q412 Mr Timpson: Can I go back to the original point is that Lord Sutherland had access to all the question—do you agree that it would be helpful to papers that he needed and he drew his conclusions the Committee to have access to all that based on all of those papers. That is the most correspondence in order to clear up the confusion definitive statement of what happened: Lord that has run on for months? Sutherland, an independent inquirer, published an Ed Balls: I did answer that question, by saying that independent report. it is not appropriate for me as Secretary of State to Jim Knight: Incidentally, Edward, the Permanent comment on the release of information from an Secretary was copied into the letter to Michael Gove independent inquiry or through freedom of from Lord Sutherland, where he raised similar sorts information. of questions. It is a very thorough response from Mr Timpson: I am not asking for it from Lord Lord Sutherland which details the places in his Sutherland. I am asking for it from you. report that Michael could go to in order to see the Ed Balls: But all of the papers were provided to the sequence of events, and how all of those questions Sutherland inquiry. were properly answered in his inquiry. I would imagine that Michael would be very happy to share that. Q413 Mr Timpson: But we have not seen them, and that is the point I am making. Is there any reason why we cannot see them? Q416 Mr Timpson: As we are talking about Ed Balls: The answer I am giving you is that it is not transparency, and independence seems to have been for Ministers to decide on the release of those papers. the theme of this session, can I ask one further It is a matter for the Sutherland inquiry and the question about Ofqual? Can you give assurance that freedom of information processes. As far as I am the full independence of Ofqual—you have referred concerned, personally, there is absolutely nothing to to it as having the same independence as the Bank of hide and everything to gain from full transparency. I England—will mean that there won’t be any DCSF am completely relaxed about it. What I do not want observers appointed to Ofqual? Ed Balls: I think that in the end that is a matter for 4 Correspondence published on the QCA website. the Ofqual board. Processed: 17-07-2009 19:25:09 Page Layout: COENEW [O] PPSysB Job: 406183 Unit: PAG4
Children, Schools and Families Committee: Evidence Ev 71
20 May 2009 Rt hon Ed Balls MP, Rt hon Jim Knight MP and David Bell
Q417 Mr Timpson: What is your view? me for onscreen marking. It was the QCA’s decision Ed Balls: I am very happy to have the same not to proceed, including in the case of the ETS arrangements as the Bank of England. contract in 2008.
Q418 Mr Timpson: So, your preferred view is for Q420 Derek Twigg: So you are saying that Dr Boston and the QCA have completely diVerent there not to be DCSF observers on Ofqual? views about this? Ed Balls: No, because the Bank of England has an Jim Knight: In his statement, David Gee said that the observer from the Treasury. primary driver of this decision was the impact on Jim Knight: We have got to be very clear, during the schools, and in particular how to achieve the safe course of the Bill that is going through Parliament, return of the original scripts to schools to both time that whether Ofqual invites observers or not is a and budget. That was the principal reason why the decision for its board. NAA—which is a division, but as Sutherland says, Mr Timpson: I understand that, I was just using this probably too separate a division, within QCA— opportunity to find out your views. made that decision. As I understand, when the Ed Balls: The Bank of England chooses to have a contract was procured for ETS it was procured on Treasury observer at the monthly meeting of the the basis that it would start with conventional Monetary Policy Committee, but I don’t think that marking, although it would use all the online things anyone has ever suggested that that in any way that Ed referred to earlier. That was to move it on compromises the independence of the Bank of from the quill pen, cottage industry that Ken likes to England when it comes to making decisions on describe it as, but some of the problems during the interest rates. summer were caused by how those online things Jim Knight: The reason why a board would operated. The contract then left it open to move to sometimes want to make that decision is that onscreen marking over the course of that contract. Ministers still make policy decisions and there are People recognise that that works perfectly well with times when it might be helpful to a board to other public examinations, but there are issues about understand the background to how those policies the return of scripts to schools, because schools were arrived at. If it has someone present that it can really value that. That was a part of the problems ask those questions of, that can be helpful to it, but that we had over the summer last year. in the end, that is the board’s judgment. Chairman: We are running out of time. Paul Holmes. Ed Balls: That is absolutely how it works in the case of the observer on the Monetary Policy Committee, Q421 Paul Holmes: You talked about observers on who will update regularly on aspects of fiscal policy. the Bank of England Monetary Policy Committee Chairman: Thank you for that, witnesses. A quick and mentioned whether they might be required on one from Derek. Ofqual. You had observers on the QCA. What was their role exactly? Ed Balls: The point that Lord Sutherland makes as Q419 Derek Twigg: I want to return to the issue of a recommendation for the DCSF is that we should onscreen marking because you made a very be clear about the role of those observers. We must unequivocal statement. I reread the evidence that clarify that role, and that is what we have done in Ken Boston gave during the hearing, in answer to discussion with the QCDA. The role of the observers my question. He said, “It certainly was not the case is to ensure a proper flow of information and an that there was risk in the system; this is proven understanding of the decision makers at the point of technology and it is used widely throughout the decision. It is not to be involved in or seek to world for school marking. It is used here extensively, influence those decisions, or to take away with the general qualifications. It was not a risk.” accountability for those decisions from the decision That is completely in contradiction to what you have maker, which in this case is the QCA board. It would said. I cannot believe that you have come here and be entirely inappropriate for a DCSF observer to said something that is not true, in any way, but I start trying to influence the final decision, but it wondered whether you have evidence from the QCA would be appropriate for them to ensure that the saying what it recommends. I find it bizarre that you QCA board knew the full facts in advance of that came here and made a statement like that to this decision. If the observer felt that there were Committee. concerns, they could relay them back to the Ed Balls: In my opening statement, I said that while Department. It is about information flow. Dr Boston was clearly in favour of onscreen marking, which I understand from reading the Q422 Paul Holmes: So were they there to clarify to testimony to the Committee, the QCA itself decided, the QCA what was happening in the Department, or on the basis of trials that were carried out in 2005, were they there as an early warning to tell you that not to proceed further. That was the QCA decision. there were problems coming up regarding the It may well be the case that Dr Ken Boston has, at marking of the tests? It is clear that they did not do some point, in writing, or to oYcials in the that. Department, made the case for onscreen marking Ed Balls: One of the problems, which the Sutherland since I have been Secretary of State. But what I said inquiry makes very clear, is that the QCA board did was that at no point, to my knowledge, verbally or not have a discussion that raised those kinds of in writing, did Dr Ken Boston ever make the case to concerns and risks. It was not the case that we had Processed: 17-07-2009 19:25:09 Page Layout: COENEW [E] PPSysB Job: 406183 Unit: PAG4
Ev 72 Children, Schools and Families Committee: Evidence
20 May 2009 Rt hon Ed Balls MP, Rt hon Jim Knight MP and David Bell observers who observed discussions but did not pass Therefore, it would have been very diYcult for an the information on. Those discussions were not observer on the board to report back something that really occurring, which is part of the problem about was not being reported to the board. the management of those risks. If the observers saw something that concerned them at the point of Q425 Mr Stuart: There would be issues raised about decision, of course I would have expected them to the Bank of England’s independence if the Governor tell me. of the Bank of England announced that, in his opinion, observers from Government were trying to Q423 Paul Holmes: What level are these observers? shape the advice given to Ministers by the Bank of Are they level three, five, seven? England. That is exactly the allegation being made Ed Balls: They are sort of middle-ranking, by Ken Boston in this context. responsible oYcials. David Bell: That is not true. If you think of the David Bell: We usually have a director general or a people on the QCA board, they are not sitting there senior-level oYcial on the board. As the Secretary of in regular meetings having their decisions or advice State said, we have given greater clarity to the role shaped by the departmental observer. What the of observers as we accepted the criticism that it was departmental observer would be doing— vague. It is exactly as the Secretary of State has said. Mr Stuart: That is what Ken Boston said, though. There is no real diYculty in providing a policy David Bell: Well, that is Ken Boston’s view, but I framework for the QCA board to consider at the don’t think it accords with the facts. same time as acting as an early warning back to the Ed Balls: In any case, Dr Ken Boston was not the Department. That is entirely reasonable and the chief executive of the QCA at the point at which he QCA board has accepted that. made those comments and he never once made any Jim Knight: As Lord Sutherland makes clear, there such comment to me at any point in the last 18 is a problem with scaling up the problems that were months in all the discussions that we ever had. At no happening. I was told throughout June that things point did he ever say to me, “DCSF observers were were going to be okay. In what became daily notes not playing a proper role.” The issue never, ever from oYcials on how it was going, I was told that the arose. NAA was confident that things were going okay. It said that it was confident that it could deliver Q426 Chairman: What lessons have we learned from results to schools—I am flicking through the all this? correspondence now. That is the information that we Ed Balls: We have learned that it is very important were getting, but it is also the information that that any organisation that has clear responsibilities David Gee, Ken Boston and the board were getting. must also have proper risk management procedures At the same time, about 40 people were put into the and ways of ensuring that information is escalated NAA to try to sort the problems out, but it now up to the decision-making level, and that decision appears that people at the top of the organisation did makers are then able to look across the piece at all not know that. the facts in assessing risks. On this particular Ed Balls: It would clearly be silly to say that because occasion, that did not happen at the level of the there is an observer from the Treasury at a meeting QCA’s chief executive and board. That was a of the Monetary Policy Committee every month, the substantial failure identified by Lord Sutherland, responsibility for making interest rate decisions is alongside the failure of ETS itself to deliver. There therefore really with the Chancellor. The same would are some lessons for our Department, although Lord apply in the case of the CQA board. Observers have Sutherland is very clear that our risk management a role to play, but it is really important to be clear procedures were eVective. However, there is a need about where the accountabilities lie and where the for some clarity about the role of the observers. decisions are taken. In this case, it was the QCA When you are managing big contracts and you have board. highly paid executives, it is essential that those executives are on top of all the facts at all times, Q424 Paul Holmes: So you have a fairly senior know exactly what is going on and act in a pre- observer there, a level 3 director general, but they are emptive way, not just on individual risks but on the neither influencing the QCA nor reporting back to accumulation of risks across the piece. In this case, you. I am not sure exactly what— that did not happen at the senior management and David Bell: That is not the case, Mr Holmes—they board level of the QCA. are reporting back. However, if you are making a Chairman: Thank you for that. That is the end of the specific point about the national curriculum tests evidence session. Permanent Secretary, we have and how they were going, it is very clear from Lord received your letter and we will be in correspondence Sutherland’s report that the board was not being with you about it. given information about potential problems. Ed Balls: Thank you very much indeed. Processed: 17-07-2009 19:27:34 Page Layout: COENEW [SO] PPSysB Job: 406183 Unit: PAG5
Children, Schools and Families Committee: Evidence Ev 73 Written evidence
Letter to the Chairman from Dr Ken Boston AO Thank you for the opportunity to respond to the points raised in evidence given to your Committee on 20 May 2009 by the Secretary of State, the Minister and the Permanent Secretary.
Q360 As the Committee is aware, I became aware of the flaws in paragraphs 4.92, 4.93 and 4.137 in the Sutherland Report on 11 December 2008. Neither the paragraphs nor the flaws had any impact on my decision to resign the following day: they were an unnecessary irritant, but not material to the resignation. I agree with Lord Sutherland that their correction does not aVect his findings. What has disturbed me is not so much the flawed evidence itself, but the repeated and quite unnecessary reliance on it by Ed Balls and Jim Knight in the House of Commons and in the Select Committee. The Secretary of State raised four issues in his opening statement to the Select Committee. My comments are as follows.
1. The 17 June meeting: cock-up or conspiracy? Does the evidence that “the Schools Minister did, in error, conflate two meetings” really stand up to scrutiny? I think not. A note of the meeting between Jim Knight and David Gee on 17 June was taken by a DCSF oYcial. Given the existence of the note, I find it odd—but not impossible—that Jim Knight gave incorrect evidence about the nature and date of the meeting to Lord Sutherland, or was insuYciently briefed to give correct evidence. But—that mistake having been made—surely he and his oYcials should have discovered the error when invited by Lord Sutherland to check the evidence against the documentation, and corrected it at draft Report stage. The Secretary of State and the Minister then relied heavily on this incorrect evidence in the House of Commons on 16 December 2008. I found that extraordinary, given the competence of DCSF oYcials. The job of oYcials is to protect their ministers; allowing ministers to mislead the House is a hanging oVence; given the assiduous manner in which oYcials attend to matters of detail, this account of events is highly implausible. Further, between my meeting with the QCA Panel on 14 January, and their letter to me on 30 January, the Panel obtained a copy of the note of the 17 June meeting from the DCSF. The letter of 30 January confirmed that the note did not record me as being present on 17 June. This meant that by 30 January at the latest, DCSF oYcials were aware not only that paragraphs 4.92 and 4.93 of the Sutherland Report were wrong, but that statements made by the Secretary of State and the Schools Minister in the House of Commons were incorrect. I cannot believe that DCSF oYcials did not immediately advise ministers of their error, knowing that ministers were about to appear before the Select Committee. What circumstances led Messrs Balls and Knight to repeat their statements—which their oYcials knew to be wrong—in the Select Committee on 4 February 2009? Had they not been told of their error? Or did ministers ignore that advice? It is incomprehensible that it was not until two days after his Select Committee appearance that Jim Knight was supposedly informed of the error by oYcials. Further, the manner of correction of the error was contemptible. It is not suYcient to have written to Lord Sutherland and to the Chairman of the Select Committee, and to have placed the letters in the libraries of both Houses. It is disgraceful that I should learn of the correction, and be given a copy of the letter of correction, only on the morning of my appearance before the Select Committee on 22 April 2009. I should have received a letter of explanation from both the Secretary of State and the Minister immediately the error was discovered, and so should the QCA. The correction was mean, concealed and shabby. No explanation has been attempted by ministers of paragraph 4.93 in the Sutherland Report. Lord Sutherland clearly saw and read a note of the meeting on 17 June, which he referenced as a footnote. It evidently lists me as attending the meeting. In late January however, the QCA Panel obtained from DCSF a note of the meeting, which does not list me as attending. What is the explanation? Are there two diVerent notes of the meeting? Or was the note seen by Lord Sutherland subsequently amended to correct the error?
2. “Ministers usually pressed QCA’s Chief Executive for answers” The explanation given by the Secretary of State is not consistent with any plain and fair reading of paragraph 4.137 of the Sutherland Report, or with his responses in the Commons to Michael Gove and David Laws. As I said in my letter of 15 April 2009 to the Select Committee “During the test delivery period, QCA was of course closely monitoring the blogs and the many reports in the media about problems with ETS, and working around the clock to resolve every issue which arose. We were in close contact with DCSF oYcials, who were also monitoring the situation”. Normal ministerial correspondence and telephone Processed: 17-07-2009 19:27:34 Page Layout: COENEW [E] PPSysB Job: 406183 Unit: PAG5
Ev 74 Children, Schools and Families Committee: Evidence
enquiries were properly being referred by DCSF to QCA throughout the test period, as a matter of course: getting his oYce to contact my oYce on 19 May, or my answering on 16 June just two of perhaps a hundred items of ministerial correspondence referred to us and answered as a matter of routine during this time, can hardly be construed as the Secretary of State personally and frequently pressing me for answers. During the 2008 test delivery period, I met with Ministers only twice: on 18 March, when the tests were not discussed; and on 2 June, when—as I have previously said to the Select Committee—I gave reassurances on progress to that date, on the basis of evidence I believed to be sound. Further, I take exception to the spin in the Secretary of State’s statement “It is true that Dr Boston did not attend the meeting on 17 June called at the request of the Schools Minister. Instead, his subordinate, Mr David Gee, did attend”. I did not attend the meeting because I was not asked to attend; I was not aware that Jim Knight had arranged a meeting directly with David Gee and not through my oYce until David came to me a day or so before to clear the papers he proposed to table at the meeting; I was annoyed that DCSF was again treating the NAA as if it were an independent organization rather than a division of the QCA; and as I had an important and prearranged teleconference at the time of the meeting I was unable to reschedule.
3. The terms of reference for Lord Sutherland The Secretary of State has not dealt with the real issue, which is that he himself caused Ofqual to stumble at its first real hurdle. The 2008 test failure presented Ofqual with the opportunity to establish itself as a truly independent regulator. If Government had wanted Ofqual to exercise real independence, and to get to all the root causes of the failure, it would have allowed Ofqual to establish terms of reference for Lord Sutherland which encompassed the role of all three partners in test development and delivery: Government, QCA and ETS. It should have been beyond question that Ofqual, with Lord Sutherland as its agent, would account to Parliament, not to the Secretary of State, and would do so through the Select Committee. Instead, the Secretary of State had Lord Sutherland report to him, not Ofqual, on QCA management of the ETS contract, which would otherwise have been a critical part of a comprehensive Ofqual inquiry. He narrowed the potential scope of the inquiry; put a fence around the DCSF; focused the spotlight on QCA and ETS; prevented some of the real causes of the failure from being identified; and compromised the independence of Ofqual. This should be remembered whenever we hear hollow Government rhetoric about Ofqual being an independent regulator. Ed Balls cut oV its oxygen at birth. For example, I refer to the timeline on page 22 of the Sutherland Report, and to the chapter on procurement as a whole. Lord Sutherland’s terms of reference required him to examine the role of QCA and its division the NAA in the procurement process from March 2006, and he did this very thoroughly. The remit he was given however did not allow him to take evidence from former ministers and oYcials on the role they played in determining the specifications to tender in that year. He did not interview Ruth Kelly and Jacqui Smith on the events up to May 2006, or Alan Johnson and ministers on events after that date; he did not interview present and former DCSF oYcials on the role they played in relation to deciding precisely what was to be procured; he was not able to interrogate oYcials on what messages they conveyed to the Procurement Board from ministers; nor was he able to investigate the decisions made by DCSF in 2006 for implementation in 2008 in relation to the maintenance of manual marking while insisting on the introduction of online question-level data capture. In the preface to his questions to me at the Select Committee on 22 April, the former minister Derek Twigg confirmed that ministers had discussed these issues in 2006 and decided against onscreen marking. In making this point, I am trying to ensure that Ofqual is never again wrong-footed by any government. I am not trying to spread the the blame, or criticize Lord Sutherland, or diminish my own accountability for the test failure in 2008. Flawed though it was, I accepted the remit given by ministers in 2006 for 2008, and I do not expect anyone else apart from ETS to take responsibility for the failure. I do not question Lord Sutherland’s findings. It is imperative however that, if Ofqual is to be a truly independent regulator reporting to Parliament, neither it nor its agent should accept terms of reference dictated by Government.
4. The QCA itself decided not to proceed with onscreen marking (!) From 2004, ministers and oYcials declined to accept QCA advice to introduce onscreen marking, on the grounds that it must first be proven by a pilot study that the national results obtained by that method would be the same as (or better than) those obtained by manual marking. Onscreen marking is not novel; it has been proven internationally to be a far more valid and reliable process than manual marking; and it is now widely used by UK awarding bodies for delivery of GCSEs and GCEs. Such a trial would have been worthless: onscreen marking cannot be judged by the extent to which it replicates results obtained by the much inferior process of manual marking. Inevitably the results would have been diVerent, although not necessarily worse. It is wrong for the Secretary of State to claim that the QCA itself decided not to proceed with onscreen marking. We were told by ministers and oYcials that it was not an option. The final stumbling block was the Government requirement that whole scripts be returned to schools at the same time as the results, as in the past. Onscreen marking requires that the spines be stripped from the scripts, so that each page can be scanned electronically. It would have been possible to send electronic facsimiles of the scripts or even hard Processed: 17-07-2009 19:27:34 Page Layout: COENEW [O] PPSysB Job: 406183 Unit: PAG5
Children, Schools and Families Committee: Evidence Ev 75
copies of the separately marked answers to schools by the results date, but not to reassemble and return whole scripts. The fact is that onscreen marking can provide schools with a far richer and more useful analysis of pupil performance than the return of marked scripts, as has been shown by the awarding bodies. The purpose of the letter signed by David Gee on 3 November 2005 was to resolve uncertainty for 2006, and to paper over what would otherwise have been a very public rift between the QCA and Government. With the benefit of hindsight, we negotiated for far too long: I should have stood our ground and put QCA advice publicly and in writing.
Q361 I did not work closely with Ed Balls. In 2007 I had telephone calls from him on 5 and 19 July; I first met him two months later on 19 September; we met on 23 October, spoke by telephone on 22 November, and met on 13 December. In 2008, I met him on 18 March and 2 June. In more than thirty years of working directly with ministers in five educational jurisdictions, including England, I have never had less contact. Further, it is not true that there has never been a cross word or diYcult exchange: it was only once, but it was memorable. I have been around for many years; I am hardly a shrinking violet; but I left the meeting on 18 March 2008 with the certain knowledge that the Secretary of State had no understanding of what the QCA was for, nor what it had achieved; that he had no respect for the capacity, integrity and dedication of its staV, its Executive and its Board; and that he had no willingness to listen.
Q363 This was the meeting on 18 March 2008. The Chairman and I presented, on behalf of the QCA Board, a case for retaining the name and identity of QCA although, in compliance with the Secretary’s wishes, not its regulatory functions. We outlined the risk to ongoing service delivery (including key stage tests, Diplomas, revision of GCSEs and GCEs, and the Qualifications and Credit Framework) arising from Government requirements to relocate to Coventry (with the loss of at least 85 per cent of our staV and at a cost in excess of £60m); from eYciency savings of 30 per cent over three years; and from the splitting of the organization into QCA and Ofqual ahead of the necessary legislation. It was in my view an aggressive and confrontational meeting.
Q 367 The process for marking the 2008 tests was set in stone from the point at which the decision to award the contract to ETS was approved by the then Secretary of State in January 2007. The introduction of onscreen marking, which requires special paper and printing and other technical requirements, involves a run-up of two years. My letter of 28 March 2008 simply reported progress with the 2008 test series.
Qs 368, 369, 370, 371 See above comments on responses to Q360, first issue.
Q 380 The QCA and DCSF oYcials were in regular contact over the the progress of test delivery, as noted in my comment on Q360, second issue. It is wrong however for the Secretary of State again to say “Ken Boston regularly gave me reassurance” and that “he was telling me on regular occasions that things were on track”. I did so once, and once only: on 2 June 2008. The meeting with Jim Knight on Wednesday 2 July 2008, which I attended with David Gee and others, could not reasonably have been confused with the meeting on 17 June. It had been arranged by the minister because we had been formally notified by ETS on Friday 27 June that they would fail to deliver. It was a meeting to assess the magnitude of the failure and to agree what to do about it. There is no agreed note of the meeting. We had been monitoring the figures from ETS every few hours for the previous five days; David Gee brought to the meeting the latest figures which had been received within the previous hour. Being the only one familiar at that stage with the latest data, he led us through it. The picture was very confused: as late as the next day, the ETS President emailed QCA to say they would meet the July 8 deadline, when it was clear to us they would not. The critical issue at this meeting was whether or not to release all available results on July 8, or withhold them all until a later date. Jim Knight asked me to give him formal advice to delay the release of results. I refused, on the grounds that I thought it better not to inconvenience the majority of schools, for which results would be available. He did not accept this advice: on that basis, I agreed to provide advice on the length of the delay, which I did next day in writing. This was a very tense exchange, which is etched in my memory and I am sure in his: I cannot see how it can possibly have been confused with the meeting on 17 June at Processed: 17-07-2009 19:27:34 Page Layout: COENEW [E] PPSysB Job: 406183 Unit: PAG5
Ev 76 Children, Schools and Families Committee: Evidence
which I was not present. Had the results been released as scheduled on 8 July, 93.5 per cent of key stage 2 results and 85.1 per cent of key stage 3 results would have been available to schools, instead of no results being available to any schools at all.
Q391 See my comments on Q360, third issue.
Q396 The “maxwellisation” drafts of the Sutherland Report were emailed to us in Word format, not PDF. They consisted of a series of extracts from the Report, below each of which a response was to be made. The drafts were forwarded by email by me and other recipients to the relevant members of the team which had prepared the QCA documentation and data for Lord Sutherland. The team began drafting proposed responses in Word format, using the “track changes” facility so that drafts could be shared. This immediately revealed, by name, the authors of the “maxwellisation” drafts, and the authors of subsequent changes. Substantial sections had been drafted by the DCSF oYcial seconded to the Sutherland Inquiry from the Permanent Secretary’s private oYce. This created a great deal of consternation. My advice to the QCA team was that in preparing a Report such as this it is normal for subordinates to do some of the drafting; that Lord Sutherland would without question take sole responsibility for every word in the entire Report; and that as our job was to prepare a response, rather than edit the drafts, the authorship was irrelevant and should be ignored.
Q400 On 24 September 2008, six days after I had given evidence to Lord Sutherland, three weeks before ministers had given their evidence, and nearly three months before the Sutherland report was published, the Permanent Secretary sought to negotiate and secure my early retirement. My subsequent response was that any negotiation would need to be on the basis of paying out the value of my contract to its conclusion on 30 September 2009. The contractual conditions (Clause 13 of the contract) included a potential bonus of up to 15%. Given the successful delivery by QCA in 2007–08 of such major reforms as the Diplomas, the review of GCSEs and GCEs, the Secondary Curriculum Review, the recognition of employer qualifications, and the Qualifications and Credit Framework—combined with the relocation to Coventry, the establishment of Ofqual and the achievement of the final year of eYciency savings—a bonus of 10% would have seemed reasonable even given the test delivery failure. Negotiations were never entered into: I had no intention of standing down. My resignation, on 12 December 2008, was entirely unrelated to the September meeting, and would have taken place without it. I resigned in the light of reading the Sutherland Report, and did so unilaterally under Clause 9 of my contract, without negotiation, without any special non-contractual deal being requested or received, and without requesting any bonus.
Q 406 The Permanent Secretary significantly misrepresents my position. At no time during the discussion on 24 September, or in the subsequent response to it, did I seek any non-contractual bonus or other non- contractual payments. At no time did I seek anything more than the value of the contractual entitlements. And although my contract was with the QCA Board, the details of the contract and variations to it since 2002 had all been approved by the Permanent Secretary or his predecessor.
Q411 I agree that all correspondence should be made available to the Select Committee, including the correspondence relating to Qs 400 and 406.
Q 419, 420 See comments in response to Q360, fourth issue.
Q421-425 The DCSF (and DIUS) observers within QCA have performed three functions: they have provided specialist expertise; they have served the traditional civil service role of providing separate advice to Ministers; and they have spoken on behalf of Ministers. My view is that Ofqual should not have observers from any Government agency. The first of the above three functions can be achieved by direct consultation with Government departments prior to decision- making; the other two are quite inappropriate for an independent regulator reporting to Parliament. Processed: 17-07-2009 19:27:34 Page Layout: COENEW [O] PPSysB Job: 406183 Unit: PAG5
Children, Schools and Families Committee: Evidence Ev 77
A particular problem arises when observers see their role as that of speaking for Ministers: typically, in phrases such as “ministers would be minded to” or “not be minded to” agree with a likely proposal, or that “ministers would be content” or “not content” with a particular recommendation. The risk is that independent advice oVered to the minister by the regulatory authority is negotiated in advance with DCSF observers, rather than arrived at independently but with the benefit of prior DCSF input. This leads to compromise solutions which erode the independence and public accountability of the regulator.
Ofqual will not be seen as conspicuously and unquestionably separate from Government so long as it has it accepts DCSF observers on its Board and committees. The public and the Parliament need to be absolutely certain that Ministers are being told what they need to know, not what they or their oYcials might want to hear.
A related problem is that formal QCA advice to ministers is in due course made public, and all other correspondence is available under FOI provisions. Hence DCSF oYcials like to see potential correspondence in draft, and often seek to negotiate its terms so that it might be acceptable to both ministers and the public. In the interests of good working relationships between organizations, and on many issues, this is perfectly satisfactory, but on the big issues it is not. Regulatory authorities must be prepared to draw lines in the sand, and do so publicly.
In conclusion Barry, since the Secretary of State’s statement in the House of Commons on 16 December, all I have wanted was an opportunity to be heard. I thank the Select Committee for giving me that opportunity, and for testing me and ministers thoroughly on substance and detail. I failed to deliver an important QCA remit, and I resigned accordingly. My expectation was that I would have been allowed an honourable resignation without Government “sinking the boot”, as they say in the Antipodes. I now want to rule oV on this latest chapter—the only sad chapter among many happy chapters—of my work in England, which has been so professionally stimulating and personally rewarding.
It has been a privilege to lead the QCA, to serve the cause of education and skills, and to be a participant in current important reforms in the building of human and social capital in the United Kingdom. My wife and I shall continue to live in England for some time. My very best wishes to you and to the Select Committee, and my thanks for your courtesy and fairness. June 2009
Letter to Lord Sutherland from the Rt Hon Jim Knight MP, Minister for Schools and Learners, Department for Children, Schools and Families In giving oral evidence to your Inquiry, I stated that I met Ken Boston and David Gee on 17 June. That statement is quoted in the report of your Inquiry, as follows:
[paragraph 4.137]: “As late as 17 June when the Schools Minister met QCA’s Chief Executive and NAA’s Managing Director, they provided reassurances.”
and
[paragraph 4.92]: We had that meeting on the 17 June with Ken [Boston] and David Gee. I may well have met David Gee before that point but it was the first time I had had a substantial discussion with him about things in great detail. He basically answered all the questions; Ken [Boston] referred everything to him. We went through everything with David [Gee]. He gave me some reassurance about the measures that were being taken to mitigate the loss of time and capacity, in particular the recruitment of marker panels, which seemed a reasonable way of dealing with the problem to me.”
This evidence is absolutely correct in all but one regard. I have now realised that in my account of events, I inadvertently conflated two separate meetings. On 17 June, I met David Gee (Managing Director of the National Assessment Agency (NAA)). Ken Boston was not present at that meeting. My oral evidence to you, quoted at paragraph 4.92 of the Inquiry’s report, accurately reflects the way in which Ken Boston referred questions to David Gee, but at a meeting with both of them which took place on 2 July. Although this does not alter the substance of my evidence, in relation to the points at which we were told of delays to delivery and the fact that in order to receive accurate information I needed to question QCA very closely, it may be helpful if I confirm the sequence of meetings that led to Ministers being informed of delays to the 2009 national curriculum tests.
As the evidence correctly records, on questioning David Gee on 17 June, I was told that ETS were on track to deliver results on 8 July, that there were contingencies in place to mitigate the late delivery of scripts for marking, including marker panels, and that—although Key Stage 3 English would be “a struggle”—results should be released on time. Processed: 17-07-2009 19:27:34 Page Layout: COENEW [E] PPSysB Job: 406183 Unit: PAG5
Ev 78 Children, Schools and Families Committee: Evidence
As your report makes clear, “In practice, the first time QCA notified Ministers that ETS would not deliver test results on time was 30 June 2008.” It was not until the afternoon of 30 June that the Schools Director General was first informed by David Gee and Ken Boston that there would be a delay to some results. They judged that 90% of school English Key Stage 3 and 95% of school English Key Stage 2 marks would be available on 8 July, and were confident that mathematics and science would be on time, except for a residual concern about a small proportion of Key Stage 3 science scripts. In a further message to oYcials later that day these figures were revised to show that there were diYculties in mathematics as well as English and science. At Key Stage 3, NAA expected only 80% of schools to have a full set of results in English by 8 July. Having been alerted to these serious delivery issues by oYcials on 30 June, I met both Ken Boston and David Gee at a subsequent meeting on 2 July. It was at this meeting that I questioned David Gee and Ken Boston in detail, with Ken referring questions to David. I apologise unreservedly for the error in my evidence. There was no intention to mislead you, and the position on attendance at the meeting on 17 June was correctly recorded in the Department’s written evidence to your inquiry. But I believe that it is important to take this opportunity to set the record straight on exactly when those meetings took place and with whom. I am copying this letter to Barry Sheerman, Chair of the Children, Schools and Families Select Committee. 9 February 2009
Letter to The Chairman from the Rt Hon Jim Knight MP, Minister for Schools and Learners, DCSF Since the Secretary of State and I gave evidence to your committee, it has come to our attention that there is an error in the oral evidence which I gave to Lord Sutherland during his inquiry. This error is reported in paragraph 4.137, which was quoted by the Secretary of State at your committee last week: “As late as 17 June when the Schools Minister met QCA’s Chief Executive and NAA’s Managing Director, they provided reassurances.” I have now realised that in my account of events, I inadvertently conflated two separate meetings. On 17 June, I met David Gee (Managing Director of the National Assessment Agency (NAA)). Ken Boston was not present at that meeting. I have written to Lord Sutherland explaining the mistake and the accurate series of events, and I attach a copy of my letter to him. As I say in my letter, there was no intention to mislead Lord Sutherland, and the position on attendance at the meeting on 17 June was correctly recorded in the Department’s written evidence to the inquiry. But I believe that it is important to take this opportunity to set the record straight on exactly when those meetings took place and with whom. I am copying this letter to Lord Sutherland. 9 February 2009
Letter to the Rt Hon Jim Knight MP, Minister for Schools and Learners, DCSF from Lord Sutherland Thank you for your letter of 9 February 2009. You have explained that in a meeting of 17 June 2008, you met with the National Assessment Agency (NAA) Managing Director David Gee but that the Qualifications and Curriculum Authority (QCA) Chief Executive Dr Ken Boston was not present. You have apologised for conflating your recollection of this meeting with that of a meeting on 2 July 2008 at which both Mr Gee and Dr Boston were present. As reflected in my report, you had previously made this comment during your interview and had the opportunity to review your transcript. However, I appreciate that you have now provided this clarification and I have asked my secretariat to place a short note on the Inquiry’s website to reflect this. I agree that your clarification does not substantially alter the points that were made. It remains clear that NAA provided reassurances in the 17 June meeting that test results could be delivered by 8 July 2008 and that contingency measures such as marking panels were in place. It also remains clear that NAA was playing the leading role in managing the tests—with its Managing Director attending a meeting with you alone on 17 June and a joint meeting with the QCA Chief Executive present on 2 July. My report concluded that within the QCA there was a strong emphasis on NAA’s responsibility for the tests and that there was insuYcient corporate oversight at QCA Executive and Board level. This is borne out by extensive evidence Processed: 17-07-2009 19:27:34 Page Layout: COENEW [O] PPSysB Job: 406183 Unit: PAG5
Children, Schools and Families Committee: Evidence Ev 79
detailed throughout my report, for example at paragraphs 3.27, 3.30, 3.65, 3.67, 3.69, 3.95, 3.107–3.112, 4.80–4.88, 4.104–4.106, 4.125–4.135. I am therefore satisfied that your clarification does not aVect my findings. I am copying this letter to the Chairman of the Children, Schools and Families Select Committee. 12 February 2009
Letter to The Chairman from the Rt Hon Jim Knight MP, Minister for Schools and Learners, DCSF I wrote to you on 9 February to inform you that I had written to Lord Sutherland, to correct an error in the oral evidence I gave to him during his inquiry. Lord Sutherland has now replied to my letter, and has made clear that my clarification of the evidence does not substantially alter the points that were made. I am placing copies of my letters to you and Lord Sutherland, and Lord Sutherland’s reply to me, in the libraries of both Houses. 12 February 2009
Printed in the United Kingdom by The Stationery OYce Limited 7/2009 406183 19585