SECTION 10: RESPONSE RELATING TO THE QUALITY CRITERIA

Instruction to Bidders: There is no word limit, but try to keep responses concise, answering the questions for each criterion– refer to the specification if necessary and be mindful of the weighting. Please do not include or attach reference documents or brochures unless specifically requested to as evidence:

Qualifications Cymwysterau Cymru Q2 Building, Pencarn Lane Adeilad Q2, Lôn Pencarn Imperial Park, Parc Imperial, Newport Casnewydd NP10 8AR NP10 8AR  0333 077 2701 www.qualificationswales.org  0333 077 2701

Quality Evaluation Criteria 1: Methodological approach 30% Suppliers Response:

Why are we responding to this tender opportunity? We are a partnership of vocational education providers and national membership bodies. We believe we are the best people to understand the challenges and benefits of the use of technology in assessment and we are the best people to identify a wide range of examples. The collated list of examples (subject to Qualification Wales' permission), will in its own right be of great value to the sector and provide an incentive for making a response.

Challenge The key challenge for this research project will be ensuring that a balanced, representative collection of substantive examples from across all of vocational education – further education, work-based learning and in-work learning is compiled. It may be necessary to look for examples in across the rest of the UK.

Phase One Phase one, as prescribed by the tender specification is a short exercise to identify a range of candidate examples of the use of technology in assessment. We have split this down into two activities: 1) Canvassing for examples and 2) Initial collation and review. Canvas If successful, we will establish an expert reference group of professionals either involved in promotion of the use of technology in teaching and learning from the sector or representing key elements of vocational education curriculum management. The expert reference group will consist of the following and details of their roles are identified in our response to Quality Evaluation Criteria 3:

 Karen Coyle, Learning Systems Developer, ;  Colin Bevan, Head of ILT, Group of Colleges;  Kelly Edwards, Head of WBL Quality, NTFW;  Dr Geoff Elliott, Learning and Technology Development Manager, ;  Peter Kilcoyne, Head of the Blended Learning Consortium, Heart of Worcester College;  Jonathan Morgan, Assistant Principal, Coley y Cymoedd;  Kate Pearce, ILT Team Leader, ; and  Berni Tyler, Head of the bWBL consortium, Pembrokeshire College.

Dr Geoff Elliott will be the project manager and Pembrokeshire College will be the coordinating body. The process for canvassing for examples will consist of two elements to seek examples from:  The sector via the expert reference group; and  Wider afield via sector related groups and organisations.

The actual canvassing will be consistent and systematic in nature and will follow this methodology: Element 1: expert reference group canvassing  A standard message will be crafted to convey the purpose of the research, the request and the timeline. The message will also cover all ethical, confidentiality and GDPR aspects.  Respondents will be asked to complete a template-form (see below).  As an incentive, all respondents will be offered a copy of an anonymised list of all examples submitted (subject to Qualification Wales approval).  Each member of the expert reference group will list all the people they will send the message proceed to send it via email.  All completed forms will be returned either directly to the Project Manager’s office or via the member of the expert reference group first for compiling.  We will use a matrix that attempts to cover vocational subject area, qualification level and identifying examples as WBL or further education. Element 2: Wider reach canvassing  A similar standardised message will be used to reach the wider vocational education sector and associated organisations including but not exclusively: o ColegauCymru; o Association of Colleges; o Association of Employment and Learning Providers; o JISC; o Sector Skill Councils; and o Awarding bodies.  In the timescale, it is anticipated that these organisations will only be able to provide examples that they have already collated and promoted.  These examples will not be in the prescribed format and it may be necessary to contact the original providers responsible for more details of the example, if possible.  The work will be carried out by a research assistant, who will attempt to redefine each example found in the prescribed format of the other examples.  All examples will be returned to the project manager’s office. Case study / example Response Template Using a standardised template for collecting examples means that they will all be consistent with each other and therefore make it easier to compare and select the best examples subset. The use of the standardised template will also enable us to respond to the nine objectives outlined in section 2 of the tender specification document.

The response template for each example will contain these headings:

Title of example A useful handle for referring to the example Organisation (WBL, FE, Employer based) Distinguishing between types of provider is clearly important Subject Sector Area (using WG This is to ensure that we can curate a Government categories) representative sample, ideally from each relevant Sector Subject Area. Learning aims (using WG categories, e.g. This is to ensure that we can curate a NVQ, First Diploma) representative sample from across difference Learning Aims. Language Was the example conducted through the medium of English, Welsh or mixed? Description Description of the context of the example -the key people involved, the type and age of the learners, the specific element of a qualification that was being assessed, the timescale over which the example took place. This question will include prompts to enable respondent to answer as fully as possible Challenges Description of the challenges that the respondent faced that merited the use of technology to mitigate. Benefits and Impact Description of the benefit and the impact. We will also define at least one Likert scale to help respondents determine the level of benefit/impact, each point on the scale will have a descriptor. 1: High benefits – 5: Very low benefits. This is will help to compare one example against another. Drawbacks Description of the drawbacks. We will also define a Likert scale to help respondents determine the severity of the drawbacks, each point on the scale will have a descriptor. 1: Many drawbacks – 5: Few drawbacks. This is will help to compare one example against another. Tags We will also define a set of tags that can be selected so that it is easier from someone to complete. The tags will be grouped into ‘Benefits’, ‘Drawbacks’ and include accessibility, ethnicity. Contact details For use for the second phase Online form We will also build the response form as an online form that we can then broadcast and promote so that we can reach the widest possible audience and therefore maximise our chances of finding the best examples. Use of rich media We will not be collating images and rich media in stage one due to potentially the large volume of examples and time required to process and compare examples. Interview It may be necessary to conduct interviews with respondents, if this is the case, we may also use the Trint.com, an AI-based transcription service to enable us to deconstruct the interview in order to complete the response template ourselves. This could be the case with examples from employers, who will not understand all elements of the response form. Examples from outside of Wales It would be short sighted not to look for examples from outside of Wales and potentially outside the UK. For this reason, we have co-opted the Director of the Blended Learning Consortium, based at Heart of Worcester College that represents over 100 vocational providers across the UK (including Pembrokeshire College, Neath Port Talbot College, and Coleg Cambria). Sampling frame In the context of this project, the concept of a sampling frame, as used in rigorous, science-based methodologies is not particularly relevant. The most important consideration is that the whole sector has been canvassed and examples are sort from all Sector Subject Areas and Learning Aims and from different types of provider. We believe our expert reference group providers a substantive representation of all aspects of vocational education in Wales. Recruiting ‘Recruiting’, as used in a rigorous, science-based methodology is not particularly relevant for this project; what is important is that the whole sector has been canvassed and examples sort from all Sector Subject Areas and Learning Aims and from different types of provider. Ethical guideline The details of the staff and learners involved in the examples will not be shared with the expert reference group at phase one. The staff and learners in the examples selected for phase two will be approached for permission to be consulted about their example of technology in assessment. Initial collate and review Once all the respondents have been gathered via our online form. It will be downloaded as a spreadsheet upon which we will add filters as shown in the indicative example below:

The spreadsheet database of examples will be shared with the expert reference group via Google Drive. The total number of examples will be shared out amongst the group, depending on the mix of examples the examples could be shared out by SSA, Learning Aim, Language or a union of these. Each member of the expert reference group will be required to score the examples they have been given allocated, from 1: Highly relevant to 5: Very irrelevant. The examples spreadsheet will be shared via Google Drive or Office 365 and enable everyone to edit the spreadsheet at the same time. Members of the group will also be encouraged to look at all other examples listed and identify others they think deserve to be considered for inclusion in the final 10 examples

Following this exercise, the expert reference group will meet at least once to agree the final set of examples to include in phase 1 of this project. Members will propose their top two examples and then the group will work through each example to make a decision. This process will inevitable result in some discussion but all members will understand objective is to identify the best examples. Once this initial set of examples has been agreed, a second or third iteration may be required if the group feel that certain SSAs, LAs are underrepresented or sector leading examples have been omitted.

In the timescale allowed, the workshop(s) may take place via web conferencing. Members will also have the liberty to delegate to trusted colleagues with the permission of the project manager.

Phase Two The biggest challenge for phase two is finding sufficient research findings in the literature to triangulate against the findings from the individual examples identified. Delivery methods in vocational education are not a major focus of research that tends to focus on the compulsory school or the higher education sectors. Therefore, it may be the case that identified examples will have to be triangulated against research findings in the school or university sectors. We will use ‘within-method’ methodological triangulation (Denzen, 1979) to compare the findings from the selected examples, the literature; and interviews, in order to either confirm or contradict findings from the selected example or demonstrate that no conclusions can be made.

We will be using the expert reference group extensively in phase two to:  Provide contacts to relevant people related to the chosen examples;  To validate, corroborate or contra wise the findings of literature review, or interviews with anyone involved in each of the selected examples;  Identify other relevant connections (people or other literature) for the researchers to pursue with reference to each example; and  Endorse the draft report submitted to Qualification Wales. The research process for each example will consist of these sequence of activities:  Initiate literature review;  Contact individuals involved in example, or secondary individuals if any of original people have moved on, seek permission and then conduct telephone interviews;  Correlate interview data with literature review and formulate a ‘standpoint’ via a report on the example;  Refer the standpoint write up to the expert reference group for a response;  Follow up any further sources identified; and  Alter the standpoint report following feedback from the expert group.

An initial set of standardised interview questions will be formulated for the basis of the interviews to ensure consistency between examples but clearly as each example will be unique there will be a range of supplementary questions. Typical questions will include:

 What is the background context to the example?  What were the objectives of using technology in assessment?  What were the challenges?  How did the use of technology in the assessment progress?  What were the outcomes?  What were the benefits?  What were the conclusions? and  How have you revised your approach to using technology in assessment?

Denzin, Norman K. (1978) The Research Act, 2d ed. New York: McGraw-Hill

Quality Evaluation Criteria 2: Implementation 25% Suppliers Response:

Plan and Timescales The Gantt chart below demonstrates how we will achieve the requirements of this specification. The project will employ a researcher for the duration of the project, who will be responsible for collating all the examples, communicating with the expert reference group and conducting the desk research to identify pre-existing examples of technology used in assessment. The project manager will oversee the project and ensure:

 The project remains on track by monitoring the Gantt chart daily;  The methodology is followed rigorously; and hence  Maintain a high quality of service.

The expert reference group will be responsible for relaying the invitation to submit examples, and will be responsible for evaluating and selecting the best examples for relaying to Qualification Wales.

The Gantt chart for phase one is shown below:

We recognise that Qualification Wales would like phase one completed by the 7th of January however, the Christmas and New Year holidays will create a hiatus that would mean realistically finishing phase one a week later than specified.

In Phase two, we would plan to start work on two examples at a time, with up to two weeks before starting examples 3 and 4 and two weeks before starting 5 and 6 (if needed). We estimate that the work to complete each example will last over 4 working weeks (not continuously). The Gantt chart for each example will follow this pattern of activities as shown here:

The schedule for working on each example will therefore be:

Example Start End 1 22.01.19 25.01.19 2 22.01.19 25.01.19 3 04.02.19 08.03.19 4 04.02.19 08.03.19 5 15.02:19 21.03.19 6 15.02:19 21.03.19

The overall Gantt chart for phase 2 is shown below. We will be conducting a generic literature review on the use of technology in assessing vocational qualifications throughout most of phase two in order to identify any wider findings related to the use of technology in assessment of vocational qualifications that have not picked up research each of the examples.

Examples communicated through Welsh All the communication with people and learners in the sector canvassed for examples will be provided bilingually. Anyone providing an example will also be asked if these people wish to continue through the medium of Welsh, if so, suitable arrangements will be put in place within 24 hours, during the working week. Pembrokeshire College employ two Welsh language speakers, whose roles include the facilitation of communication through Welsh, who will be co-opted as necessary.

The expert reference group will consist of the following professionals and details of their roles are identified in our response to Quality Criterion Three but include:

 Karen Coyle, Learning Systems Developer, Coleg Cambria;  Colin Bevan, Head of ILT, Neath Port Talbot Group of Colleges;  Kelly Edwards, Head of WBL Quality, NTFW;  Dr Geoff Elliott Learning and Technology Development Manager, Pembrokeshire College;  Peter Kilcoyne, Head of the Blended Learning Consortium, Heart of Worcester College;  Jonathan Morgan, Assistant Principal, Coley y Cymoedd;  Kate Pearce, ILT Team Leader, Gower College Swansea; and  Berni Tyler, Head of the bWBL consortium, Pembrokeshire College.

Risks We have identified these risks and list the measures we will adopt to mitigate against them occurring.

Major risks Response Not enough examples are collated or it This is extremely unlikely. The members of the transpires that it is hard to find examples expert reference group has been selected to represent the majority of vocational education providers in Wales and very large part of the vocational education sector across the UK. The collation of examples becomes The project manager has significant experience in protracted beyond agreed timelines. running successful projects on time (some award winning) and will ensure that adherence to the timeline is adhered. The use of the template for collecting examples will ensure that it is easy to record them. Members of the expert reference group The expert reference group is relatively large so choose to leave the group during the the loss of one member would not have a project for whatever reason significant impact and the timeline for the project is relatively short. In addition, as the response form will be mailed out, it is to some extent independent of members of the expert reference group once sent.

Data handling GDPR We will be governed by the GDPR principles of:

Content We will ask for consent formally a recording via a consent form. Transparency As part of the communication with anyone who will be potentially providing an example or referred, we will use standard messages that have been approved by the expert reference group. These messages will be governed by Article 13 of the GDPR - “Information to be provided where personal data are collected from the data subject” in as accessible a format as possible. Safeguards All data collected will be anonymised wherever possible and we will limit it to the role and involvement in the examples collected. We will retain a mechanism to be able to refer back to the originators of each example. Data subject rights In the communicating with people who may be referred-to in any reports, we will make it clear that they have the right to:  erasure  access by the data subject  rectification  restrict processing  object to processing

Storage of data We will be using Google Drive to share work that will require all those working as part of this project to have accounts to access. Google Drive uses AES-256 to protect file transfers, and AES- 128 to encrypt files at rest. AES is a very secure encryption algorithm without any currently feasible attacks, and is the current US Government encryption standard. We may choose to use two-step verification on any particularly sensitive files. Quality Evaluation Criteria 3: Relevant experience 20%

Suppliers Response:

This project will be led and managed by Dr Geoff Elliott. Dr Elliott, who is an expert in learning technology and has excellent experience in qualitative data research methodologies in his own research, including the main research methodologies in used in this doctorate research. (https://repository.cardiffmet.ac.uk/handle/10369/6424).

Dr Elliott has had published articles in a number of journals and conference proceedings and has two books published as indicated in this brief bibliography.

 Elliott, G. J., Jones, Eleri and Barker, P (2003) A Grounded Theory approach to modelling learnability of hypermedia authoring tools. Interacting with Computers.  Elliott, G. J. and Cunliffe, D. J. (2003) Crucial Study Guide: Multimedia Computing, Learning matters Ltd. ISBN: 1903337186.  Daniels, B., & Elliott, G. (2003). Developing and Applying a Generic Set of Accessibility Guidelines to E-Learning Web Sites. World Conference on Educational Multimedia, Hypermedia and Telecommunications 2003. (award winning paper)  Elliott, G., Cook, J., Monk, D., Burnett, S., & Lynch, M. (2003). Building an e-learning development team, platform and process from scratch. World Conference on Educational Multimedia, Hypermedia and Telecommunications 2003(1), 2813-2820  Elliott, G. J. and Richardson, P. (2006) Towards an Institutional Benchmarking and Staff Skills Profiling Tool for Learning Technology, Association of Learning Technology, Edinburgh.  Elliott, G. J. (2007) Web site Management, Lexden Publishing, ISBN-10: 1904995217, London.

The paper ‘A Grounded Theory approach to modelling learnability of hypermedia authoring tools.’ identified above is included with our submission to demonstrate writing ability.

Since completing his PhD in 1999 in Learning Technology, Geoff has had a role in developing the use of learning technology in higher education and further education. Geoff was the project manager for eCollege Wales (2000-2003), a pioneering elearning initiative at the . Geoff is now the Head of elearning at Pembrokeshire College, whose ILT work has led to two national AoC Beacon awards. Geoff was a member of the Welsh Governments National Digital Learning Council and is currently a member of Welsh Government’s Digital 2030 steering group.

expert reference group The expert reference group consists of professionals from the vocational education sector, representing the further education and the private provider aspects in either generic roles or learning technology roles. Colin Bevan Head of ILT at NPTC Group of Colleges. Colin has worked in the FE sector for over 30 years and taught on a range of IT courses from Level 1 to Level 4. He still teaches computer programming at Level 3. Current role involves managing IT Services for the Group of Colleges and overseeing the use of technology in the curriculum that includes the virtual learning environment and Office 365. He is responsible for the strategic direction of IT across NPTC Group of Colleges. Karen Coyle Karen has had an active role in ILT development at Coleg Cambria and previously since 2002. Karen was responsible for the development and rollout of the VLE for seven years and her current role is to gain an understanding of the College’s Technology strategy and to contribute and develop technology support for these activities. Karen is also the lead in the developing applications and methods with respect of teaching and learning. Karen works with colleagues in the College to develop a strategic approach to technology-enhanced learning, in particular the Director of Technology and Library Services and the Assistant Directors of Curriculum at each college site. Kelly Edwards Kelly is the Head of WBL Quality at NTFW, with strategic responsibility for leading and coordinating quality improvement across the Welsh WBL sector. Supported by Welsh Government, Kelly’s work involves liaising with the network of WBL providers commissioned to deliver the apprenticeship and traineeship programme, to identify opportunities to enhance the quality of delivery and share best practice from within the WBL sector in Wales.

Prior to this, Kelly worked in Adult Community Learning as Senior Research and Policy Officer at Adult Learning Wales and at the University of South Wales for over ten years in a variety of roles, including Senior Lecturer (WBL), Project Manager, Foundation Degree (Business Practice) Programme Leader, Learning Development Officer (WBL) and Graduate Careers Adviser. Her research interests focused on WBL as a route to widening participation to Higher Education.

Kelly is a member of the Welsh Government’s Digital 2030 Steering Group Peter Kilcoyne Peter leads the team that won 2015 TES and 2014/15 AoC awards for their work in successfully implementing blended learning across Heart of Worcestershire College and supporting other colleges across the sector in implementing their own blended learning curriculum.

This work has been built upon through setting up the Blended Learning Consortium which now has over 100 Colleges as members and in its first three years developed and shared over 1300 hours of online learning content.

Peter has been ILT Director at Heart of Worcestershire College (previously Worcester College of Technology) since 2006. Before this, he worked for Jisc as an ILT Curriculum Advisor and has had 14 years’ experience working as a Lecturer and Manager in the FE and HE sectors.

Peter will be able facilitate finding examples of technology in assessment across the rest of the UK. Jonathans Morgan Jonathan is the Assistant Principal Data and Planning at and his main responsibilities are:

 Preparing the curriculum plan for the college which involves working with colleagues to prepare the curriculum plan for submission to Welsh Government to ensure that the college achieves its funding target of £40 million;  Interpreting, analysing, evaluating data, and measuring the impact of quality improvement processes. Jonathan’s work has resulted in innovative changes to quality processes, resulting in achievement rates rising by 3% between 16/17 and 17/18; and  Designing and implementing digital platforms and services. Jonathan is also a member of Welsh Government Digital 2030 Steering group. Kate Pearce Kate is ILT Team Leader at Gower College Swansea, whose role encompasses much of the college operations, both business support and academic activities. The responsibilities cover the development and management of the College’s learning platforms, all college websites, the development of distance and blended learning courses and supporting teaching staff in the use of technology to support and enhance teaching and learning. The department is also been successful in the completion a number externally funded projects, focused on the development of innovative ideas with technology including working with national bodies to create interactive learning materials. One of the Kate’s recent projects was the development of electronic signatures for Educational Maintenance Allowance (EMA) that now been adopted by Welsh Government and most recently developed a pan-Wales diagnostic assessment for student with additional learning needs. Berni Tyler Berni Tyler has over 20 years’ experience in the management and delivery of apprenticeships and in curriculum development in both the private sector and FE. Berni is currently the head of the bWBL work based learning consortium, a mix of 10 FE and private WBL providers. Berni developed the Welsh Baccalaureate Foundation level qualification on behalf of the WJEC (2005-2007) and has led a number of projects for CollegesWales including Lloyds bank Money for Life and CQFW (2007-2011). Berni is a member of the board of the NTFW the EWC and an Estyn Peer inspector.

SECTION 11: PRICING SCHEDULE

To be completed by the bidder

INVITATION TO TENDER FOR: Qualitative research to explore opportunities for technological innovation in the effective assessment of skills in vocational contexts

TENDER REFERENCE: QW181929

NOTE: Qualifications Wales requires prices to be provided inclusive of VAT

Price is worth 25% of the overall score, Please provide a breakdown of costs for each element of the work

Description Price including VAT Phase One £10,860 Phase Two £12,360

Total cost £23,220