SECTION 10: RESPONSE RELATING TO THE QUALITY CRITERIA Instruction to Bidders: There is no word limit, but try to keep responses concise, answering the questions for each criterion– refer to the specification if necessary and be mindful of the weighting. Please do not include or attach reference documents or brochures unless specifically requested to as evidence: Qualifications Wales Cymwysterau Cymru Q2 Building, Pencarn Lane Adeilad Q2, Lôn Pencarn Imperial Park, Parc Imperial, Newport Casnewydd NP10 8AR NP10 8AR 0333 077 2701 www.qualificationswales.org 0333 077 2701 Quality Evaluation Criteria 1: Methodological approach 30% Suppliers Response: Why are we responding to this tender opportunity? We are a partnership of vocational education providers and national membership bodies. We believe we are the best people to understand the challenges and benefits of the use of technology in assessment and we are the best people to identify a wide range of examples. The collated list of examples (subject to Qualification Wales' permission), will in its own right be of great value to the sector and provide an incentive for making a response. Challenge The key challenge for this research project will be ensuring that a balanced, representative collection of substantive examples from across all of vocational education – further education, work-based learning and in-work learning is compiled. It may be necessary to look for examples in across the rest of the UK. Phase One Phase one, as prescribed by the tender specification is a short exercise to identify a range of candidate examples of the use of technology in assessment. We have split this down into two activities: 1) Canvassing for examples and 2) Initial collation and review. Canvas If successful, we will establish an expert reference group of professionals either involved in promotion of the use of technology in teaching and learning from the sector or representing key elements of vocational education curriculum management. The expert reference group will consist of the following and details of their roles are identified in our response to Quality Evaluation Criteria 3: Karen Coyle, Learning Systems Developer, Coleg Cambria; Colin Bevan, Head of ILT, Neath Port Talbot Group of Colleges; Kelly Edwards, Head of WBL Quality, NTFW; Dr Geoff Elliott, Learning and Technology Development Manager, Pembrokeshire College; Peter Kilcoyne, Head of the Blended Learning Consortium, Heart of Worcester College; Jonathan Morgan, Assistant Principal, Coley y Cymoedd; Kate Pearce, ILT Team Leader, Gower College Swansea; and Berni Tyler, Head of the bWBL consortium, Pembrokeshire College. Dr Geoff Elliott will be the project manager and Pembrokeshire College will be the coordinating body. The process for canvassing for examples will consist of two elements to seek examples from: The sector via the expert reference group; and Wider afield via sector related groups and organisations. The actual canvassing will be consistent and systematic in nature and will follow this methodology: Element 1: expert reference group canvassing A standard message will be crafted to convey the purpose of the research, the request and the timeline. The message will also cover all ethical, confidentiality and GDPR aspects. Respondents will be asked to complete a template-form (see below). As an incentive, all respondents will be offered a copy of an anonymised list of all examples submitted (subject to Qualification Wales approval). Each member of the expert reference group will list all the people they will send the message proceed to send it via email. All completed forms will be returned either directly to the Project Manager’s office or via the member of the expert reference group first for compiling. We will use a matrix that attempts to cover vocational subject area, qualification level and identifying examples as WBL or further education. Element 2: Wider reach canvassing A similar standardised message will be used to reach the wider vocational education sector and associated organisations including but not exclusively: o ColegauCymru; o Association of Colleges; o Association of Employment and Learning Providers; o JISC; o Sector Skill Councils; and o Awarding bodies. In the timescale, it is anticipated that these organisations will only be able to provide examples that they have already collated and promoted. These examples will not be in the prescribed format and it may be necessary to contact the original providers responsible for more details of the example, if possible. The work will be carried out by a research assistant, who will attempt to redefine each example found in the prescribed format of the other examples. All examples will be returned to the project manager’s office. Case study / example Response Template Using a standardised template for collecting examples means that they will all be consistent with each other and therefore make it easier to compare and select the best examples subset. The use of the standardised template will also enable us to respond to the nine objectives outlined in section 2 of the tender specification document. The response template for each example will contain these headings: Title of example A useful handle for referring to the example Organisation (WBL, FE, Employer based) Distinguishing between types of provider is clearly important Subject Sector Area (using WG This is to ensure that we can curate a Government categories) representative sample, ideally from each relevant Sector Subject Area. Learning aims (using WG categories, e.g. This is to ensure that we can curate a NVQ, First Diploma) representative sample from across difference Learning Aims. Language Was the example conducted through the medium of English, Welsh or mixed? Description Description of the context of the example -the key people involved, the type and age of the learners, the specific element of a qualification that was being assessed, the timescale over which the example took place. This question will include prompts to enable respondent to answer as fully as possible Challenges Description of the challenges that the respondent faced that merited the use of technology to mitigate. Benefits and Impact Description of the benefit and the impact. We will also define at least one Likert scale to help respondents determine the level of benefit/impact, each point on the scale will have a descriptor. 1: High benefits – 5: Very low benefits. This is will help to compare one example against another. Drawbacks Description of the drawbacks. We will also define a Likert scale to help respondents determine the severity of the drawbacks, each point on the scale will have a descriptor. 1: Many drawbacks – 5: Few drawbacks. This is will help to compare one example against another. Tags We will also define a set of tags that can be selected so that it is easier from someone to complete. The tags will be grouped into ‘Benefits’, ‘Drawbacks’ and include accessibility, ethnicity. Contact details For use for the second phase Online form We will also build the response form as an online form that we can then broadcast and promote so that we can reach the widest possible audience and therefore maximise our chances of finding the best examples. Use of rich media We will not be collating images and rich media in stage one due to potentially the large volume of examples and time required to process and compare examples. Interview It may be necessary to conduct interviews with respondents, if this is the case, we may also use the Trint.com, an AI-based transcription service to enable us to deconstruct the interview in order to complete the response template ourselves. This could be the case with examples from employers, who will not understand all elements of the response form. Examples from outside of Wales It would be short sighted not to look for examples from outside of Wales and potentially outside the UK. For this reason, we have co-opted the Director of the Blended Learning Consortium, based at Heart of Worcester College that represents over 100 vocational providers across the UK (including Pembrokeshire College, Neath Port Talbot College, Cardiff and Vale College and Coleg Cambria). Sampling frame In the context of this project, the concept of a sampling frame, as used in rigorous, science-based methodologies is not particularly relevant. The most important consideration is that the whole sector has been canvassed and examples are sort from all Sector Subject Areas and Learning Aims and from different types of provider. We believe our expert reference group providers a substantive representation of all aspects of vocational education in Wales. Recruiting ‘Recruiting’, as used in a rigorous, science-based methodology is not particularly relevant for this project; what is important is that the whole sector has been canvassed and examples sort from all Sector Subject Areas and Learning Aims and from different types of provider. Ethical guideline The details of the staff and learners involved in the examples will not be shared with the expert reference group at phase one. The staff and learners in the examples selected for phase two will be approached for permission to be consulted about their example of technology in assessment. Initial collate and review Once all the respondents have been gathered via our online form. It will be downloaded as a spreadsheet upon which we will add filters as shown in the indicative example below: The spreadsheet database of examples will be shared with the expert reference group via Google Drive. The total number of examples will be shared out amongst the group, depending on the mix of examples the examples could be shared out by SSA, Learning Aim, Language or a union of these. Each member of the expert reference group will be required to score the examples they have been given allocated, from 1: Highly relevant to 5: Very irrelevant. The examples spreadsheet will be shared via Google Drive or Office 365 and enable everyone to edit the spreadsheet at the same time.