UKRAINIAN STANDARDIZED EXTERNAL TESTING INITIATIVE (USETI) LEGACY ALLIANCE PHASE II

FEBRUARY 1, 2013– DECEMBER 31, 2016

GLOBAL DEVELOPMENT ALLIANCE

QUARTERLY REPORT: Y4Q4 PLUS TWO MONTHS OF NO-COST –EXTENSION PERIOD (OCTOBER 1, 2016 – FEBRUARY 28, 2017)

SUBMITTED BY: AMERICAN COUNCILS FOR INTERNATIONAL EDUCATION

______

GDA TABLE OF CONTENTS: PAGE SUMMARY OF ACTIVITIES…………………………………2 ACTIVITIES COMPLETED…………………………………..5

ANNEXES: 1. STATUS OF ACTIVITIES OF TASKS 2. PERFORMANCE MONITORING AND EVALUATION CHART 3. USETI ALLIANCE PARTNER LEVERAGE CONTRIBUTION

USETI Alliance Phase II Quarterly Report Y4Q4 plus NCE (October 2016 – February 2017)

1. Summary of Activities

The USETI Legacy Alliance Phase II is a Global Development Alliance funded by the U.S. Agency for International Development (USAID) under a cooperative agreement. American Councils for International Education is the Cooperating Partner and lead implementer, collaborating with USAID and 19 other founding partners. The project builds on accomplishments of previous U.S. Government assistance to: build a sustainable, strengthened UCEQA system more capable of independently and transparently developing and implementing secure tests that meet international standards; secure a legislative and regulatory policy basis for testing and higher education admissions; and build a sustainable high level of public support for equal access to Ukrainian higher education.

Throughout the reporting period (October 2016 through February 2017 which includes a two month-long no cost extension period (also referred to in the report as Q16) USETI Legacy Alliance Phase II activities were focused on supporting UCEQA with test development, UCEQA IT System recovery, conducting an international conference, completing a validity study, developing a Road Map with UCEQA and executing a number of other policy development-related activities.

Issues

1.1. Significant Events

Preparations for SET. In 2016 the registration for SET will take place from February 6 to March 17, 2017. As in previous years, SET participants will have the opportunity to gauge their readiness, familiarize themselves with procedures for testing and perform tasks as close as possible to the problems used in the testing session during the trial SET. Trial testing is a paid service, with registration from 5 to 31 January 2017. Details are available online at UCEQA and RCEQA web- sites.

Preparatory work took place on schedule, according to which testing 2017 will take place from May 23 to June 16 (regular session) and July 3-12 (supplementary session). Details of the testing schedule for all subjects can be found at the UCEQA web-site.

Area One: Strengthening institutional capacity of testing system elements and components

Restoring UCEQA operational IT systems capacity for SET. During the reporting period UCEQA and Infopulse LLC completed the following tasks: • All functionality needed for SET 2016 was successfully restored and modernized • Software to build network of testing centers • Software to process SET materials and results • Software to support appeal process • All secondary tasks are completed • Improve personal web-cabinet (registration status, invitations to SET tests, result presentation, data validation, captcha, etc.) • Extend software functionality for SET process to prisons • Develop software for opinion poll of SET test-takers • Develop scripts for regular task automation • Continuous improvement of software quality after completion of testing • Enhanced security and reliability of systems currently used: • Introduced secure connection for SET test-takers by using authorized SSL- certificate.

USETI Alliance Phase II Quarterly Report Y4Q4 plus NCE (October 2016 – February 2017) • Introduced an additional server for the website and set up a fault-tolerance high availability cluster • Implemented a digital signature in the module for presenting SET results • Performed system penetration test • Performed government assessment of the Comprehensive Data Protection System and its components for the Ukrainian Center • Developed and updated software for SET in upcoming years • Prepared systems for Master degrees • Created and prioritized list of improvements • Develop and tested agreed-upon improvements • Migrated System to new DBMS (Oracle) • Supported and fixed bugs • Documented project results • Finalized systems security improvements • Performed government assessment of the Comprehensive Data Protection System and its components for the Regional Center and Web-site. • Implemented measures related to Comprehensive Data Protection System and updated operations and maintenance documentation accordingly.

Technical Assistance delivered – Strengthening institutional capacity of testing system elements and components. In October 2016, per recommendation of USETI Alliance Consultant, USETI Alliance ordered and purchased the 2014 edition of The Standards for Educational and Psychological Testing (“Testing Standards”) for UCEQA specialists. The Testing Standards is a product of the American Educational Research Association, the American Psychological Association (APA), and the National Council on Measurement in Education (NCME). Published collaboratively by the three organizations since 1966, it represents the gold standard in guidance on testing in the United States and in many other countries.

Between October 10 and November 11 UCEQA external item-writers and reviewers led by UCEQA subject specialists and USETI international consultants gathered for their regular annual item writing workshops in Mathematics, Physics, Biology, Chemistry, German, English, French and Spanish. During the workshops the following topics were covered: - Tests-2016 analysis; - Examples of various new interesting multiple-choice items that respond to various difficulty levels; - How to design items for best discrimination; - How to develop so-called “cluster” tasks; - Item review based on psychometric characteristics derived from item piloting; Nearly 100 participants including UCEQA and RCEQA staff, item writers and reviewers from all over attended the workshops.

On November 7-16, USETI Alliance consultants worked on the following: • Continued psychometric training for UCEQA Psychometrics Department staff; • Worked with UCEQA psychometric staff to further improve item and test analysis programs used by UCEQA; • Worked with UCEQA Item Bank, Content and Psychometric departments on test equating.

Technical Assistance delivered – Support of MA Program on Educational Measurement. On October 17 USETI international consultant gave a guest-lecture to students at Drahomanov National Pedagogical University pursuing their MA in Educational Measurement. The topic of the lecture was “Education quality monitoring in secondary and higher school in the US.”

USETI Alliance Phase II Quarterly Report Y4Q4 plus NCE (October 2016 – February 2017) Area Two: Establishing legislative and regulatory policy foundations for fair and objective university admissions policy and practices

Build collaborative capacity for quality control by Policy Development Stakeholders. Within the reporting period the USETI Alliance in collaboration with the Institute of Higher Education of the National Academy of Pedagogical Sciences of Ukraine and Union of Rectors finished studying SET’s quality. Currently experts are collecting data for validity study of SET 2012-2014.

Area Three: Transforming support for testing into a proactive contemporary public expectation

XXV, XXVI USETI Alliance Partners meetings On December 19, American Councils for International Education hosted the XXV USETI Alliance Partners Meeting at its office. The following topics were discussed: the results of a recently conducted (November 2016) national poll on public perceptions of standardized external testing; a documentary familiarizing numerous stakeholders and broad audiences with the philosophy of standardized testing, and the procedures developed and carried on to create quality tests, administer nationwide testing and administer other aspects of UCEQA; and priorities for UCEQA in 2017: peculiarities of administering SET; PISA 2018; education quality monitoring.

On February 28, 2017 the XXVI USETI Alliance Partners Meeting was held in the premises of its founding Partner – International Renaissance Foundation. The following topics were discussed at the meeting: Validity Study (based on the results of external testing in 2012-2014); a Road Map “Education Assessment in Ukraine”.

IV National Forum: “Business and Universities. Skills 2030: Is Education Providing What’s Needed?” On October 20-21, 2016 the IV National Forum: ‘Business and universities. Skills 2030: Is Education Providing What’s Needed?” was held in . The event was organized by the Corporate Social Responsibility Development Center and supported by USETI Alliance.

UCEQA Strategic Planning. Within the reporting period USETI Alliance partner organization Pro.Mova organized and conducted two strategic planning sessions on educational assessment and two sessions on UCEQA strategic development. All events were held at the premises of American Councils for International Education in Kyiv. In the sessions conducted December 11 and December 20 UCEQA senior managers and directors of nine RCEQSa participated. The sessions were moderated by Aleksandra Baklanova, Partner, Pro.Mova. The work of the participants of these four sessions resulted in drafting of the Strategy for Secondary Education Quality Assurance. The main change that the strategy recommends is transition from control/assessment of education quality to education quality assurance. To implement this change the system has to transform into the one which is capable of assuring the quality of education. The goals of the strategy were to effect implementation of the system for constant and consistent enhancement of education quality and influence the forming of public trust in the system of education in Ukraine

International Conference “Ten Years of Testing: Ways Forward for Educational Assessments in Ukraine.” On November 23, the Ukrainian Center for Education Quality Assessment with the support of USETI Alliance conducted a one-day conference “Ten Years of Testing: Ways Forward for Educational Assessments in Ukraine.”

The event was held at the Museum of the National University “Kyiv-Mohyla Academy” (NAUKMA) in Kyiv.

USETI Alliance Phase II Quarterly Report Y4Q4 plus NCE (October 2016 – February 2017) The key message of the conference was: Standardized external testing (SET) has helped reduce bribery and corruption in Ukrainian Higher Educational Institution admissions (particularly at the undergraduate level). Over sixty representatives of the Ministry of Education and Science of Ukraine, leading universities of Ukraine, UCEQA and the nine Regional CEQAs and their deputies, National Academy of Pedagogical Sciences, civic organizations participated in the event.

Media Monitoring. According to the InfoStream© monitoring service, within the reporting period (October 2016 – February 2017) 3049 articles on education and external independent appeared in Ukrainian media — 626 articles in October 2016, 662 in November 2016, 492 in December 2016, 634 in January 2017, and 635 in February 2017.

Extend reporting on testing and HEI admissions processes and issues through electronic listserv and social media. During the reporting period, three bulletins were published – for October, November and December 2016. The USETI Alliance bulletins illustrate the most important educational developments within the USETI Alliance mandate and are available at www.useti.org.ua.

Extend advocacy outreach to employers to promote fair access to higher education and testing-based HEI admissions. In December 2016 the USETI Alliance developed and distributed a quarterly Education & Business bulletin. Through this bulletin Ukrainian employers receive quarterly updates on higher education access, quality and its relevance to the needs and trends in the workplace.

1.3 Variances from Work-plan none

1.4 Reports and Deliverables Completed and Submitted to USAID Reports ▪ Updates on Project Activities sent to USAID.

2 Activities Completed See attached chart of activities completed.

USETI Alliance Phase II Quarterly Report Y4Q4 plus NCE (October 2016 – February 2017) USETI Legacy Alliance Year 4/Q4 plus NCE (October 2016 - February 2017) - Annex 1: Status of Activities of Tasks

Activities End-of-project target Progress Area 1: Strengthening institutional capacity of testing system elements and components Activity 1.1: Secure UCEQA capacity for item banking, test development and test structure/content quality assurance UCEQA psychometricians trained in using psychometric During his visit in November Consultant Zarko Vukmirovic worked with 1.1a Train UCEQA and RCEQA specialists in applied psychometrics methods and applications; increased psychometric UCEQA psychometrics Department. capacity UCEQA skilled at carrying out pilot tests within live tests USETI Alliance consultant Algirdas Zabulionis in July and September 1.1b Scale up UCEQA item piloting system worked with UCEQA on test items piloting during live tests.

IB was blocked due to seizure of the CIB by law UCEQA calibrated item bank developed and works 1.1c Roll out UCEQA calibrated item bank enforcement (PGO investigators). properly During his visit in November Mike Fast worked with UCEQA on MFL tests UCEQA introduces audio component into the MFL tests development. 1.1d Enhance Modern Foreign Language subject tests Audio component of MFL tests is announced for 2018 testing year.

During their visits in October-November 2016 USETI Alliance consultants UCEQA skilled in using the advanced item development 1.1e Advise UCEQA on advanced item development techniques Mike Fast and Mark Zelman supported UCEQA with advanced item techniques development techniques. UCEQA received recommendations for special need test 1.1f Diversify UCEQA special needs testing capacity Completed in Y3. development based on international experience

UCEQA uses international experience to develop SET 1.1g Develop strategy for management of test development policy USETI Alliance developed and published the Roadmap. (Per a firm UCEQA plan for 2017) Activity 1.2: Establish UCEQA capacity for ancillary testing mechanisms and applications The General Ability Test piloted on the experimental USETI Alliance supports efforts of Fair Justice USAID Project in 1.2a Support rollout of the General Ability Test for HEI Admissions basis development of master level test in Law which includes ability test part.

During their visits in October-November USETI Alliance consultants Mike UCEQA is capable to develop multi-level test on 1.2b Develop multi-level subject tests for HEI Admissions Fast and Mark Zelman supported UCEQA subject groups with multi-level advanced level tests on advanced level. HEIs begin using a standardized toolkit for bachelor Standardized admission for master degree in law defined as obligatory for 1.2c Conceptualize Summative Assessment tools for HEIs students in additional disciplines as an academic all law schools. experiment Activity 1.3 Enhance UCEQA staff development capacity Support of MA Program on Educational Measurement at 1.3a Synchronize UCEQA in-house training capacity with academic program in educational One lecture was conducted by USETI Alliance Consultant Mark Zelman the Dragomanov National Pedagogical University measurement during his mission to Kyiv in October.

UCEQA staff members introduced to American and 1.3b Enable UCEQA staff development through international professional exchanges Completed. European best practices in educational measurement/management UCEQA agreed on the need to create Internal Quality Assurance UCEQA responds to new challenges in proper ways 1.3c UCEQA Internal Quality Assurance Working Group WGUCEQA within its structure. Activity 1.4 Enable Kherson and Donetsk RCEQAs to work equipped USETI Legacy Alliance Year 4/Q4 plus NCE (October 2016 - February 2017) - Annex 1: Status of Activities of Tasks

Activities End-of-project target Progress Kherson and Donetsk RCEQA work in efficient way like 1.4a Enable Kherson and Donetsk RCEQAs to work equipped Completed. other regional centers

Activity 1.5 Restoring UCEQA operational IT systems capacity for SET

In January 2017 UCEQA confirmed completion of IT system recovery by UCEQA operational IT systems capacity for SET work 1.5a Enable UCEQA operational IT systems capacity for SET . properly and securely Area 2: Establishing legislative and regulatory policy foundations for fair and objective university admission policy and practices Activity 2.1 Reform legislative foundations for fair access to higher education New Higher Education Law passed securing testing and During October-December of Y4 USETI Alliance supported implementation 2.1a Facilitate new Law on Higher Education HEI autonomy of Law on HE.

New Law on Education passed securing educational 2.1b Facilitate ancillary legislation Completed in Y4Q3. quality monitoring system Activity 2.2 Solidify reformed regulatory policy foundations for fair access to higher education Users of admission reform have body to serve as venue 2.2a Conditions on Admission Advisory Platform Completed in Y4Q3. to influence reform 2.2b Strengthen Cooperation between MOES Public Council, HEIs, experts and Civil Committee to assist MOES in development of education Completed. Society quality monitoring exists Activity 2.3 Institutionalize key practices in implementing reformed legislative and regulatory policy foundations for fair access to higher education Reliability and validity is ensured through external review 2.3a Build collaborative capacity for quality control by Policy Development Stakeholders URU along with USETI Alliance developed and published validity study.

2.3b Facilitate implementation and compliance with new or reformed legislative and Bylaws regulating equal access to higher education are on demand basis. regulatory provisions accepted by state authorities Area 3: Transforming support for testing into a proactive contemporary public expectation Activity 3.1 Broaden public outreach competencies of teachers, testing organizers, and regional education officials on testing and HEI admissions issues Teachers and administrators in secondary schools use 3.1a Orient teachers as advocates for testing culture and fair access to higher education Completed. basics of educational assessment in classroom 3.1b Introduce UCEQA and RCEQA staff and regional education officials to advocacy During reporting period pro.mova supported UCEQA strategy UCEQA, RCEQAs and regional officials respond to new techniques in public outreach development. challenges in effectively Ukrainian employers are familiar with the latest 3.1c Extend advocacy outreach to employers to promote fair access to higher education In Y4Q4 the 15th issue of Education&Business Bulletin was developed and development of reform to reach fair access to higher and testing-based HEI admissions electronically distributed. education 3.1d Support secondary analysis of standardized external testing data focusing on the UCEQA and RCEQAs develop secondary analysis Completed. local level (Support secondary analysis of SET data) studies of standardized external testing. Activity 3.2 Institutionalize monitoring of and reporting on testing and HEI admissions processes and issues Monitoring of SET administration and HEIs admission 3.2a Further develop public monitoring of test administration and university admissions Completed in Y4Q3. positively influence public support 3.2b Integrate institutional non-GOU monitoring of the HEI admissions processes with Integrate institutional non-GOU monitoring of the HEI Completed in Y4Q3. GOU databases admissions processes with GOU databases In Y4Q4 plus NCE four issues (October, November, December and Ukrainian educators are familiar with the latest 3.2c Extend reporting on testing and HEI admissions processes and issues through January) of USETI Alliance bulletin were developed and electronically development of reform to reach fair access to higher electronic listserv and social media distributed. education USETI Legacy Alliance Year 4/Q4 plus NCE (October 2016 - February 2017) - Annex 1: Status of Activities of Tasks

Activities End-of-project target Progress Activity 3.3 Monitor and integrate public and expert opinion on testing and fair access to higher education Media monitoring October 2016 - February 2017 was developed and Stakeholders receive media monitoring regularly 3.3.a Monitor media throughout project implementation distributed. National Polls on external testing and admissions public 3.3.b Conduct annual poll on testing and access to higher education in Ukraine Conducted in Q4/Y4. opinion Recommendations on implementing international standards to Ukrainian HE system developed according 3.3.c Conduct IV International Educational Assessment Conference Completed. to IV International Conference EA-2013

Wide availability of media products leading to public 3.3 d Provide direct messaging and as-needed interventions to sustain and build public There is a need to keep public informed about circumstances of the GP understanding of link between testing and anti-corruption support investigation.

Activity 3.4 Building the Alliance In December 2016 and February 2017, two USETI Alliance partners Alliance functioning is enhanced and membership and 3.4a Coordination meetings and outreach meetings were conducted. contributions are expanded Activity 3.5 Establish communications strategy and execution norms UCEQA agreed on the need to establish Supervisory Council to secure its UCEQA capacity to prevent and respond to information 3.5a Establish UCEQA Supervisory (Public) Council activities and strategic challenges is enhanced SETI Legacy Alliance Phase II Performance Monitoring and Evaluation Chart

Year Year Year Target Year 4 Expected Methodology for Measurement 1 2 3 End of Performance Indicator Result Q1 Q2 Q3 Q4 Y4 Notes Data Source Collection Responsible Baseline Area 1: Strengthening institutional capacity of testing system elements and components Activity 1.1 Secure UCEQA capacity for item banking, test development and test structure/content quality assurance a. UCEQA staff use current Workshops and quarterly Area 1 leader & psychometric methods and # of workshops in consultant reports international - 2 6 10 11 12 12 13 13 Conducted in November 2017. applications; demonstrate increased psychometrics consultants psychometric capacity Consultant quarterly Area 1 leader & b. UCEQA skilled at carrying out UCEQA uses live tests for UCEQA has insufficient skills to carry out reports international No No No No No No No No Yes pilot tests within live tests carrying out pilot tests pilot tests consultants Item Bank software procured Procurement of Annually UCEQA does not have software to optimize Area 1 leader & No No No No No No No No Yes c. UCEQA calibrated item bank and installed software item bank international developed and works properly # of workshops in usage of IB Workshops and Quarterly UCEQA will need to be trained in newly consultants - - - 2 2 2 2 3 software consultant reports developed software # of workshops MFL item & Training and Semi- - 2 4 7 8 8 8 9 9 test development consultant reports annually Area 1 leader & Mike Fast d. UCEQA introduces audio Report Annually international Audio Component is scheduled for 2018 component into the MFL tests UCEQA introduces audio consultants No No No No No No No No Yes testing year. component into the MFL tests Consultant semi-annual Area 1 leader & UCEQA was trained in item writing and test e. Advanced item development # of workshops in advanced reports international - 1 4 8 8 8 9 10 10 development during the USETIO Alliance techniques are used in SET item development techniques consultants international consultants visit in September f. UCEQA processes Training and semi-annual Tests for more categories of Area 1 leader & recommendations for special needs consultant reports UCEQA needs to extend availability for tests takers with special needs international No No No Yes Yes Yes Yes Yes Yes test development based on special needs categories to take SET are offered consultants international experience Consultant Annually g. UCEQA uses international reports Area 1 leader &

experience to develop SET (Per a Roadmap developed international - - - - No process Yes Yes Developed in January 2017.

firm UCEQA plan for 2017) consultants process In In

Activity 1.2 Establish UCEQA capacity for ancillary testing mechanisms and applications Reports on Area 1 leader & # of workshops, conferences, roundtables, Quarterly international - 2 8 15 15 15 15 16 23 RTs, etc. conference, consultants workshops USETI Alliance supports efforts of Fair Justice USAID Project in development of master level a. General Ability Test piloted on Reports test in Law which includes ability test part. the experimental basis Area 1 leader & At the experimental basis HEIs In 2017 standardized testing is obligatory for Semi- international use GAT for admission No No No No No Yes Yes Yes master degree in law. annually consultants purposes Area 2 leader process In Year Year Year Target Year 4 Expected Methodology for Measurement 1 2 3 End of Performance Indicator Result Q1 Q2 Q3 Q4 Y4 Notes Data Source Collection Responsible Baseline Consultant reports During their visits in October-November b. UCEQA demonstrates capability Area 1 leader & USETI Alliance consultants Mike Fast and # of workshops on multi-level Semi- to produce an advanced, multi-level international - 1 3 5 5 5 6 7 7 Mark Zelman supported UCEQA subject test development annually test consultants groups with multi-level tests on advanced level. ELPEG meetings, Such an instrument will be a brilliant addition Semi- Concept developed Consultants - 1 1 1 1 1 1 1 1 to anticorruption efforts within USETI annually Area 1 leader & c. Selected HEI(s) pilot a reports Alliance framework international standardized toolkit for bachelor a standardized instrument for Report consultants degree students bachelor students as an Standardized admission for master degree in

Annually Area 2 Leader No No No No No In Yes Yes Yes academic experiment is used law defined as obligatory for all law schools. by selected HEI(s) process Activity 1.3 Enhance UCEQA staff development capacity a. MA Program in Educational UCEQA/ Area 1 leader & One lecture was conducted by USETI Alliance Measurement and Administration at # of guest lectures by USETI Semi- Consultants international - 10 16 20 22 24 24 25 28 Consultant Mark Zelman during his mission to the strengthened and produces more Consultants and UCEQA staff annually reports consultants Kyiv in October. qualified specialists b. UCEQA staff members use Area 1 leader & American and European best Study tour trips Semi- # of Study Tours international - 1 7 8 8 8 8 8 8 Completed. practices in educational reports annually consultants measurement/management # of working meetings of the UCEQA agreed on the need to create Internal c. UCEQA responds to new DPD and Area UCEQA Internal Quality Meetings reports Quarterly - - - - 1 1 1 1 8 Quality Assurance WGUCEQA within its challenges in proper ways managers Assurance WG structure. Activity 1.4 Enable Kherson and Donetsk RCEQAs to work equipped USETI a. Kherson and Donetsk RCEQA Kherson and Donetsk RCEQAs Alliance/Donetsk Semi- works in efficient way like other supported with new computer DPD - - - Yes Yes Yes Yes Yes Yes Completed. and Kherson annually regional centers and furniture equipment RCEQAs reports Activity 1.5 Restoring UCEQA operational IT systems capacity for SET a. UCEQA operational IT systems In January 2017 UCEQA confirmed Reports on work done by Vendor, UCEQA, for SET works properly and Quarterly DPD - - - - 3 5 5 6 6 completion of IT system recovery by Infopulse modules development IT CCN reports securely Ukraine. Area 2: Establishing legislative and regulatory policy foundations for fair and objective university admission policy practices Activity 2.1 Facilitate new Law on Higher Education Stage 1 – Interested groups propose that legislation is needed on this issue;

Stage 2 – Issue is introduced in the relevant a. New Higher Education Law Legislation securing testing and legislative committee; passed securing testing and HEI VR voting report Annually Area 2 leader HEI autonomy is adopted Stage 3 – Legislation is drafted by relevant autonomy 1 Stage 3 Stage 4 Stage 4 Stage 4 Stage 4 Stage 4 Stage 4 Stage 4 Stage committee; Stage 4 – Legislation is debated by the legislature.

2

Year Year Year Target Year 4 Expected Methodology for Measurement 1 2 3 End of Performance Indicator Result Q1 Q2 Q3 Q4 Y4 Notes Data Source Collection Responsible Baseline Stage 1 – Interested groups propose that legislation is needed on this issue;

Stage 2 – Issue is introduced in the relevant b. New Law on Education passed Legislation securing

legislative committee; securing educational quality educational quality monitoring VR voting report Annually Area 2 leader - Stage 3 – Legislation is drafted by relevant monitoring system system is adopted 1 Stage 3 Stage 3 Stage 3 Stage 3 Stage 3 Stage 3 Stage 4 Stage committee; Stage 4 – Legislation is debated by the legislature. Activity 2.2 Solidify reformed regulatory policy foundations for fair access to higher education a . Committee to influence admissions reform established and Committee to influence Users of admissions reform do not have a body Minutes of functioning. Users of admissions admissions reform serves as a Semi- to where they can become informed of best- Committee’s Area 2 leader No No Yes Yes Yes Yes Yes Yes Yes reform have body to serve as venue venue to influence admission annually practices in testing in admissions, meeting to influence reform and apply best- reform and can influence such standards practices Committee to assist MOES in Minutes of b. Education quality monitoring is development of education Semi- There is a need to assist MOES with Committee’s Area 2 leader No No Yes Yes Yes Yes Yes Yes Yes secured within regulatory basis quality monitoring makes annually development of education quality monitoring meeting recommendations regularly Activity 2.3 Institutionalize key practices in implementing reformed legislative and regulatory policy foundations for fair access to higher education

USETI Alliance along with Union of Rectors a. Reliability and validity is ensured URU & Area 2 # of validity studies Study Annually 2 2 4 4 4 5 5 and NAPSU developed and published the next through external review leader validity study in Y4Q4. In process In process In b. Covered persons and institutions Minutes of # of recommendations on new comply with legislative and ELPEG meetings, Semi- On Ukrainian legislation on education needs to be regulations accepting equal Area 2 leader - OD OD OD OD OD OD OD regulatory provisions on equal RTs, similar annually demand improved access to higher education access to higher education events Area 3: Transforming support for testing into a proactive contemporary public expectation Activity 3.1 Broaden public outreach competencies of teachers, testing organizers, and regional education officials on testing and HEI admissions issues a. Teachers and administrators in # of seminars with teachers and secondary schools use basics of Area 1 leader, administrators of secondary Seminars reports Quarterly - 12 14 18 18 18 18 18 18 Completed. educational assessment in DPD school classroom b. UCEQA, RCEQAs and regional Semi- officials respond to new challenges # of seminars Seminars reports Pro.Mova, DPD 1 - 1 3 3 3 3 3 3 Completed. annually in a quick and proper manner c. Ukrainian employers are familiar with the latest development of # of Education&Business Alliance Dev’t Bulletin Quarterly - 3 7 11 12 13 14 15 15 The Bulletin is published in December 2016. reform to reach fair access to higher Bulletin Advisor education

3

Year Year Year Target Year 4 Expected Methodology for Measurement 1 2 3 End of Performance Indicator Result Q1 Q2 Q3 Q4 Y4 Notes Data Source Collection Responsible Baseline d. UCEQA and RCEQAs develop Semi- Area 1 leader & secondary analysis studies of # of studies Study - - 2 4 4 4 4 4 4 Completed. annually UCEQA standardized external testing

Activity 3.2 Further develop public monitoring of test administration and university admissions # of trainings for monitoring of Training report Annually OPORA, DPD - 1 2 3 3 4 4 4 4 Completed. test administrations Public monitoring of test administration procedures Monitoring reports Annually OPORA - 1 2 3 3 4 4 4 4 Completed. a. Public support of testing conducted properly increases due to monitoring of test administration and HEI admissions # of trainings for Admissions Training reports Annually OPORA, IRF - 1 2 3 3 3 4 4 4 Completed. Committee monitoring

Admission campaigns’ Monitoring reports Annually OPORA - 1 2 3 3 3 4 4 4 Completed. monitoring conducted properly # of data actualizations with Konkurs SSDBE per day during HEIs Annually CEP, Area 2 leader - 1 4 6 6 6 6 6 6 Completed. administrator reports admission campaign b. Public is informed of real-time Functionality of Konkurs System status of the admission campaign allows test-taker to recommend through the Konkurs System Konkurs the optimal HEIs/specialization Annually CEP, Area 2 leader No No No Yes Yes Yes Yes Yes Yes Completed. administrator reports to enter according to their priorities identified Four bulletins (October, November, December c. Ukrainian educators are familiar # of USETI Alliance Bulletin Bulletin Quarterly Area 3 leader - 7 19 31 34 37 40 44 43 2016 and January 2017) published in the with the latest development of reporting period reform to reach fair access to higher TIMO Foundation - Journal for informing stakeholders of education # of Timo published Timo journals Monthly Publishing House - 12 24 36 39 42 45 50 48 innovations in testing and admissions reform “Fakt” Activity 3.3 Monitor and integrate public and expert opinion on testing and fair access to higher education Reports on media a. Stakeholders receive media # of media monitoring and Stakeholders receive media monitoring monitoring & Quarterly Area 3 leader - 3 7 11 12 13 14 15 15 monitoring regularly analysis regularly Analysis b. Educational professionals, policymakers and other stakeholders # of National Polls conducted informed about public support of Reports Annually DIF - 1 2 3 3 3 3 4 4 Conducted in Y4Q4. and analyzed external testing and admission reforms c. Recommendations on The Conferences were conducted implementing international standards Area 1 leader, and recommendations on in Ukrainian higher education Conferences reports Annually UCEQA, URU, - 1 1 2 2 2 2 2 2 Completed implementing standards were developed according to IV and V NAPSU introduced International Conference on

4

Year Year Year Target Year 4 Expected Methodology for Measurement 1 2 3 End of Performance Indicator Result Q1 Q2 Q3 Q4 Y4 Notes Data Source Collection Responsible Baseline Educational Assessment d. Wide availability of media Potential damage from the products leading to public criminal investigation and likely Public information Quarterly Area 3 leader ------1 3 Video PSA developed and published in Y4Q4. understanding of link court do not effect the public campaigns reports between testing and anti-corruption support of testing Activity 3.4 Building the Alliance a. Alliance function is enhanced and Semi- Alliance Dev’t Two Partners meetings conducted in December membership and contributions are # of meetings Meetings reports - 3 7 11 12 12 14 16 15 annually Advisor, PD 2016 and February 2017. expanded Activity 3.5 Establish communications strategy and execution norms a. UCEQA capacity to prevent and # of meetings of UCEQA Semi- UCEQA agreed on the need to establish respond to information and strategic Meetings reports PDP, Area leaders ------4 Supervisory (Public) Council annually Supervisory Council to secure its activities challenges is enhanced

5