UKRAINIAN STANDARDIZED EXTERNAL TESTING INITIATIVE (USETI) LEGACY ALLIANCE PHASE II FEBRUARY 1, 2013– DECEMBER 31, 2016 GLOBAL DEVELOPMENT ALLIANCE QUARTERLY REPORT: Y4Q4 PLUS TWO MONTHS OF NO-COST –EXTENSION PERIOD (OCTOBER 1, 2016 – FEBRUARY 28, 2017) SUBMITTED BY: AMERICAN COUNCILS FOR INTERNATIONAL EDUCATION _________________________________________ GDA TABLE OF CONTENTS: PAGE SUMMARY OF ACTIVITIES…………………………………2 ACTIVITIES COMPLETED…………………………………..5 ANNEXES: 1. STATUS OF ACTIVITIES OF TASKS 2. PERFORMANCE MONITORING AND EVALUATION CHART 3. USETI ALLIANCE PARTNER LEVERAGE CONTRIBUTION USETI Alliance Phase II Quarterly Report Y4Q4 plus NCE (October 2016 – February 2017) 1. Summary of Activities The USETI Legacy Alliance Phase II is a Global Development Alliance funded by the U.S. Agency for International Development (USAID) under a cooperative agreement. American Councils for International Education is the Cooperating Partner and lead implementer, collaborating with USAID and 19 other founding partners. The project builds on accomplishments of previous U.S. Government assistance to: build a sustainable, strengthened UCEQA system more capable of independently and transparently developing and implementing secure tests that meet international standards; secure a legislative and regulatory policy basis for testing and higher education admissions; and build a sustainable high level of public support for equal access to Ukrainian higher education. Throughout the reporting period (October 2016 through February 2017 which includes a two month-long no cost extension period (also referred to in the report as Q16) USETI Legacy Alliance Phase II activities were focused on supporting UCEQA with test development, UCEQA IT System recovery, conducting an international conference, completing a validity study, developing a Road Map with UCEQA and executing a number of other policy development-related activities. Issues 1.1. Significant Events Preparations for SET. In 2016 the registration for SET will take place from February 6 to March 17, 2017. As in previous years, SET participants will have the opportunity to gauge their readiness, familiarize themselves with procedures for testing and perform tasks as close as possible to the problems used in the testing session during the trial SET. Trial testing is a paid service, with registration from 5 to 31 January 2017. Details are available online at UCEQA and RCEQA web- sites. Preparatory work took place on schedule, according to which testing 2017 will take place from May 23 to June 16 (regular session) and July 3-12 (supplementary session). Details of the testing schedule for all subjects can be found at the UCEQA web-site. Area One: Strengthening institutional capacity of testing system elements and components Restoring UCEQA operational IT systems capacity for SET. During the reporting period UCEQA and Infopulse LLC completed the following tasks: • All functionality needed for SET 2016 was successfully restored and modernized • Software to build network of testing centers • Software to process SET materials and results • Software to support appeal process • All secondary tasks are completed • Improve personal web-cabinet (registration status, invitations to SET tests, result presentation, data validation, captcha, etc.) • Extend software functionality for SET process to prisons • Develop software for opinion poll of SET test-takers • Develop scripts for regular task automation • Continuous improvement of software quality after completion of testing • Enhanced security and reliability of systems currently used: • Introduced secure connection for SET test-takers by using authorized SSL- certificate. USETI Alliance Phase II Quarterly Report Y4Q4 plus NCE (October 2016 – February 2017) • Introduced an additional server for the website and set up a fault-tolerance high availability cluster • Implemented a digital signature in the module for presenting SET results • Performed system penetration test • Performed government assessment of the Comprehensive Data Protection System and its components for the Ukrainian Center • Developed and updated software for SET in upcoming years • Prepared systems for Master degrees • Created and prioritized list of improvements • Develop and tested agreed-upon improvements • Migrated System to new DBMS (Oracle) • Supported and fixed bugs • Documented project results • Finalized systems security improvements • Performed government assessment of the Comprehensive Data Protection System and its components for the Regional Center and Web-site. • Implemented measures related to Comprehensive Data Protection System and updated operations and maintenance documentation accordingly. Technical Assistance delivered – Strengthening institutional capacity of testing system elements and components. In October 2016, per recommendation of USETI Alliance Consultant, USETI Alliance ordered and purchased the 2014 edition of The Standards for Educational and Psychological Testing (“Testing Standards”) for UCEQA specialists. The Testing Standards is a product of the American Educational Research Association, the American Psychological Association (APA), and the National Council on Measurement in Education (NCME). Published collaboratively by the three organizations since 1966, it represents the gold standard in guidance on testing in the United States and in many other countries. Between October 10 and November 11 UCEQA external item-writers and reviewers led by UCEQA subject specialists and USETI international consultants gathered for their regular annual item writing workshops in Mathematics, Physics, Biology, Chemistry, German, English, French and Spanish. During the workshops the following topics were covered: - Tests-2016 analysis; - Examples of various new interesting multiple-choice items that respond to various difficulty levels; - How to design items for best discrimination; - How to develop so-called “cluster” tasks; - Item review based on psychometric characteristics derived from item piloting; Nearly 100 participants including UCEQA and RCEQA staff, item writers and reviewers from all over Ukraine attended the workshops. On November 7-16, USETI Alliance consultants worked on the following: • Continued psychometric training for UCEQA Psychometrics Department staff; • Worked with UCEQA psychometric staff to further improve item and test analysis programs used by UCEQA; • Worked with UCEQA Item Bank, Content and Psychometric departments on test equating. Technical Assistance delivered – Support of MA Program on Educational Measurement. On October 17 USETI international consultant gave a guest-lecture to students at Drahomanov National Pedagogical University pursuing their MA in Educational Measurement. The topic of the lecture was “Education quality monitoring in secondary and higher school in the US.” USETI Alliance Phase II Quarterly Report Y4Q4 plus NCE (October 2016 – February 2017) Area Two: Establishing legislative and regulatory policy foundations for fair and objective university admissions policy and practices Build collaborative capacity for quality control by Policy Development Stakeholders. Within the reporting period the USETI Alliance in collaboration with the Institute of Higher Education of the National Academy of Pedagogical Sciences of Ukraine and Union of Rectors finished studying SET’s quality. Currently experts are collecting data for validity study of SET 2012-2014. Area Three: Transforming support for testing into a proactive contemporary public expectation XXV, XXVI USETI Alliance Partners meetings On December 19, American Councils for International Education hosted the XXV USETI Alliance Partners Meeting at its office. The following topics were discussed: the results of a recently conducted (November 2016) national poll on public perceptions of standardized external testing; a documentary familiarizing numerous stakeholders and broad audiences with the philosophy of standardized testing, and the procedures developed and carried on to create quality tests, administer nationwide testing and administer other aspects of UCEQA; and priorities for UCEQA in 2017: peculiarities of administering SET; PISA 2018; education quality monitoring. On February 28, 2017 the XXVI USETI Alliance Partners Meeting was held in the premises of its founding Partner – International Renaissance Foundation. The following topics were discussed at the meeting: Validity Study (based on the results of external testing in 2012-2014); a Road Map “Education Assessment in Ukraine”. IV National Forum: “Business and Universities. Skills 2030: Is Education Providing What’s Needed?” On October 20-21, 2016 the IV National Forum: ‘Business and universities. Skills 2030: Is Education Providing What’s Needed?” was held in Kyiv. The event was organized by the Corporate Social Responsibility Development Center and supported by USETI Alliance. UCEQA Strategic Planning. Within the reporting period USETI Alliance partner organization Pro.Mova organized and conducted two strategic planning sessions on educational assessment and two sessions on UCEQA strategic development. All events were held at the premises of American Councils for International Education in Kyiv. In the sessions conducted December 11 and December 20 UCEQA senior managers and directors of nine RCEQSa participated. The sessions were moderated by Aleksandra Baklanova, Partner, Pro.Mova. The work of the participants of these four sessions resulted in drafting of the Strategy for Secondary Education Quality Assurance. The main change