EVALUATION OF THE RESPONSIBLE ACCOUNTABLE DEMOCRATIC ASSEMBLY (RADA) PROGRAM PO No. AID-121-O-17-00040 DISCLAIMER: This is an external assessment. The view expressed in this document are the authors’ and do not necessarily reflect the view of the United States Agency for International Development or the United States Government. Submitted to: USAID/Ukraine Prepared by: John Lis, Team Lead Taras Kuzio, International Governance Expert Andriy Meleshevych, Local Governance Expert Oksana Vynnychuk, Logistician Submitted February 2018 Contractor: Democracy International, Inc. 7600 Wisconsin Avenue, Suite 1010 Bethesda, MD 20814 Tel: 301-961-1660 www.democracyinternational.com I EXECUTIVE SUMMARY i EVALUATION PURPOSE AND QUESTIONS 1 PROGRAM BACKGROUND AND CONTEXT 2 EVALUATION METHODS AND LIMITATIONS 4 FINDINGS AND CONCLUSIONS 8 LESSONS LEARNED 33 ANNEX A: EXECUTIVE SUMMARY IN UKRAINIAN 35 ANNEX B: EVALUATION SOW 43 ANNEX C: EVALUATION TEAM 58 ANNEX D: EVALUATION WORK PLAN 59 ANNEX E: CONFLICT OF INTEREST STATEMENTS 94 ANNEX F: LIST OF DOCUMENTS REVIEWED 95 ANNEX G: LISTS OF KEY INFORMANTS, FOCUS GROUP DISCUSSANTS AND SURVEY RESPONDENTS 97 ANNEX H: DATA COLLECTION TOOLS 107 ANNEX I: SURVEY RESULTS 116 ANNEX J: FOCUS GROUP DISCUSSION SUMMARIES 127 ANNEX K: TABLE OF EVALUATION FINDINGS AND CONCLUSIONS 131 ANNEX L: MS POWERPOINT PRESENTATION OF EVALUATION DESIGN, FINDINGS AND CONCLUSIONS 134 ANNEX M: RADA PROGRAM ACTIVITIES 135 EXECUTIVE SUMMARY EVALUATION PURPOSE AND QUESTIONS The objective of this final performance evaluation is to assess the relevance and effectiveness of the RADA Program in advancing parliamentary reform in Ukraine and, more specifically, in improving public representation in the legislative process and strengthening independent legislative oversight of the executive branch. The Evaluation Questions are: 1. Which RADA Program activities do RADA Program stakeholders perceive to have been the most beneficial for improving public representation in the legislative process and why? 2. Which RADA Program activities do RADA Program stakeholders perceive to have been the most beneficial for strengthening independent oversight of the legislature over the executive branch and why? 3. What changes in public representation in the legislative process do RADA Program stakeholders perceive to be the result, in whole or in part, of the RADA Program’s work? 4. What changes in independent oversight of the legislature over the executive branch do RADA Program stakeholders perceive to be the result, in whole or in part, of the RADA Program’s work? 5. How well did the RADA Program respond to opportunities to leverage resources and advance relevant parliamentary reforms through collaboration with other USAID and non-USAID development assistance programs? 6. How relevant was the RADA Program in advancing parliamentary reform in Ukraine? 7. How effective was the RADA Program in advancing parliamentary reform in Ukraine? 8. How well did the RADA Program promote gender equality in its programming, in the Verkhovna Rada, and in public policy in Ukraine? BACKGROUND AND CONTEXT The opposition that came to power following the Euromaidan Revolution of Dignity prioritized returning Ukraine to a system with a more powerful parliament. This was accompanied by presidential and parliamentary elections in 2014 and local elections the following year. The Ukrainian parliament elects 450 deputies for five-year terms through a mixed system of majoritarian single-mandate districts and national party lists. Prior to the current program, USAID had funded the Parliamentary Development Program from 1994 to 2013. The RADA Program was initially drafted in early 2013 and was launched in November of that year. Effective implementation of the RADA Program was delayed because of the Euromaidan Revolution, elections, and a new Ukraine-EU relationship. Internally, management difficulties and changes in leadership continued until November 2015. The RADA Program has three objectives: 1. Improved public representation in the legislative process; 2. Expanded role of citizens in monitoring the work of Parliament; 3. Role of legislature in providing independent oversight of the executive branch strengthened. EVALUATION METHODS AND LIMITATIONS i | EVALUATION OF UKRAINE RADA PROGRAM This evaluation utilized a mixed-methods approach that relied on both qualitative data (primarily collected from program documents and through key informant interviews and focus group discussions) and quantitative data (primarily monitoring data and results of the online mini-surveys). The DI evaluation team collected data from a variety of sources, including document reviews, key informant interviews, focus group discussion sessions and mini-surveys to reach a broader sample of members of parliament (MPs), staff, local officials, civil society organizations (CSOs) and journalists who participated in RADA activities. Parallel analysis was used to analyze the evidence from interviews, document review, and web-based surveys. The team conducted 71 key informant interviews with 151 people in Kyiv, Volodymyr-Volynski, Kharkiv, Brovary and Chernihiv and conducted four focus groups with 21 people in Kyiv. A total of 172 people were interviewed. Survey responses were fewer than expected. While these survey data were insufficient to draw independent conclusions, they did corroborate findings from interviews, focus groups, and document review. The evaluation team encountered recall bias, response bias and selection bias. The most effective approach to combating bias is to use multiple sources of data to triangulate on an evaluation issue. By combining information found in documents or interviews from multiple sources, any one piece of biased data did not skew the analysis. EVALUATION FINDINGS EVALUATION QUESTION 1: RELEVANCE OF PUBLIC REPRESENTATION ACTIVITIES The evaluation team found that the Model District initiative was the most widely known representational activity of the RADA Program and one that stakeholders viewed as addressing a need of the Verkhovna Rada for greater connection between MPs and the people they represent. Model District activities resulted in public comments on legislation being incorporated into law. The Model District program also served as a RADA Program vehicle for strengthening cooperation between MPs and local governments. Training provided by the RADA Program offered MPs and staff relevant skills and knowledge that they used in their parliamentary outreach work. MPs and staff who used the European Information and Research Center reported that it filled a need for comparative international information that could not be found elsewhere in the parliament. For the most part, users found the products to be useful in their parliamentary work, and the Center’s rapid turn-around time on requests enabled requesters to use the information in their legislative work. Stakeholders valued RADA Program efforts to increase transparency and openness, particularly OPORA transparency-promotion efforts. The transparency initiative cited most often was an effort for committees to publish more information on their websites. The Civic Platform NGO Register, E-Petitions and Citizen E-Platform were not among the activities perceived by stakeholders in parliament as most beneficial for improving public representation. EVALUATION OF UKRAINE RADA PROGRAM | ii EVALUATION QUESTION 2: RELEVANCE OF OVERSIGHT ACTIVITIES While aimed at improving public representation in legislation, the Model District initiative included many activities that contained an oversight component. Visits to their districts provided MPs with information about the implementation of laws, programs and agencies, which they took back to Kyiv to inform further oversight activities. Many activities aimed at strengthening cooperation between deputies and local governments had an oversight component. The RADA Program established partnerships with the Committee on State Building, Regional Policy and Local Self-Government and the Committee on Corruption Prevention and Counteraction. The program organized field visits, meetings and hearings for those committees. The RADA Program’s support for oversight of decentralization legislation also extended to initiatives beyond parliamentary committee events. The European Information and Research Center (EIRC) provided support to parliamentary oversight through policy papers, info briefs and discussions of oversight internationally. Shadow reports by the implementing partner Agency for Legislative Initiatives (ALI) were of limited utility to parliamentary committees. Other oversight initiatives failed to gain support. EVALUATION QUESTION 3: EFFECTIVENESS OF PUBLIC REPRESENTATION ACTIVITIES Deputies, staff and outside stakeholders most frequently associated the Model District initiative with the RADA Program. “People know the RADA Program through the Model District program,” one deputy said. Stakeholders in Kyiv and in the districts said that the Model District initiative increased deputies’ contact with voters through a series of public events, including town hall meetings, forums, hearings and roundtables. The effectiveness of the Model District activity is illustrated by the interest in expanding the activity to include additional deputies and by additional MPs participating in Model District activities or similar activities outside of the RADA Program. The RADA Program included party-list deputies in the Model District initiative, but it did not include a component to work with parliamentary