PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM FINAL REPORT Prepared under the Democracy and Governance Analytical Services Indefinite Quantity Contract, #DFD-I- 00-04-00229-00 Submitted to: USAID/Serbia JANUARY 2013 This publication was produced for review by the United States Agency for InternationalPrepared Development. by: It was prepared by Democracy International, Inc. DISCLAIMER PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM This is an external evaluation. The views expressed in this document are the authors' and do not necessari- ly reflect the views of the United States Agency for International Development or the United States Gov- ernment. Submitted to: USAID/Serbia Prepared by: Catherine Barnes Troy Etulain Contractor: Democracy International, Inc. 4802 Montgomery Lane Bethesda, MD 20814 Tel: 301-961-1660 www.democracyinternational.com PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM FINAL REPORT JANUARY 2013 PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM TABLE OF CONTENTS EXECUTIVE SUMMARY ..................................................................... i PROJECT DESCRIPTION ................................................................... 1 EVALUATION PURPOSE AND .......................................................... 2 METHODOLOGY ................................................................................. 2 KEY FINDINGS ..................................................................................... 4 CONCLUSIONS AND LESSONS LEARNED .................................. 20 RECOMMENDATIONS ...................................................................... 29 ANNEX A: STATEMENT OF WORK ............................................ A-1 ANNEX B: EVALUATION SCHEDULE ........................................ B-1 ANNEX C: DOCUMENT REVIEW ................................................. C-1 ANNEX D: PERSONS INTERVIEVED .......................................... D-1 ANNEX E: SMAP PROJECT EVALUATION WORK PLAN ..... E-1 ANNEX F: ILLUSTRATIVE TIMELINE ....................................... F-1 ANNEX G: DIGITAL CONVERSION AND THE DIGITAL DIVIDEND ......................................................................................... G-1 ANNEX H: CONSISTENCY OF STORY AND PROGRAM QUALITY STANDARDS ................................................................. H-1 ANNEX I: TYPES OF ASSISTANCE UNDER SMAP.................... I-1 ANNEX J: DONORS’ FUTURE PLANS .......................................... J-1 PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM LIST OF ACRONYMS ABC Serbia Audit Bureau of Circulation Serbia ANEM Association of Independent Electronic Media AOR Agreement Officer's Representative BBC British Broadcasting Corporation BIRN Balkan Investigative Reporters’ Network BIRODI Bureau for Social Research CATI Computer-Assisted Telephone Interface CEPPS Consortium for Elections and Political Process Support CINS Center for Investigative Journalism CoP Chief of Party CSO Civil Society Organization DCoP Deputy Chief of Party DI Democracy International, Inc. DG Democracy and Governance ECG European Consulting Group ETV Emisiona Tehnika I Veze EU European Union GoS Government of Serbia IAB Interactive Advertising Bureau IR Intermediate Result ITU International Telecommunications Union IREX International Research and Exchanges Board ISC Institute for Sustainable Communities MDLF Media Development Loan Fund MSI Media Sustainability Index MSO Media Support Organization NCE No-Cost Extension NDNV Independent Journalists’ Association of Vojvodina ODIHR Office for Democratic Institutions and Human Rights OFCOM Office of Communications NUNS Independent Journalists’ Association of Serbia OSCE Organization for Security and Co-operation in Europe OSI Open Society Institute PMP Performance Monitoring Plan RAB Radio Advertising Bureau RATEL Republic Telecommunication Agency RBA Republic Broadcasting Agency RFA Request for Applications RTS Radio-Television Serbia RTV Radio-Television Vojvodina SMAP Serbia Media Assistance Program SNA Social Network Analysis SOKOJ Serbian Music Authors Organization SoW Statement of Work UNS Journalists’ Association of Serbia USAID United States Agency for International Development PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM EXECUTIVE SUMMARY This report is an evaluation of the $8.8 million, four-year (2008–2012) Serbia Media Assistance Program (SMAP) implemented by the International Research and Exchanges Board (IREX) with funding from the United States Agency for International Development (USAID). The program sought to: (1) strengthen the legal and regulatory environment for media, particularly the role of media support institutions in ensuring fair, timely, and transparent implementation; (2) increase the managerial and financial capacity of independ- ent media outlets, particularly local and regional television stations following privatization and licensing; and (3) enhance the quality and availability of information important to citizens, with a focus on news and program production at the local and regional levels and coverage of reform issues at the national level. USAID contracted Democracy International, Inc. (DI) to provide USAID/Serbia with a performance evalua- tion of SMAP. According to the Statement of Work (SoW), the purpose of the evaluation was to measure the change in development outcomes attributable to the program. Overall, the evaluation was intended to demonstrate: (1) how the program’s performance influenced the Serbian media sector, primary and second- ary partners, and other stakeholders; (2) how and if the program assisted Serbian citizens to access profes- sional and independent news and information, and (3) how IREX implemented the program to support the transition of the Serbian media sector into a legally sound, economically viable system that provides profes- sional and independent news and information for citizens throughout Serbia. For more information on the SoW for the evaluation, please see Annex A. DI relied on a rapid appraisal methodology, utilizing several evaluation methods to quickly, yet systemati- cally, collect data. This approach included a review of documents, key informant interviews, and observa- tion during site visits. Given the limited number of individuals who were able to provide substantive feed- back on the program, the evaluation design featured purposive, nonrandom sampling methodologies. The team conducted interviews based on an interview guide that served to structure discussions, providing for conversational, yet focused, communication. The interview guide was tailored to each SMAP program ob- jective (legal and regulatory framework, managerial and financial capacity, and quality and availability of information) and media topics more generally. For more information on the evaluation methodology, includ- ing the interview guide, please see the approved work plan under Annex E. From September 10–28, 2012, the evaluation team met with more than 80 individuals representing USAID, the U.S. Embassy, IREX, other international donors and implementers, government ministries, regulatory bodies, radio and television broadcasters, print media, online publications, production houses, membership and trade associations, research firms, and media centers. Information on the team’s schedule, document review, and contacts, can be found under Annexes B, C, and D, respectively. The team’s key findings, con- clusions and lessons learned, and recommendations are summarized below. KEY FINDINGS 1. Stakeholders universally agreed that IREX faced major external obstacles to achieving the objec- tives and expected results of SMAP. These included the global economic crisis; limited political will within the government and by political actors to pursue genuine media reform, a low level of official respect for the independence of the media, and entrenched political capture of some segments of the media sector and advertising industry that created unfair market conditions. External obstacles were cited as a major reason why expected legacies of the program are only partly in place. 2. The Mission and IREX noted that changes to the original program design provided SMAP with greater flexibility to adapt to a rapidly changing media landscape and the possibility to choose pro- gram partners that, regardless of medium, had influence, particularly at the local level, and the greatest prospects for success. Evidence suggested that further adjustments to program planning and implementation intended to ensure the survival of local outlets in the near term were successful for radio but not for television partners. PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM i 3. The Serbian government appreciated and partially applied SMAP support related to the digitaliza- tion process. The team found that many individual beneficiaries of SMAP digitalization activities still occupy official positions and will be able to apply what they learned beyond the life of the pro- gram. 4. SMAP partners repeatedly emphasized the value of new media activities, regardless of the IR under which assistance was provided, and credited SMAP with helping to diversify and expand platforms for delivering content to users, reaching consumers, providing member services, and creating oppor- tunities for new revenue streams. Media Support Organizations (MSOs) and media partners have applied new media knowledge, skills, and tools