Performance Evaluation of the Serbia Media Assistance Program

PERFORMANCE EVALUATION OF

THE SERBIA MEDIA ASSISTANCE

PROGRAM FINAL REPORT

Prepared under the Democracy and Governance Analytical Services Indefinite Quantity Contract, #DFD-I- 00-04-00229-00

Submitted to:

USAID/Serbia JANUARY 2013

This publication was produced for review by the United States Agency for InternationalPrepared Development. by: It was prepared by Democracy International, Inc.

DISCLAIMER PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM

This is an external evaluation. The views expressed in this document are the authors' and do not necessarily reflect the views of the United States Agency for International Development or the United States Gov- ernment.

Submitted to: USAID/Serbia

Prepared by: Catherine Barnes Troy Etulain

Contractor: Democracy International, Inc. 4802 Montgomery Lane Bethesda, MD 20814 Tel: 301-961-1660 www.democracyinternational.com

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM

FINAL REPORT

JANUARY 2013

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM TABLE OF CONTENTS

EXECUTIVE SUMMARY ...... i PROJECT DESCRIPTION ...... 1 EVALUATION PURPOSE AND ...... 2 METHODOLOGY ...... 2 KEY FINDINGS ...... 4 CONCLUSIONS AND LESSONS LEARNED ...... 20 RECOMMENDATIONS ...... 29 ANNEX A: STATEMENT OF WORK ...... A-1 ANNEX B: EVALUATION SCHEDULE ...... B-1 ANNEX C: DOCUMENT REVIEW ...... C-1 ANNEX D: PERSONS INTERVIEVED ...... D-1 ANNEX E: SMAP PROJECT EVALUATION WORK PLAN ..... E-1 ANNEX F: ILLUSTRATIVE TIMELINE ...... F-1 ANNEX G: DIGITAL CONVERSION AND THE DIGITAL DIVIDEND ...... G-1 ANNEX H: CONSISTENCY OF STORY AND PROGRAM QUALITY STANDARDS ...... H-1 ANNEX I: TYPES OF ASSISTANCE UNDER SMAP...... I-1 ANNEX J: DONORS’ FUTURE PLANS ...... J-1

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM LIST OF ACRONYMS

ABC Serbia Audit Bureau of Circulation Serbia ANEM Association of Independent Electronic Media AOR Agreement Officer's Representative BBC British Broadcasting Corporation BIRN Balkan Investigative Reporters’ Network BIRODI Bureau for Social Research CATI Computer-Assisted Telephone Interface CEPPS Consortium for Elections and Political Process Support CINS Center for Investigative Journalism CoP Chief of Party CSO Civil Society Organization DCoP Deputy Chief of Party DI Democracy International, Inc. DG Democracy and Governance ECG European Consulting Group ETV Emisiona Tehnika I Veze EU European Union GoS Government of Serbia IAB Interactive Advertising Bureau IR Intermediate Result ITU International Telecommunications Union IREX International Research and Exchanges Board ISC Institute for Sustainable Communities MDLF Media Development Loan Fund MSI Media Sustainability Index MSO Media Support Organization NCE No-Cost Extension NDNV Independent Journalists’ Association of Vojvodina ODIHR Office for Democratic Institutions and Human Rights OFCOM Office of Communications NUNS Independent Journalists’ Association of Serbia OSCE Organization for Security and Co-operation in Europe OSI Open Society Institute PMP Performance Monitoring Plan RAB Radio Advertising Bureau RATEL Republic Telecommunication Agency RBA Republic Broadcasting Agency RFA Request for Applications RTS Radio-Television Serbia RTV Radio-Television Vojvodina SMAP Serbia Media Assistance Program SNA Social Network Analysis SOKOJ Serbian Music Authors Organization SoW Statement of Work UNS Journalists’ Association of Serbia USAID United States Agency for International Development

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM EXECUTIVE SUMMARY

This report is an evaluation of the $8.8 million, four-year (2008–2012) Serbia Media Assistance Program (SMAP) implemented by the International Research and Exchanges Board (IREX) with funding from the United States Agency for International Development (USAID). The program sought to: (1) strengthen the legal and regulatory environment for media, particularly the role of media support institutions in ensuring fair, timely, and transparent implementation; (2) increase the managerial and financial capacity of independent media outlets, particularly local and regional television stations following privatization and licensing; and (3) enhance the quality and availability of information important to citizens, with a focus on news and program production at the local and regional levels and coverage of reform issues at the national level. USAID contracted Democracy International, Inc. (DI) to provide USAID/Serbia with a performance evaluation of SMAP. According to the Statement of Work (SoW), the purpose of the evaluation was to measure the change in development outcomes attributable to the program. Overall, the evaluation was intended to demonstrate: (1) how the program’s performance influenced the Serbian media sector, primary and secondary partners, and other stakeholders; (2) how and if the program assisted Serbian citizens to access professional and independent news and information, and (3) how IREX implemented the program to support the transition of the Serbian media sector into a legally sound, economically viable system that provides professional and independent news and information for citizens throughout Serbia. For more information on the SoW for the evaluation, please see Annex A. DI relied on a rapid appraisal methodology, utilizing several evaluation methods to quickly, yet systematically, collect data. This approach included a review of documents, key informant interviews, and observation during site visits. Given the limited number of individuals who were able to provide substantive feedback on the program, the evaluation design featured purposive, nonrandom sampling methodologies. The team conducted interviews based on an interview guide that served to structure discussions, providing for conversational, yet focused, communication. The interview guide was tailored to each SMAP program objective (legal and regulatory framework, managerial and financial capacity, and quality and availability of information) and media topics more generally. For more information on the evaluation methodology, including the interview guide, please see the approved work plan under Annex E. From September 10–28, 2012, the evaluation team met with more than 80 individuals representing USAID, the U.S. Embassy, IREX, other international donors and implementers, government ministries, regulatory bodies, radio and television broadcasters, print media, online publications, production houses, membership and trade associations, research firms, and media centers. Information on the team’s schedule, document review, and contacts, can be found under Annexes B, C, and D, respectively. The team’s key findings, conclusions and lessons learned, and recommendations are summarized below.

KEY FINDINGS 1. Stakeholders universally agreed that IREX faced major external obstacles to achieving the objectives and expected results of SMAP. These included the global economic crisis; limited political will within the government and by political actors to pursue genuine media reform, a low level of official respect for the independence of the media, and entrenched political capture of some segments of the media sector and advertising industry that created unfair market conditions. External obstacles were cited as a major reason why expected legacies of the program are only partly in place. 2. The Mission and IREX noted that changes to the original program design provided SMAP with greater flexibility to adapt to a rapidly changing media landscape and the possibility to choose program partners that, regardless of medium, had influence, particularly at the local level, and the greatest prospects for success. Evidence suggested that further adjustments to program planning and implementation intended to ensure the survival of local outlets in the near term were successful for radio but not for television partners. PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM i 3. The Serbian government appreciated and partially applied SMAP support related to the digitalization process. The team found that many individual beneficiaries of SMAP digitalization activities still occupy official positions and will be able to apply what they learned beyond the life of the program. 4. SMAP partners repeatedly emphasized the value of new media activities, regardless of the IR under which assistance was provided, and credited SMAP with helping to diversify and expand platforms for delivering content to users, reaching consumers, providing member services, and creating opportunities for new revenue streams. Media Support Organizations (MSOs) and media partners have applied new media knowledge, skills, and tools with varying degrees of success. 5. Evidence suggests that activities under IR 1: Legal and Regulatory Framework contributed to a number of successful advocacy initiatives and increased capacity among most, if not all, media support organizations. Despite these successes, results under this IR were mixed as key media issues and reforms targeted by SMAP remain unresolved or stalled. Stakeholders cited a lack of political will within the government and parliament to push genuine media reform, a lack of respect for media independence, and a limited constituency for reform as hampering additional progress. 6. The team found evidence that SMAP support under IR 2: Sustainability of Media Partners helped some, but not all, key media partners weather the economic crisis and put into place infrastructure, capacities, tools, and/or networks that have the potential to provide longer-term sustainability. Stakeholders cited the ongoing economic crisis as the key reason why the program fell short of its targets under this IR. 7. Evidence points to notable achievements under IR 3: Access to Quality Information, particularly with respect to investigative reporting, building skills in niche reporting and technical capabilities, and diversifying platforms to deliver content to citizens. Results under this IR were mixed, however, as self-censorship continued to be an issue and most partner media outlets had difficulties consistently maintaining higher standards. Stakeholders indicated that outcomes under this IR were negatively affected by the ongoing economic crisis, which contributed to an exodus of trained journalists from the profession. 8. SMAP partners were in broad agreement about the nature and scope of external economic and political factors undermining media development in Serbia and about the value of SMAP assistance, regardless of region or gender. Women media managers credited the program with helping to maintain a focus on gender in a context where virtually all the attention of media partners was on economic survival. 9. SMAP assistance reached many activities and beneficiaries, including public institutions, media outlets (e.g., television, radio, print, and web-based), news agencies, production houses, press centers, research firms, and membership and trade associations and was highly valued by partners and beneficiaries.1 10. Multiple stakeholders familiar with the management of the program indicated that the implementation of SMAP was flexible but that the balance tilted more toward an opportunistic than a strategic approach. 11. Multiple stakeholders familiar with program management issues identified shortcomings in IREX’s management of the program, including work planning, in-house capacity, and handling of the leadership transition.

1 The term beneficiary is used in reference to all individuals and entities assisted/engaged in some way by SMAP (via training, advising, mentoring/coaching, networking, research, subgrants (financial and in-kind, e.g., commodities), and study tours). It is not limited to the core program partners or subgrantees. PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM ii CONCLUSIONS

OBSTACLES TO IMPLEMENTATION AND PROGRAM ADJUSTMENTS 1. Given the scope of external obstacles SMAP confronted, the overarching goal of the program— namely, the transition of the Serbian media sector into a legally sound, economically viable system providing professional and independent news and information to citizens throughout Serbia—as well as some outcome indicators established in the performance monitoring plan (PMP)—were overly ambitious. 2. Adjustments to the original program design were essential for SMAP’s continued relevance. While the diversification of program partners created the possibility to focus assistance on media outlets with the greatest prospects of success, program management appeared slow to move away from some unsustainable partners. 3. Despite the leading role played by European donors vis-à-vis the Government of Serbia’s digital conversion process, the potential of digitalization to undermine local media diversity and sustainability and, thereby, USAID’s development priorities, values, and past investments justified SMAP engagement of decision-makers and processes. 4. The expansion of new media activities to the program was the right decision given the rapid growth of the internet in Serbia and media partners’ limited knowledge of and experience with the web and mobile technologies. Assistance in this area helped partners to understand the importance of new media and produced tangible results under all three intermediate results (IRs). However, the program could have done more to help partners further achieve and own conceptual and structural advances.

INTERMEDIATE RESULTS 1. Coalition-building efforts under SMAP were critical to helping the program’s MSO partners undertake more strategic and effective advocacy initiatives and have a seat at the table. At the same time, SMAP missed opportunities to further broaden the constituency and political will for media reform. 2. By getting competitors to appreciate their common interests and work together to the benefit of the industry, SMAP was able to achieve a multiplier effect that will continue beyond the life of the program. This was an effective approach and significant achievement of SMAP. 3. The provision of market research was essential to better understanding the market position of local partners, to establish the importance of decision-making and planning on an evidentiary basis, and to approach advertisers. 4. Training provided by SMAP was not sufficient to ensure the success of media partners. Tailored mentoring and coaching proved more effective and should have been employed more routinely and thoroughly by the program. Participatory approaches to business and sustainability planning also proved to be more conducive to partner ownership than having external experts present business plans on the basis of data provided by local partners. 5. IREX could not have predicted the negative effect that the economic crisis would have on the quality of professional journalism. The exodus of trained journalists from the profession exposed the need for more sustainable training solutions. 6. Despite the reporting excellence and public legitimacy of the program’s investigative journalism partners, online publications primarily dedicated to investigative reports will likely remain dependent upon donor funds for the foreseeable future.

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM iii PROGRAM MANAGEMENT AND IMPLEMENTATION

1. Shortcomings in IREX program management and technical leadership adversely affected program implementation and progress toward expected results, particularly during Y4. 2. Due to both external and internal factors, SMAP primarily pursued an opportunistic approach. More strategically oriented and focused management could have strengthened program effectiveness despite external obstacles to implementation. 3. While extensive use of outsourcing may have provided for implementation efficiency, it came at the cost of in-house technical capacity and leadership, both within IREX/Serbia and, in some cases, media partners. 4. IREX was slow to recognize the seriousness of problems with the leadership transition in Y4 and did not act quickly or decisively enough to respond to the Mission’s concerns and requests, and therefore was unable to minimize damage to program quality. 5. USAID’s program management was alternately permissive and strict in its oversight of SMAP. In general, the Mission inconsistently and under-utilized its authority over standard program management tools, such as work plans and approval of key personnel.

PROJECT MANAGEMENT AND IMPLEMENTATION LESSONS LEARNED

1. Including the possibility of working with a variety of types of media into program design (Request for Applications and Cooperative Agreement) allowed the program to respond both to local developments and broader trends within the media industry, even if a particular medium is planned to be the primary beneficiary. 2. In-house technical capacity not only makes program implementation more efficient, but also en- hances the program’s ability to help local partners more regularly and more quickly than outsourcing. 3. To ensure adequate program management and implementation, planning documents need to present details that establish the strategic orientation and progression of the program. 4. From the outset of a program, an implementer’s headquarters needs to be sufficiently engaged with the field and the donor to move quickly to limit damage to program quality and donor relations as problems develop. 5. Localization of program leadership positions should be not be undertaken until local staff have the necessary skills. Capacity building needs to be undertaken from the outset to ensure that required skills are in place by the time of the transition. 6. When confronted by ongoing concerns or evidence of poor leadership and an inadequate response by the implementer, the donor should move decisively to deny proposed key personnel/request their removal. 7. The profound changes facing the media sector due to increasing internet and mobile connectivity require implementers to plan farther into the future and utilize cutting-edge technical and economic expertise in designing, planning, and implementing media assistance programs. 8. Subgrants have a limited ability to promote or secure the financial viability of media partners. If a local partner is facing an inevitable closure, a clear and strong case should be made for continued assistance.

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM iv 9. Tailored, hands-on advisory services, such as mentoring and coaching, provided on-site, add value to other activities by helping to ensure that new knowledge, skills, and technologies are synthesized in a way that maximizes the efficacy of assistance for each beneficiary. 10. Fostering cooperation among MSOs on the basis of mutual interests is hard and takes time, but can lead to positive outcomes that benefit MSOs as well as the sector as a whole. 11. Research provides a valuable evidentiary basis for planning and decision-making both by the implementer and by local partners, and can contribute to fundamental shifts in attitude and behavior.

RECOMMENDATIONS

Given the timing of this evaluation vis-à-vis the worldwide economic crisis, which had a significant and protracted impact on the Serbian economy, including the media sector, the Mission may want to consider conducting a review in two to three years (presuming an end to the crisis) to reassess the longer term impacts of SMAP under IRs 2 and 3.

While IREX assistance through SMAP is coming to a close, the Mission remains committed to advancing free and independent media in Serbia and has multiple opportunities to do so. At a policy level, there are several key areas where USAID and the U.S. Government, should continue to focus their efforts and available resources, including:

1. Digital conversion. A host of discrete issues are critical to Serbia’s transition from analog to digital television broadcasting. As the Serbian government’s plans become clear, the U.S. Government could play a positive role in ensuring a smooth, efficient conversion process, potentially including new assistance. The U.S. should also voice its concerns about media freedom and media diversity more frequently and with greater coordination between USAID and the U.S. Embassy. 2. Transparency of ownership and political capture of media. Major political forces continue to exert undue influence on Serbian media and advertising sectors, which stifles political debate and democratic reform. With its media program ending, the U.S. Government should now increase diplomatic efforts to attempt to resolve this issue given the central role the media plays in ensuring democratic reforms. 3. Divestment of the State from media ownership and government support for public media. As evidenced by its plan for regional public service stations and behavior over the past several years, the Government of Serbia (GoS) appears to have given up on media privatization. USAID and the U.S. Embassy should press the GoS to follow its own privatization and advertising policies, which would have a significant effect on the media sector, while also encouraging transparent and accountable procurement vis-à-vis any state subsidization of the media based on a tangible commitment to public service. 4. Public awareness and media literacy. Stakeholder feedback suggests a high level of public misperception and uninformed expectations of the media and apathy about the adoption and implementation of necessary media reforms. USAID should consider support to public awareness and media literacy programs that increase Serbian citizens’ appreciation of independent, diverse, and professional media; expand the constituency for media reform; and, build the skills of inquiry and self- expression that are essential to democracy. Beyond engagement at the policy level, the Mission has several options to further advance media reform, either through existing democracy and governance programs or direct funding of local organizations: 1. The recently awarded Civil Society Forward program can serve as a vehicle for further support of the media sector via activities directed at policy advocacy, capacity building, networking, and the enabling environment. Support to the Media Coalition or select MSOs, as well as efforts by the

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM v Government Office on Cooperation with Civil Society to encourage cooperation between the media and civil society organizations (CSOs), would be compatible with the program’s objectives. 2. To the greatest extent possible, on-going (and future) political process and parliamentary assistance programs should leverage existing relationships and communication channels with political parties, the National Assembly, and Minority Councils to clearly, repeatedly, and consistently convey the importance of media independence and diversity in Serbia. Such programs should also work with the National Assembly to abandon the practice of adopting laws through special procedures, which would have direct and consequential effects for media reform in addition to providing for greater transparency and accountability of policy-making and decision-making in Serbia. 3. Efforts by USAID, in tandem with the Commercial Section of the U.S. Embassy, related to digitalization, antimonopoly legislation, and the development of e-commerce as well as government procurement reform in Serbia, could have important implications for the media sector and should be pursued. 4. Direct funding of local partners by the Mission should include media partners with an emphasis on supporting organizations that can leverage past media investments, achieve a multiplier effect, and sustain support to the sector upon USAID’s departure from Serbia. Direct funding of local organizations could also be used to support the provision of quality news and information programs, particularly investigative reporting.

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM vi PROJECT DESCRIPTION

This report represents results from an evaluation of a four-year media support program implemented by the International Research and Exchange Board (IREX) under United States Agency for International Develop- ment (USAID) Cooperative Agreement No. 169-A-00-08-00101-00. The Serbia Media Assistance Program (SMAP) started on September 24, 2008 and, as a result of a no-cost extension approved on September 21, 2012, has a current completion date of December 31, 2012. USAID obligated funds for the program total $8.8 million. SMAP aimed to support the transition to a robust legal environment in which Serbian media attain economic viability and consistently provide professional and independent news and information for citizens throughout Serbia. The program sought to build on upon previous USAID media support by: 1. Assisting media support institutions, many of which have become solid organizations with previous USAID assistance, to act as more effective advocacy organizations; 2. Using existing Serbian talent and resources to provide technical assistance and training, to the maximum extent possible; 3. Replicating the journalism standards established at the national level with USAID assistance at the local and regional level; and 4. Refocusing support from previous long-term partners who have reached standards of quality journalism and are nearing sustainability. The program was designed to produce the following intermediate results (IRs): 1. Strengthen the legal and regulatory environment for media, particularly the role of media support institutions and other organizations in ensuring fair, timely, and transparent implementation; 2. Increase the managerial and financial capacity of independent media outlets, particularly local and regional television stations following privatization and licensing; and, 3. Enhance the quality and availability of information important to citizens, with a focus on news and program production at the local and regional levels and coverage of reform issues at the national level. The expected legacies of SMAP included: 1. Fair, timely, and transparent implementation of a consistent legal and regulatory framework for media; 2. Sustainable capacity of media support institutions and other organizations to effectively advocate for both the media sector and the public interest; 3. A core of private local and regional media outlets throughout the country that consistently provide quality coverage of local, regional, minority, and reform issues; and 4. Strong local media assistance providers supported by the domestic media market.

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM 1 EVALUATION PURPOSE AND METHODOLOGY

At the request of USAID/Serbia, Democracy International, Inc. (DI) conducted a performance evaluation of SMAP. According to the Statement of Work, the purpose of the evaluation was to measure the change in development outcomes attributable to the program. Overall, the evaluation intended to demonstrate: (1) how the program’s performance influenced the Serbian media sector, primary and secondary partners, and other stakeholders; (2) how and if the program assisted Serbian citizens to access professional and independent news and information; and (3) how IREX implemented the program to support the transition of the Serbian media sector into a legally sound, economically viable system that provides professional and independent news and information for citizens throughout Serbia. (For more information on the Statement of Work for this evaluation, please see Annex A.) DI relied on a rapid appraisal methodology, utilizing several evaluation methods to quickly, yet systematically, collect data. This approach included a document review, key informant interviews, and observation during site visits. Given the limited number of individuals who were able to provide substantive feedback on the program, the evaluation design featured purposive, nonrandom sampling methodologies, including:  A modified version of quota sampling, where the team identified various categories to be included in the evaluation design (e.g., USAID, the U.S. Embassy, other donors, IREX, other civil society and media implementers, media outlets, media support organizations, policy makers, media regulators, and independent experts);  Judgment sampling, where the team selected the sample based on expert judgment and understanding of the program; and  Snowball sampling, where the team obtained referrals from initial subjects to generate additional interview subjects. The team interviewed stakeholders with direct knowledge of and some level of involvement in the program (i.e., in the capacity of donor, implementer, service provider, partner, or beneficiary). Due to the nonrandom nature of the sampling, the evaluation was subject to selection bias, with findings reflecting the feedback of stakeholders who were not fully representative of the media sector. In addition, as the evaluation asked stakeholders to recall events over a four-year period—and due to the fact that many of SMAP’s partners and beneficiaries were also participants in previous IREX programs in Serbia dating as far back as 1998 (i.e., ProMedia I and II)—there were risks of recall and attribution bias. The team sought to mitigate the occurrence of recall and attribution bias by instructing stakeholders that the evaluation was focused solely on SMAP (i.e., assistance dating from October 2008), through vigorous questioning intended to improve stakeholder recollections, and by a comparison of stakeholder feedback against program planning and reporting documents. Before arriving in Serbia, the evaluation team conducted a detailed desk review of relevant documents. This initial review helped the team to better understand the social and political context and related challenges and opportunities for the Serbian media sector, as well as SMAP’s goals, implementation plans, and performance monitoring efforts. Through the desk review, the team identified relevant and measurable indicators of program outcomes and developed and refined its evaluation methodology, work plan, and interview guide based on a more thorough understanding of the program’s activities. (For more information on the evaluation methodology and activities, please see the work plan in Annex E.) Following an initial briefing and finalization of evaluation planning with the Mission, the team conducted key informant interviews in Aranđjelovac, Belgrade, Kragujevac, Niš, Novi Sad, and Požarevac. On-site key

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM 2 informant interviews provided the team with a better appreciation of program stakeholders’ attitudes and behaviors and the opportunity to make direct observations as well as to collect in-depth information on specific issues based on individuals’ perspectives and experience. The interviews yielded insights not readily apparent or fully captured by formal programmatic reporting. (For a full list of persons interviewed, please see Annex D.) The team conducted interviews on the basis of an interview guide tailored to each SMAP program objective (i.e., legal and regulatory framework, partner sustainability, and quality information) and specific media topics. The guide served to structure discussions with key informants and provided for conversational, yet focused, communication. (For more information on specific questions, please see the illustrative interview guide in Annex E.) The specific mix of questions used in a given interview was based on the background and expertise of interviewees and demographic considerations, as appropriate. By using a standardized questionnaire, the team was able to qualitatively compare the answers and reactions of different stakeholders and beneficiaries to facilitate inferences on program performance and impact. At the same time, this flexible approach also allowed the team to explore unanticipated topics and adjust its data collection accordingly. SMAP institutional partners are located throughout Serbia, including Novi Sad in the north, and Niš, Novi Pazar, Sandžak, Užice, and Zaječar in the south. Given the size of the team, the timeframe for the evaluation, and budget considerations, the evaluation team was not able to visit all of these locations. In addition, due to the timing of IREX’s summer school in Aranđjelovac (September 12–15, 2012) and the second Southeast Europe Media Conference of the Organization for Security and Co-operation in Europe (OSCE) in Belgrade (September 20–21, 2012), some partners were traveling to and from these locations during fieldwork. As such, the team structured its schedule to be able to meet with summer school and conference attendees from further locations while they were in Belgrade or Aranđjelovac. The team was not in a position to fully isolate the impacts of the SMAP program from media assistance provided by other international organizations and implementers, including the European Union (EU), the OSCE, the U.S. Embassy Public Affairs Section, and Open Society Institute (OSI), which were often involved in mutually reinforcing activities related to media reform and the work of media support organizations (MSOs), journalism training, study tours, workshops, digitalization efforts, and content development. In particular, IREX and the OSCE worked together closely and regularly implemented joint events.

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM 3 KEY FINDINGS

Below, we present key findings in four areas: (1) obstacles to program implementation; (2) intermediate results; (3) regional and gender considerations; and (4) program management and implementation.

OBSTACLES TO PROGRAM IMPLEMENTATION This section of the report considers various obstacles to program implementation, including economic and political factors not anticipated in the original program design, how the program responded, related effects on SMAP’s work and results, how the program utilized new media, and the effect that new media activities had on program partners and goals. (Further elaboration on this point can be found in the discussion of results under each of the program’s intermediate results beginning on page 10.)

ECONOMIC AND POLITICAL FACTORS Key Finding Stakeholders universally agreed that IREX faced major external obstacles to achieving SMAP’s objectives and expected results. These included the global economic crisis; limited political will within the government and by political actors to pursue genuine media reform, a low level of official respect for media independence, and entrenched political capture of some segments of the media sector and advertising industry that created unfair market conditions. External obstacles were cited as a major reason why expected legacies of the program are only partly in place.

ECONOMIC FACTORS Soon after the program began in late September 2008, the world economy entered a deep recession from which it has yet to recover. Beginning in October 2008, the value of the Serbian dinar began to fall precipi- tously. During the four-year span of SMAP, the value of the dinar against the euro fell nearly 34 percent.2 This presented a major setback to the sustainability of media outlets that needed to pay some portion of their operational costs, such as newsprint, or to pay down loans in euros. Several leading television and radio stations had taken out loans from the Media Development Loan Fund in New York. These loans were pegged to the euro. As the dinar plummeted, the cost of these loans increased. According to the Association of Independent Electronic Media (ANEM), in 2007, experts were predicting that Serbia would have one of the fastest growing advertising markets in the world. Between 2008 and 2009, advertising revenue fell by 21 percent. In 2011, advertising revenue remained well below 2008 levels. This significant decline was further compounded by and intertwined with political interference within the advertising industry, via leading advertising agencies and local companies seeking to buy ads directly. A local radio manager explained, “All serious advertising goes through very few agencies that are politically con- trolled. If you want advertising, you have to play with them, according to their terms. They pay you when they want. So, we are forced to have local advertisers. But there is no money at all. Every day another company goes out of business.” The editor-in-chief of a local television station told the team, “If we call [local] companies that have money, and there are fewer and fewer, they say that they would like to support our programming but the mayor would be furious if their advertising appeared on our station.” SMAP also inherited partial media privatization with continued state media ownership and subsidization. Public media at the local level continues to be funded by municipal budgets. Subsidies to the media are distributed on the basis of politics and official connections rather than on ratings or program quality. State- owned and state-subsidized media are able to undercut commercial stations’ prices and advertising rates, and thereby create unfair market conditions. Various local stakeholders also told the team that some local television stations were purchased for the property upon which they stood, in failed attempts to use bankruptcy laws to cover unrelated debts, or to advance the political or personal agendas of the owners. Given the opacity of media ownership in Serbia, or at least the money behind the owners, the vested interests in these

2 Source: Oanda.com PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM 4 stations was not necessarily apparent. New legislation on media ownership and the concentration of media ownership had yet to be passed at the time of the team’s visit. The owners and directors of newly privatized stations reportedly knew little about media, and brought on board people with similarly low capacity. Serbia also suffers from having many more media outlets than can be sustained in a country of seven million people. According to the 2012 Media Sustainability Index, there are 517 print outlets (20 dailies, 83 weeklies, six bi-weeklies, and 72 monthlies); approximately 186 radio stations (two public service, 47 regional, 137 local); and 173 television stations (two public service, five with national coverage, 25 regional, and 102 local, 39 cable stations). Given the poor economy, digital conversion, increasing internet usage, and the increasing popularity and prevalence of cable TV, a consolidation of the broadcast media sector is inevitable and already occurring, most dramatically in the case of local television.

POLITICAL FACTORS A lack of respect by nearly all political actors for media independence and limited political will to pursue genuine media reform further compounded the dire effects of the economic crisis for media in Serbia. “Me- dia are still being treated as mechanisms for propaganda,” the head of one MSO told the team. Constitution- al provisions and several supporting laws protect free speech in Serbia. However, there are longstanding issues with the legal and institutional framework for media that remain unresolved or have escalated in recent years.3 During the course of the program, the National Assembly adopted media legislation in a non- transparent manner according to special procedures, introduced bills at the eleventh hour that were contrary to those discussed with media stakeholders, and passed laws that inhibit free speech. For instance, the Public Information Law contradicted Serbia’s international obligations and violated Serbia’s constitution. Incon- sistent progress on all laws and by-laws (i.e., implementing regulations) affecting the media either directly or indirectly led to incompatible provisions that impeded implementation or created conditions for arbitrary enforcement. While the adoption of the Media Strategy in 2011 established an essential roadmap for reform, in the year since the strategy has been adopted, none of the implementation deadlines have been met and no key piece of legislation has been adopted, such as new laws on the concentration and transparency of media ownership, public information, public services, and electronic media. Moreover, the adoption of the 2012 budget included approximately $4.5 million for the direct financing of state-owned media. This directly contradicted the principles of the Media Strategy and suggested no willingness to withdraw from media ownership.4 “The state recognized the Media Strategy only to the extent that it was convenient to them—not to the media. They were able to tick off an EU box,” the head of one radio station observed. In May 2012, the country experienced a change in government. Despite public statements and assurances made to the team about the new government’s commitment to move swiftly to implement the Media Strategy, this remains to be seen. Most stakeholders the team met with were skeptical. According to one local media expert, “The media was under the tight reign of the Democratic Party and will be under any party that wins elections. Ten years after the fall of Milošević, we find ourselves at a crossroads.” State media ownership and financing at all levels continues with subsidies distributed on the basis of political considerations rather than the quality or reach of programming and occurs without transparent and accountable procurement processes. With respect to commercial media, transparency of ownership (or the sources of funds used to purchase media) does not exist. According to virtually all stakeholders the team met with, political capture extends broadly to commercial media, with the exception of online media, which remains largely unregulated and where many stakeholders believe the greatest media freedom currently exists. Political influence also extends to the leading advertising firms in the country, which according to various estimates, control 50 to 80 percent of advertising revenue. “The advertising business is run by political play- ers in a near-monopoly situation. This reinforces other political controls,” the head of one news agency told the team. “This allows political actors to exert pressure by intimidating clients,” he explained.

3 Media Sustainability Index, 2012. 4 IREX Quarterly Report, 31 January 2011. PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM 5 During fieldwork, the team heard of other threats to free speech, such as attacks against journalists, litiga- tion, and other forms of intimidation by a range of political, institutional, criminal, and militant groups.5 Several leading investigative journalists at the national and local level require full-time police protection as a result of ongoing threats against them. During the course of the evaluation, for example, an investigative journalist based in Loznica was threatened despite being under 24-hour police protection since 2005 when he was severely beaten by a former police officer.6 According to the 2012 Media Sustainability Index, “Ser- bian society reacts inadequately to freedom of speech violations. Typically, political removal of editors and directors, threats to the freedom of speech, prohibitions to individual journalists to enter to public press con- ferences, attacks on journalists, and trumped-up court cases against journalists are greeted with mild—if any— public reaction.”7

PROGRAM ADJUSTMENTS In response to these external factors as well as emerging opportunities within the media sector, IREX, in consultation with USAID, made several significant adjustments to the original program design after the program began. These adjustments included a change in the definition of media partners targeted by assistance, a shift from television to radio partners, adjustments to the focus on digitalization activities, and an increase in new media activities under all three intermediate results. Each of these adjustments is discussed below. Radio and new media activities are also covered in the sections dedicated to IR 2: Sustainability of Media Partners and IR 3: Access to Quality Information.

SHIFT FROM TELEVISION Key Finding: The Mission and IREX noted that changes to the original program design provided SMAP with greater flexibility to adapt to a rapidly changing media landscape and the possibility to choose program partners that, regardless of medium, had influence, particularly at the local level, and the greatest prospects for success. Evidence suggests that further adjustments to program planning and implementation intended to ensure the near-term survival of local outlets were successful for radio but not for television partners.

The Request for Applications (RFA) for SMAP originally envisioned support only to television stations. At that time, ratings research consistently showed that television, particularly national television, was by far the primary medium Serbian citizens used to obtain news and information.8 At the outset of the program, IREX, in consultation with USAID, agreed to expand the definition of SMAP partners to “media outlets” and “news sources,” which included radio and, later, new media.9 In Y1, IREX initiated its own media assess- ments and market research to better understand the local media landscape and identified several local television and radio stations as program partners. While Ipsos research in Y1 indicates that while SMAP’s local television partners were strong in relation to their local television competitors, they were under growing pressure from national television stations, increasing radio market share, planned digitalization, and unfair market conditions created by government subsidies, rising concerns about the sustainability of local television, including the possibility of collapse, some of which have been borne out.10 This research also showed that the program’s local radio partners remained strong contenders in the market.11 By Y2, IREX began a more significant shift toward radio, although SMAP continued to provide assistance to several local television stations during the latter half of the program.

5 Media Sustainability Index 2012. 6 SEEMO Requests Wider Protection for Serbia Journalist on the B92 website, 23 September 2012. 7 Media Sustainability Index 2012, p. 132. 8 According to USAID Serbia, measurement of local and national television markets was available since 2002. 9 SMAP program documentation suggests that IREX was aware of sustainability issues surrounding local television as early as the outset of the program: “From the beginning of the program . . . most TV stations were not sustainable.” This contributed to a joint decision on the part of IREX and USAID at the beginning of SMAP to redefine partners as media houses. See SMAP work plan (Y4) p. 8. 10 SMAP Y1 Annual Report, p. 27. 11 Ibid. PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM 6 As the effects of the economic crisis began to take hold in Y1, IREX began to further adjust its program planning and implementation to focus support on the immediate survival of its local partners. The program’s planned sustainability grants, which were originally intended to foster local sustainability, were converted into emergency financial support grants when it became clear that key local media partners might not be able to stay in business. SMAP efforts to bolster the survival of local media outlets in the near term worked for radio but not for television. The first emergency financial support grants went to local radio stations, Boom 93, Radio 021, and City Radio, and subsequently to Luna Radio. During the evaluation, all were still in operation, although under significant financial pressure (as discussed further under IR 2: Sustainability of Media Partners, below). The evaluation team met with three local television partners that had received extensive support from SMAP. Of these three partners, one had declared bankruptcy the previous month, one was in the process of initiating bankruptcy proceedings, and the third was in the midst of a politically moti- vated management change and was reported to be on the verge of folding.

REFOCUSED DIGITALIZATION ACTIVITIES Key Finding: The Serbian government appreciated and partially applied SMAP support related to the digitalization process. The team found that many individual beneficiaries of SMAP digitalization activities still occupy official positions and will be able to apply what they learned beyond the life of the program. Serbia has an international obligation to convert from analog to digital television broadcasting no later than June 17, 2015, because the nation is a signatory of the International Telecommunications Union (ITU) GE- 06 Agreement, adopted in 2006 at the ITU Regional Radio Communication Conference. Planning and prep- arations for digitalization have been hampered by the inability of the new state company, Emisiona Tehnika I Veze (ETV), established on January 1, 2011, to move forward coherently and efficiently. Key tasks have yet to be completed, such as adopting a frequency plan and securing an estimated $70-80 million loan for necessary equipment. The government has already missed two internal deadlines for completion of the digitalization process. (For a more detailed discussion of current digitalization issues, please see Annex G: Digi- tal Conversion and the Digital Dividend.) The original SMAP agreement called for involvement in digitalization as a vehicle to assist regional and local television partners to successfully complete the digital transition. At the time, SMAP decided that television was not viable and opted out of support for local/regional television, digitalization became less of a direct program priority. Nevertheless, USAID gave IREX approval to continue to work on a limited basis in two areas: (1) to assist the Government of Serbia (GoS) with digital conversion, although there were some questions about this decision given limited political will within the GoS and considerable European support to the GoS for digitalization, and (2) to shift SMAP’s focus to the digital dividend and how new media could exploit digital conversion.12 SMAP provided various forms of support to the government’s digital conversion process, including two trips to the United Kingdom to visit agencies managing the country’s conversion process; a digitization media strategy proposal for a public information campaign; and a study of the costs of digital conversion. SMAP also conducted roundtables on digitalization, including a roundtable in 2011 featuring the president of the Association of Commercial Television in Europe who discussed lessons learned from Western Europe’s digitalization process. Government representatives from the Republic Telecommunication Agency (RATEL), the Republic Broad- casting Agency (RBA), and the Ministry of Foreign and Home Trade and Telecommunications indicated that the study tours were the most valuable digitalization assistance they received under SMAP. Participants found visits to the British Broadcasting Corporation (BBC) to discuss public information campaigns on digital conversion and to the Office of Communications (OFCOM) to observe the operations of this regulatory

12 Based on information provided by the Mission, IREX expressed interest in continuing engagement in digitalization after the program’s initial digitalization activities had concluded and the cooperative agreement’s requirements were met; however, the political and economic environment was not favorable to significant reform opportunities. Given limited resources and opportunities, informed by USAID’s rapid assessment, the Mission opted to focus on other media priorities, while encouraging IREX to take ad- vantage of opportunities to exploit the digital dividend. PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM 7 institution to be particularly beneficial.13 According to a senior RBA official, “What we saw there will be, as much as possible, implemented into the process here in Serbia. Already we are making changes to our organizational scheme, job descriptions, and operations. We learned a lot about the position and role of the regulatory body.” IREX also provided the Ministry of Foreign and Home Trade and Telecommunications with a study on the costs of digital conversion. The Ministry found the study useful, with the exception of cost estimates for digital conversion boxes, which proved to be overstated. Additional planned studies could not be carried out since the government did not have a frequency plan. The Ministry also opted not to use the communications plan developed with SMAP assistance. According to IREX, of the seven government representatives who participated in the study tour, five are still in decision-making positions within the various Serbian ministries and agencies involved in the digitalization process and a sixth is believed to offer continued input through a government task force.

EXPANDED NEW MEDIA ACTIVITIES Key Finding: SMAP partners repeatedly emphasized the value of new media activities and credited SMAP with helping to diversify and expand platforms for delivering content to users, interacting with consumers, providing member services, and creating opportunities for new revenue streams. To date, MSOs and media partners have applied new media knowledge, skills, and tools with varying degrees of success. The original program description and cooperative agreement envisioned new media initiatives within the parameters of support provided to partner television stations at the regional and local levels under IR 2 (financial and management capacity) and as a means of advancing IR 3 (quality and availability of information). Once USAID and IREX expanded the definition of SMAP partners at the outset of the program, SMAP was in a position to advance understanding and use of new media among a range of partners, regardless of medium. During the first two years of the program, IREX undertook some new media activities.14 In 2010, given the rapid growth of internet use and diminished prospects for local television, the Mission commissioned a rapid appraisal and recommended much greater emphasis on new media, committing additional funding for this purpose.15 SMAP new media assistance involved advising, training, and subgrants. The program also included a research component that involved baseline and longitudinal studies of youth and new media, which had not previously been undertaken in Serbia,16 and provided for the development and operationalization of a web- measurement system. Beginning in Y1, SMAP also worked with marketing and advertising professionals to help companies understand the value of online advertising. This involved a series of trainings carried out by the Audit Bureau of Circulation Serbia (ABC Serbia) at the national and regional levels17 and Digital Days implemented by the Interactive Advertising Bureau (IAB). In total, these events attracted thousands of participants to live events and via webcasts and as well as sponsors, thereby reducing dependence on SMAP funds.18 Program partners stressed that new media assistance under SMAP was invaluable given the novelty of new media in Serbia at the start of the program and relatively low levels of technical capacity. Stakeholders identified multiple positive results stemming from the integration of new media and social networking under SMAP. These included helping MSOs to better serve their members and providing media partners, particu-

13 There was a second study tour for MSOs, which indicated that the trip served to increase their understanding of the digitalization process and the issues confronting the GoS, media outlets, and the public. An identified shortcoming of this tour, however, was that it did not include a visit to OFCOM. 14 This assistance included advising, website development for four of its legacy partners (two of which involved web TV), a news lab that utilized Skype for television reporting, and the first subgrant dedicated to web measurement as well as nine training activities on new media topics.14 15 This change was reflected in the budget revision that provided an additional $800,000 in program funds during Y3 and Y4. 16 The Ipsos research on youth and new media was funded by a consortium of partners including IREX and the Institute for Sustain- able Communities (ISC) (with USAID funds), the OSCE, and media. 17 Including in Subotica, Kragujevac, and Niš. 18 According to IREX program reporting for Y2–Y4. PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM 8 larly radio stations, with new platforms to deliver content to users, reach consumers, and generate revenue. Web audit results in Y3 showed significant increases in unique visitors to partner station websites that were upgraded with SMAP support (see more below) and several partners reported to the team that they were able to sell or at least barter online advertising. Starting in Y3, SMAP also supported several online publications widely recognized by stakeholders as leaders in the field of investigative journalism. Efforts to build, rejuvenate, and provide technical updates to partner websites represented a significant component of IREX’s new media assistance. During the first two years of the program, IREX provided website support to four legacy partners.19 One of these, the news agency FoNet, was cited as a program partner that exemplified effective utilization of online technologies to expand its reach, enhance users’ online experience, improve service provision to its subscribers, and achieve greater financial viability. More information about FoNet’s success is described in the box below.

SUCCESS STORY: FONET

FoNet was registered in 1993 as the first independent, private news agency in Serbia. According to the head of the agency, “When we founded FoNet, we were not thinking about business. We started with no

investment and no equipment. Today, with help from IREX, FoNet is a pioneer of multimedia news production, offering video, audio, photo, and print content. The agency has 100 institutional subscribers and provides news and information on Serbia, the Balkan region, and other countries of strategic importance to Serbia.

Under SMAP, FoNet received support to introduce web TV and live streaming. The addition of web TV allowed FoNet to directly reach the public for the first time. According to the head of FoNet, web TV relies on new content produced by agency advertising and is now the most serious source of income. When FoNet first introduced TV/video clips to its news service in 2005, there were 3,900 downloads of the video clips for the entire year. In 2011, it had more than 72,000 video clip downloads. Also in 2011, users downloaded one million FoNet photos. IREX support also allowed FoNet to enhance its database, allowing improved access to the agency’s archives, which are now five times larger than in 2005. FoNet is currently self-sustaining despite the economic crisis in Serbia, when clients are not paying on time, and unfair market conditions created by the government’s 2.2 million euro subsidy of the state news agency Tanjug still in place. “Our cooperation with IREX was a real partnership,” the head of FoNet told the team. “The timing of support was perfect. It was real. Essential.”

During the latter half of the program, primarily in Y4, IREX helped its remaining partners upgrade their websites to manage a more dynamic collection of web material, share functionalities, and integrate social media, including Facebook, Twitter, and YouTube. Web radio player was added to all radio partner sites and web TV player to television partner sites. As part of this support, SMAP developed a common content management system. According to IREX, this approach reflected resource constraints and allowed for econo- mies of scale, facilitated the delivery of training on website management across partners, and provided for a core group of people who could train on the system. Yet, many partners the team met with viewed the updated sites as something IREX gave them, via the services of Inbox, rather than something they did themselves with IREX guidance.

SMAP also offered 24 new media training activities that were carried out directly by IREX or its partners.20

19 ANEM, Media Center Belgrade, Radio 021, and FoNet. Radio 021 and FoNet received support that allowed them to launch web TV. 20 According to an updated (December 2012) training list provided to the team by IREX HQ. These training activities were further supplemented by other special events, e.g., the regional conference “Journalism and New Media: New Opportunities,” sponsored by the French Embassy, IREX, OSCE, and NUNs. PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM 9 Training covered a range of topics21 of varying depth and breadth. Activities featured both one-off training exposures and structured training as part of a series and were made available to a significant number of beneficiaries, a core group of partners, or an individual outlet. Writing for the web and topics related to online advertising were the most frequently offered new media trainings. Some partners reported that they weren’t always able to absorb and apply what they learned or to utilize all of the tools that they had been given. This finding was reinforced by SMAP program documentation. For example, according to the no-cost extension (NCE) request in Y4, many partners were poorly using tools for interacting with their audience and required further training on citizen journalism, interactive content, and integrating user content.22 Multiple partners also reported challenges related to monetization, developing fee structures for online advertising, and integrating new media into business models. Despite significant and increasing mobile penetration in Serbia throughout the course of the program, SMAP reporting and other documentation indicated a relatively late (mostly in the latter part of Y4) and limited focus on the advanced use of mobile phones for news distribution or collection, website optimization, and mobile/website integration.23 A notable exception was Radio 021, which was one of the few partners that had a mobile-optimized version of its website using the Android platform by the time of the evaluation.24 Radio 021, which received SMAP support for new media as early as Y1, emerged as a leader in leveraging online and mobile technologies. According to the 2011 Media Development Loan Fund (MDLF) Media De- velopment Impact Dashboard, Radio 021 now reaches more people through its website than through radio broadcasting.25

INTERMEDIATE RESULTS This section addresses how SMAP influenced its partners through its three main intermediate results, how these intermediate results contributed to the program’s overall goal, and the extent to which the program achieved its intended results and legacies. This section also supplements the discussion above under “Obsta- cles to Implementation” regarding how the program used new media and its effect on the program’s partners and goals.

IR1: LEGAL AND REGULATORY FRAMEWORK Key Finding: Evidence suggests that activities under IR 1: Legal and Regulatory Framework contributed to a number of successful advocacy initiatives and increased capacity among most, if not all, media support organizations. Despite these successes, results under this IR were mixed as key media issues and reforms targeted by SMAP remain unresolved or stalled. Stakeholders cited a lack of political will within the government and parliament to push genuine media reform, a lack of respect for media independence, and a limited constituency for reform as factors hampering additional progress.

CAPACITY OF MEDIA SUPPORT ORGANIZATIONS The evaluation team found that SMAP contributed to increased capacity among various MSOs to conduct advocacy and to better serve their members. To varying degrees, the program helped partner MSOs intro-

21 Based on the updated list of training topics provided by IREX HQ (December 2012) and further supplemented by SMAP annual and quarterly reporting, training topics included but were not limited to new media trends and standards, multiplatform journalism, writing and editing for the web, citizen vs. professional journalism, social networking, online sales techniques and marketing campaigns, online advertising, website promotion and monetization, web measurement, website management, internet entrepreneurship, and use of mobile applications. 22 IREX Proposed Extension Period Activities for the period June 1, 2012 through December 31, 2012, p. 11. 23 By 2011, Serbia had 143 percent mobile penetration, 43 percent fixed-line internet penetration, approximately 35 percent mobile broadband penetration, 11 percent fixed-line broadband penetration, and improved regulations supporting number portability. Sources: www.ratel.rs and “An Overview of Telecom Market in the Republic of Serbia in 2011” (RATEL: 2012). 24 The mobile-optimized version had launched just a few weeks before the start of the evaluation. 25 MDLF Media Development Impact Dashboard “is an annual assessment of the impact the work of the Media Development Loan Fund on the financial sustainability of news businesses. It is compiled by analyzing sales, reach, risk indicators, the number of people they employ, and other performance data gathered over the length of MDLF’s relationship with news organizations, usually five to seven years, and provides a portrait of the development of each client—and through it, an indication of the impact of their work.” PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM 10 duce mechanisms to improve communication with their members (e.g., through member surveys and re- vamped websites), prepare business plans, improve member services, and offer income-generating programs and services. Some training programs are generating profits that are being reinvested in member services and support. In one case, a partner MSO was able to demonstrate that its training programs improved the job prospects of its trainees. Despite indications that MSOs were improving member outreach and services, evidence also suggests that efforts to achieve an engaged and active membership remained a process driven from the top down. IREX used an organizational capacity assessment tool to track the organizational development of four MSO partners. While three of these four associations showed consistent year-to-year improvement in organizational development, only one association had met its target for organizational sustainability by the time of the evaluation.26 The fourth MSO showed stagnant organizational development during the first three years of the program and exhibited backsliding in the final year. The team found that individual MSO leaders and their receptivity to this type of assistance greatly affected the effectiveness of the program’s efforts to build organizational capacity. Using the Advocacy Index, IREX also tracked the level of satisfaction with MSOs’ provision of advocacy support. On this indicator, only one of the four MSO partners monitored met or exceeded its advocacy support targets in three out of the four years of the program, while another MSO met or exceeded its SUCCESS STORY: MEDIA COALITION ADVOCACY VIS-À- targets in two out of four years. The remaining VIS THE FINALIZATION AND ADOPTION OF A GOVERN- two MSOs showed year-to-year improvements for MENT MEDIA STRATEGY the periods monitored despite falling short of their advocacy support targets. Specific advocacy suc- Stakeholders repeatedly identified the Media Coalition’s successful advocacy on the Serbian government’s Media cesses are discussed below. Strategy as a major accomplishment of the program.* This represented the first time that the leading trade and mem- LEGAL AND REGULATORY FRAMEWORK bership associations pursued a common approach to policy AND MEDIA MONITORING reform. Joint advocacy by the coalition led to substantial Virtually everyone the team interviewed consid- modification of the government’s proposed media strategy, ered the creation of the Media Coalition, its suc- which laid out a roadmap for media reform in Serbia. cessful advocacy on the finalization and adoption of the government’s Media Strategy, and its and By compartmentalizing differences and focusing on mutu- ongoing advocacy activities (see text box at right) al interests, the members of the Media Coalition were able as the most important achievements and potential to come up with a common and discrete set of priorities to legacies of SMAP. Local stakeholders strongly present to the government. Six of seven of their key negotiating points were agreed to by the GoS. endorsed the way in which IREX laid the founda- tion for the coalition, by building the capacity of According to one MSO leader, “[the Media Coalition] is the MSOs, developing their skills in consensus the biggest achievement of the past four years . . . Its im- building and conflict resolution, and identifying a pact on the Media Strategy is huge. This achievement goes common set of interests. “IREX did a lot to bring far beyond the accomplishments of any other project.” the associations together without forcing them. “The Media Coalition will continue to work after IREX. Building their capacity to work together. This was It’s something that all the members want,” another MSO a very intelligent approach,” the head of one TV told the team. “It’s proven to be an effective approach for station told the team. the media community to protect their interests vis-à-vis the government.” Coalition members, among numerous other stakeholders the team met with, strongly asserted that *Note: In addition to IREX, the OSCE was also heavily external facilitation by the first SMAP Chief of involved in efforts to support development of the Media Party (CoP), who was trusted by all partners and Strategy. OSI also provided support. had a nuanced understanding of the media situation in Serbia, was “essential” to the formation of the Media Coalition. At the same time, evidence suggest-

26 This is based on the most recent SMAP reporting (from Y3) on this indicator that was available to the team. PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM 11 ed that multiple factors came into play in a way that created the conditions for joint action, including changes in leadership at the two journalists associations, the release of the EU’s media study and the “carrot-and- stick” provided by the EU, as well as the increasingly dire media enabling environment. A social network analysis conducted in Y4 of the program confirmed “satisfactorily dense” links between the Media Coalition partners to support continued joint action. However, it also indicated weak links to other media civil society organizations (CSOs) that had vested interests in similar issues. In addition to the Media Strategy, SMAP supported other successful advocacy initiatives including: (1) efforts to have certain provisions of the Public Information Law overturned by the Constitutional Court; (2) lobbying for a 50 percent reduction in RATEL fees; (3) having the Commission for Copyrights throw out excessive Serbian Music Authors Organization (SOKOJ) tariffs; and (4) getting regulatory authorities to crack down on illegal broadcasters (the number of pirate stations decreased from over 300 in 2010 to just 30 by the time of the evaluation). Virtually all stakeholders the team met with acknowledged the importance of legal monitoring provided under SMAP.27 According to their feedback, this activity helped partners to better understand and navigate the legal environment, exercise their rights, and inform (i.e., provide an evidentiary basis for) various advocacy and lobbying efforts that were considered key successes of SMAP, including the Media Strategy. Throughout the evaluation, the team consistently heard reports about the lack of political will to prioritize or pursue genuine media reforms, safeguard media independence, and provide fair market conditions, as well as the consequent drag this placed on advocacy efforts. At the time of the evaluation, many important issues and key pieces of the legal framework that the program had planned to influence remained to be addressed. For instance, more than a year after the adoption of the Media Strategy, none of the implementation deadlines had been met and none of the required pieces of legislation, including new laws on the concentration and transparency of media ownership, public information, public services, and electronic media, had been adopted. Moreover, the 2012 GoS budget included approximately $4.5 million for the direct financing of state media in direct contradiction of the principles of the Media Strategy.

IR 2: SUSTAINABILITY OF MEDIA PARTNERS Key Finding: The team found evidence that SMAP support under IR 2: Sustainability of Media Partners helped some, but not all, key media partners weather the economic crisis and put into place infrastructure, capacities, tools, and/or networks that have the potential to provide longer-term sustainability. Stakeholders cited the ongoing economic crisis as the key reason why the program fell short of its targets under this IR. SMAP helped introduce infrastructure, capacities, tools, and relationships that have the potential to improve long-term prospects for sustainability. Program activities under this IR included financial reviews, market research, business plans, development of online and radio advertising markets and networks, increased management and financial capacity, and the integration of new media into business models, which has created new platforms to reach consumers and generate revenue. The extent to which partner media outlets were able to fully utilize this assistance varied widely and depended upon various factors, such as the ability of local partners to absorb, apply, and own the technical assistance they received and whether the assistance was participatory and tailored to their organization. SMAP was able to bring together competitors, through various trade associations and networks, to work jointly to the benefit of their respective industries. SMAP was successful in fostering collaboration in all mediums, including:  Radio: The Radio Advertising Bureau (RAB) was able to bring in new advertisers for its members, reaching its Y2 target for increased radio advertising revenue (50 percent increase) within one year. According to program reports, in its first year of operation, RAB provided more support for its ac-

27 According to information provided by ANEM, as of September 2011, it had issued 35 monthly monitoring reports.

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM 12 tivities from its own resources than other associations supported by IREX. Radio Luna also established and operationalized a network of radio stations to achieve broader regional coverage of quality news and increase its leverage through joint sales operations.28  Online media: IAB, despite its fledgling status, gathered hundreds of advertisers, marketers, and media professionals through Digital Days. Between 2010 and 2011, online advertising increased 36 percent, albeit from very low levels.  Print media: ABC Serbia negotiated a Consolidation Agreement with 35 local media outlets and ad agencies to increase market share in local campaigns. Publishers agreed to prepare joint print issues and sell their advertising space as one. Local Press created a “joint editorial system”29 that allows its members to share content and quickly transfer print editions to online editions. The system increased efficiency, lowered costs, unified the newspapers, and strengthened negotiating power. Media partners, regardless of type, note that they continue to operate under significant financial pressure due to the severity of the economic crisis, lack of real market conditions, and political influence within the advertising sector. Several program partners spoke of being on the “edge of survival.” Most outlets reported having difficulty paying salaries or bills on time and those with active MDLF loans were months behind on payments, despite refinancing these loans several times. As noted under “Program Adjustments” above, while SMAP was able to help radio partners weather the economic crisis in the near team, the program’s efforts to bolster the survival of local television were not successful, despite improvements in program content and production. “IREX was helpful in looking at various business models around the world. They just couldn’t break the big barriers of business and politics in Serbia,” one investigative reporter explained. A review of SMAP’s performance monitoring plan (PMP) reporting under IR 2 confirmed the negative effect of the economic crisis on sustainability expressed during stakeholder interviews and suggested that partner radio stations are struggling to maintain past gains. This was most dramatically represented by the Media Sustainability Index (MSI) business management score. In 2001, Serbia’s MSI score for business management was 1.73 and by 2009, the MSI score had increased to 2.64, within the range (i.e., between 2 and 3) indicating that the media sector was nearing sustainability. By 2012, however, Serbia’s score had decreased to 1.71, signifying an unsustainable system. These scores indicate that the sector was nearing sustainability as the economic downturn began to take hold and has since become unsustainable.30

IR 3: ACCESS TO QUALITY INFORMATION Key Finding: Evidence points to notable achievements under IR 3: Access to Quality Information, particularly with respect to investigative reporting, building skills in niche reporting and technical capabilities, and diversifying platforms to deliver content to citizens. Results under this IR were mixed, however, as self- censorship continued to be an issue and most partner media outlets had difficulties consistently maintaining higher standards. Stakeholders indicated that outcomes under this IR were negatively affected by the ongoing economic crisis, which contributed to an exodus of trained journalists from the profession.

28 IREX Quarterly Program Report, 30 April 2012. 29 “Joint Editorial System” of Local Press is a misnomer. The system shares content but there is no prioritization or coordination of editorials no joint editing. This represents a missed opportunity as coordinated editorial views on various national issues could carry a heavy weight in Belgrade. 30 For an explanation of IREX’s MSI methodology and a more detailed elaboration of see: http://www.irex.org/resource/media- sustainability-index-msi-methodology. PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM 13 INVESTIGATIVE REPORTING Many stakeholders expressed their belief that IREX support to various media outlets to undertake investigative reporting is among the most important contributions under this IR. Virtually everyone the team met with praised the high-quality, hard-hitting journalism of the regional online publication Juzne Vesti, which has reported on nepotism, schemes to illegally claim public benefits, and abuses at public companies (for more information about Juzne Vesti, see text box below).31 Another leading online publication, Pistaljka, received 2,000 tips in its first two years of operation, of which 200 were worthy of further investigation.32 In addition, reporting by Sumadija Press on public finances received a United Nations award for investigative reporting.33 Major daily newspapers, TV, and radio have picked up stories by all three sites.34 On television, B92’s Insider series is also considered a leader in investigative journalism. Its report, Kolubara—The Plunder of the Country, on embezzlement in the nation’s coal mines attracted one-third of all viewers in the 10–11 p.m. time slot on February 7, 201135 and led to actions by the prosecutor’s office, the police, and parliament. While the Insider series is now self-sustaining due to advertising revenue, the three other online publications have a limited number of donors and face challenges in attracting revenue.

SUCCESS STORY: JUZNE VESTI For many government officials and media representatives, the Niš-based online publication Juzne Vesti (Southern News) stands out as a source of quality journalism and serves as a model for the future of Serbia’s media. Juzne Vesti’s success is based, in part, on its strategic attention to regional news. Despite its regional focus and a staff of twelve (half of whom work part-time), the online publication has an impressively large and growing audience. The number of unique monthly visitors grew from 151,549 to 353,927 after IREX provided funding and technical assistance for a new website. When Belgrade-based media want to know what is going on in the south, they turn to Juzne Vesti and frequently reprint its articles. Juzne Vesti also received technical support directly from USAID/Washington, which provided experts to set up a secure tips-submission portal, which was subsequently named Perun. The portal, which was launched in July 2012, aimed to build upon Juzne Vesti’s already successful track record in uncovering corruption. It was designed not only to protect whistleblowers, but also to build the brand of Juzne Vesti as an investigative publication. Despite these successes and ongoing growth in readership, Juzne Vesti realizes it needs to grow its revenue in order to become fully sustainable. It is in the process of putting into place the business recommendations developed in consultation with an SMAP consultant. These recommendations include introducing a pricing plan, reaching out to would-be audience members through traditional media, and advocating for changes in the way the municipality distributes funds meant for local media. Plans to introduce personal ads on the Juzne Vesti website are on hold during to various issues with online commerce in Serbia. The online publication currently has project funds from multiple donors.

QUALITY PROGRAMMING SMAP provided training (on occasion in cooperation with the OSCE) in basic journalism, journalistic ethics, investigative reporting, and a range of technical issues related to package production, presenting for television, radio formatting, and programming, and writing and editing for the web. SMAP also developed niche reporting skills on areas of importance to Serbian citizens and issues related to EU integration, consistent with the Mission’s strategy, including economic reporting, elections, financial markets, public finances, ag- ribusiness, EU accession issues, minority issues, persons with disabilities, corruption (in politics and the

31 According to IREX PMP reporting, Juzne Vesti met its target for online news content excellence in Y4. 32 By comparison, the Anti-Corruption Agency reportedly received about 200 tips in the past three years. 33 This reporting stemmed directly from IREX training “Eye on Public Finances.” 34 By Blic, Politika, Danas, Kurir, Alo, Pravda, B92 and/or Radio 021. 35 One out of every five viewers watching TV at the time of the broadcast watched the first Kolubara episode on January 31, 2011; the second Kolubara episode, which aired on February 7, 2011, brought in one in every three viewers. PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM 14 health and education sectors), the judiciary, and the energy sector.36 The team also found evidence of special programming dedicated to minority issues and themes related to nationalism and tolerance.37 While there are indications that the trainings strengthened capacity among participants and that media outlets benefitted from having their journalists, editors, and production teams take part in training programs, the economic crisis likely compromised long-term benefits as former participants increasingly left the media sector. Ac- cording to the former IREX CoP, “The economic crisis caused greater turnover in news departments and we almost had to start from scratch with reporters every six months. There was a lot of backsliding.” IREX’s Media Sustainability Index registered drops in the quality of professional journalism between 2008 and 2012. The 2012 score was 1.72, down slightly from the 2011 score of 1.74. To be considered “near sustainable,” a score higher than two is required. According to the 2012 report, the economic crisis and its cha- otic impact on the media market, low salaries, worsening ethical standards, and heightened self-censorship stifled any chance for improvement in the professional journalism score.38 The authors noted that the indicator for coverage of key events and issues represented the “most serious drop” and reflected the panelists’ “disappointment with the media’s missing coverage of the country’s serious political, economic, and social problems.”39 Program stakeholders consistently highlighted the negative effect of the economic crisis on the quality of programming. Several radio stations confirmed that they had reduced their staff by as much as 50 percent and were operating at “minimal levels.” Layoffs reduced SMAP partners’ ability to develop original content, undertake research and investigative reporting, and report from the field. A former IREX trainer noted the detrimental effect on quality, saying, “There is lot of copy-and-paste journalism.” IREX’s content analysis indicates improvements in story and program quality during the course of SMAP, particularly in Y3 and Y4, although there was considerable fluctuation of scores from quarter to quarter. This suggests that some stations had difficulty maintaining higher standards. With respect to story quality, none of the six radio stations monitored met their target, although several were able to achieve and maintain improvements between quarters. Only Radio Luna achieved an “adequate” score, which means the station was good enough for national broadcast. Of the six monitored radio stations, five achieved an “adequate” program quality score and four met their targets at some point during SMAP, even if they weren’t able to maintain those gains. Radio Zaječar and Radio Luna showed the most sustained program improvements when compared to the other stations. For more analysis on this issue, please see Annex H: Consistency of Story and Program Quality Standards.

SELF-CENSORSHIP Throughout the evaluation, program stakeholders repeatedly told the team about self-censorship in Serbia stemming from political influence vis-à-vis the media and within the advertising industry and the intimidation of journalists by means of physical violence, verbal threats, and lawsuits. One investigative journalist summed up the situation, saying, “There remains huge self-censorship in Serbia. No one does their job, at least not reliably.” Coverage of the 2012 presidential election provides a case in point. Various stakeholders the team met with criticized the election coverage, with some representing it as a complete failure of the media sector. Media monitoring reports from various institutions, including the OSCE’s Office of Demo- cratic Institutions and Human Rights (OSCE-ODIHR) and the Bureau for Social Research (BIRODI), found a lack of analytical and critical coverage of campaign and state authorities.40 According OSCE-ODIHR, however, there was a small number of online media and social networking sites that, although they had lim-

36 Training on economic reporting and television package and presenter skills was conducted with OSCE. SMAP also supported the development of training capacities within and the delivery of training by partner MSOs. 37 Most notably at the public service broadcast RTV, but also Radio Pirot (Roma programming), Boom 93 (nationalism and tolerance), and PG Mreza (special election programming featuring minority youths.) 38 Media Sustainability Index Report (2012), pp. 135-136. 39 Ibid. 40 2012 OSCE-ODIHR Limited Election Observation Mission Final Report: Republic of Serbia Parliamentary and Early Presiden- tial Elections, May 2012, p. 15. The OSCE report also references the situation with the advertising market and to pressure exerted on media by political parties and authorities, particularly at the local level. PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM 15 ited impact, provided more critical coverage.41 IREX conducted content analysis of five special election programs funded by SMAP. Two of these programs (produced by MTV Reporters and PG Mreža), attained a score of 4 out of 5, signifying that the programs “added value.” IREX did not, however, include election coverage as part of its ongoing content analysis of radio partners. Only one analyst reported on election coverage, which was characterized as being done “according to political parties’ press releases without deeper research.”42

REGIONAL AND GENDER CONSIDERATIONS Key Finding: SMAP partners were in broad agreement about the nature and scope of external economic and political factors undermining media development in Serbia and about the value of SMAP assistance, regardless of region or gender. Women media managers credited the program with helping to maintain a focus on gender in a context where virtually all the attention of media partners was on economic survival. In response to the Mission’s question about trends and differences based on region and gender, feedback from stakeholder interviews suggests that there are more similarities than differences. The team found wide- spread agreement about the external economic and political factors negatively affecting media development and the efficacy of SMAP. Regardless of location, SMAP’s partners reported being under significant financial pressure, which affected their ability to pay salaries and bills on time, required reductions in staff, and constrained their research and reporting capabilities. They also consistently cited examples of political influence (by politicians at various levels and in a few cases involving infighting among politicians within the same party) within the media sphere that extended to advertising, whether via advertising firms or local companies seeking to buy ads directly. MSOs and media outlets the team interviewed had differing abilities to successfully apply the knowledge, skills, and technologies provided by SMAP. This appeared to be influenced more by local organizations’ varying capacity to absorb assistance and by the individual personalities involved (due to personal qualities, such as vision, entrepreneurial spirit, and willingness to work in coalitions and networks) than on regional considerations.43 With respect to gender, SMAP activities included targeted support to female media managers through networking and professional development opportunities, including training on leadership and management, briefings on issues and trends within the sector, and confidence-building activities. The women the team interviewed repeatedly emphasized that they highly valued this support. SMAP partners and beneficiaries included female journalists, senior- and mid-level positions within media outlets (e.g., editors-in-chief and editors), heads of MSOs, government officials (telecommunications and the digital conversion), and executive positions within market research firms. This group stressed the active role of women in the media sector and the success of many SMAP summer school participants. However, they also noted that this was an “un- usual” (i.e., small, elite) group and that women still confronted obstacles, including cultural mindsets, to professional advancement and media ownership. The women also pointed out that the most important issue facing the sector is the economic crisis and its negative effect on a range of issues regardless of gender, including professional journalism and media sustainability. “Everyone is struggling,” one media expert told the team. The group was skeptical that there would be much focus on gender issues within the sector once the SMAP program ends, given the economic and political difficulties that media outlets face.

41 Ibid. 42 IREX Quarterly Report, April 30, 2012. 43 Due to time constraints, the team was not able to travel to areas further to the south and southwest (e.g., Novi Pazar or Sandžak) or the far north (Subotica), which may have affected findings on regional differences. The findings represent input from partners based in Belgrade, Novi Sad, Niš, Kragujevac, Požarevac, Pirot, Vranje, Čačak, and Užice. PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM 16 PROJECT MANAGEMENT AND IMPLEMENTATION Key Finding: SMAP assistance reached many activities and beneficiaries, including public institutions, media outlets (e.g., TV, radio, print, and web-based), news agencies, production houses, press centers, research firms, and membership and trade associations and was highly valued by partners and beneficiaries.44 Key Finding: Multiple stakeholders familiar with the management of the program indicated that the implementation of SMAP was flexible but tilted more toward an opportunistic than a strategic approach. Key Finding: Multiple stakeholders familiar with program management issues identified shortcomings in IREX’s management of the program, including work planning, in-house capacity, and handling of the leadership transition.

PROGRAM LEADERSHIP AND APPROACH All local partners and beneficiaries the team met with highly appreciated the assistance they received under SMAP, regardless of activity type. They described USG media assistance via SMAP as being practical, flexible, timely, necessary, and in line with their own priorities. Program beneficiaries also positively viewed SMAP’s focus on capacity building and its accessibility by new and small groups as well as outlets outside of Belgrade. SMAP provided technical assistance, training, mentoring and coaching, subgrants (including in-kind grants) research, coalition-building, networking activities, and study tours. (For more on the value of various types of SMAP activities, please see Annex I: Types of Assistance under SMAP). Core SMAP partners received multiple forms of assistance in various technical areas, while other beneficiaries received limited assistance, (e.g., training). As a whole, evidence suggested that individualized approaches brought value to the program. The team received relatively more mixed feedback on two approaches, training, and subgranting. SMAP partners and beneficiaries indicated that tailored and in-depth training, combined with mentoring/coaching on-site, was more effective than off-site, group, and one-off training. While partners and beneficiaries provided positive feedback about the grants that they received through SMAP, multiple sources familiar with program management issues, including but not limited to the Mission (both current and past), questioned whether the provision of subgrants was sufficiently linked to the achievement of benchmarks, (e.g., related to organizational development among core partners). There were also indications that discrete activities were not always value additive and did not contribute to broader program synergies. The first SMAP CoP was frequently cited as having a uniquely influential role in the overall program approach. This individual, the SMAP CoP from 2008 to 2011, also served as IREX’s CoP in Serbia for the ProMedia program from 1998 to 2001, when Milošević was still in power and donors and their Serbian media partners operated under a very different set of circumstances. Virtually all stakeholders the team met with acknowledged the former CoP’s robust knowledge of the Serbian media sector, vast experience in the country, and trusted relationships with a wide range of media actors, many of these relationships dating back to the 1990s. Many also acknowledged that it was helpful to have a CoP that the partners knew and accepted. These attributes were seen as contributing to program successes, including SMAP’s key legacy, the Me- dia Coalition. In the minds of most program stakeholders, the initial CoP was synonymous with IREX/Serbia. While noting the CoP’s many attributes and contributions, multiple stakeholders who were familiar with program management issues pointed to shortcomings in this area, including insufficient program management skills, a preference to work opportunistically rather than strategically, a reluctance to move forward with partners on sustainability issues, and a skill set and historical frame of reference that may not have been best suited to a program that had moved from the political orientation of the 1990s to the imperative of business development and sustainability planning under SMAP. During the evaluation, the former CoP

44 The term beneficiary is used in reference to all individuals and entities assisted/engaged in some way by SMAP (via training, advising, mentoring/coaching, networking, research, subgrants (financial and in-kind, e.g., commodities), and study tours). It is not limited to the core program partners or subgrantees. PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM 17 acknowledged that the program looked at traditional solutions for too long, with respect to business planning, and held on to traditional partners for too long, with respect to local TV. Another senior representative of IREX said that the program didn’t push partners hard enough and didn’t pursue alternative partners (i.e., those that were more entrepreneurial) or new initiatives (i.e., digital and new media) early enough in the program. Multiple representatives of the Mission provided feedback that further reinforced this assessment.

WORK PLANNING The reactive approach to SMAP was evident in the work planning process throughout the life of the program, including work plans, modification requests, and the work plan for proposed extension-period activities. A review of draft work plans and related Mission comments and communications reinforced what stakeholders who were familiar with program management issues told the team about a lack of detailed activity descriptions and unclear indications of the program’s intended strategic progression.45 In a few cases, work plans established a direct link between specific planned activities and monitoring and evaluation data and/or feedback from participatory agenda setting,46 although this was not done consistently. The team found that Mission requests for more specificity in work plans and modification requests were a consistent feature of the four-year program irrespective of the Agreement Officer’s Representative (AOR) or activity manager involved, and that significant time was invested in finalizing and approving work plans. Feedback provided to the team also indicates that the Mission ultimately approved documents that it felt still contained insufficient forward-planning details, despite revisions made by IREX. SMAP work planning, and in some cases approvals, was also affected by persistent problems in the provision of accurate and up-to-date pipeline data by IREX to the Mission.

OUTSOURCING SMAP relied heavily on outsourcing. SMAP’s in-house technical capacity was limited to the former CoP. Local staff served as program coordinators, but did not have the professional qualifications to be trainers and advisors, in contrast to ProMedia II staff members.47 To provide additional technical support and in keeping with the original program design, IREX used short-term individual consultants to implement ongoing technical assistance and contracted out certain parts of SMAP activities to other organizations. According to IREX, this allowed SMAP to engage high-quality experts who would not have been available to SMAP on a full-time basis. The approach allowed IREX to keep its staff size and costs to a minimum. Yet, while praising the quality of many of the consultants, several partners told the team that more in-house expertise at IREX would have been helpful. According to one MSO leader, “it would have been better if there had been more people here day–to-day to provide technical advice rather than having consultants come back and forth.” Evidence also suggested that outsourcing (e.g., to ECG for business development and Inbox for website development) resulted in SMAP doing things for partners rather than building their in-house capacity, at the cost of sustainability of program achievements.48 In the case of ANEM, their reliance on a for-profit law firm to monitor the legal environment for media meant that in-house capacity was not built in this key service line. The sustainability of legal monitoring also continues to depend upon donor funding.49

45 While the Y3 and Y4 work plans did contain greater specificity, this stemmed primarily from information on past activities rather than activities planned for the upcoming year. 46 However, SMAP partners were not consulted in the development of proposed activities for the NCE. Positive examples of participatory agenda-setting included the prioritization of joint activities of the Media Coalition, and activities stemming from the SNA in the approved NCE work plan and with respect to women media managers. 47 The ProMedia Office closed down at one point in 2007 before the extension of ProMedia II that until SMAP began. During the hiatus, IREX Serbia lost many of its staff. Lower levels of funding for SMAP as compared to ProMedia II were seen as contributing to the shift from professional staff capable of training and advising (as under ProMedia II) to activity managers (as under SMAP). 48 For more discussion of Inbox, please see page 9. 49 The legal interns within the law firm involved in the monitoring activity did build their knowledge of media law and this should contribute to greater media specialization within the legal profession, presuming they remain in this field. However, this is not the same thing as building capacity within a partner, particularly a legacy partner. Moreover, USAID strongly encouraged both IREX and ANEM to use funding in support of monitoring activities to also build ANEM’s in-house capacity to conduct this type of work, but this was not done. PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM 18 LEADERSHIP TRANSITION SMAP underwent a difficult leadership transition during Y4. From the outset of the program, IREX had planned and budgeted for a transition to local leadership of SMAP during a later phase of the program, and the Deputy Chief of Party (DCoP) had been designated to assume the CoP position. This was reaffirmed at several points throughout the program.50 Both USAID/Serbia and IREX acknowledged ongoing discussions beginning in the spring of 2011 about the leadership transition, including concerns raised by multiple representatives of the Mission to the CoP about the suitability of the proposed successor, and assurances by IREX of the readiness of the DCoP to assume leadership of SMAP; however, the Mission and IREX differed on the directness and specificity of the concerns raised throughout the spring and summer of 2011.51 During a conference call on September 28, 2011, the Mission reiterated its concerns and IREX reaffirmed that the DCoP was capable to run and close down the program.52 On September 30, IREX submitted its revised CoP extension and budget request to the Mission—including a request to appoint the DCoP as CoP effective De- cember 17, 2011—and restated that IREX would provide senior management support to the program in Y4 with specific attention to any strategic close-out issues.53 The Mission approved the CoP’s extension, but withheld consent on the proposed successor, indicating that it would reconsider the issue by mid-December. Despite its continued and serious reservations, the Mission ultimately approved the proposed CoP in De- cember 2011, based on repeated assurances provided by the CoP and IREX headquarters about readiness of the DCoP to assume the duties of the CoP. USAID/Serbia’s Democracy and Governance (DG) team’s limited exposure of to the DCoP made it difficult to gauge her skills in program and personnel management or involvement in overall decision-making.54 The evaluation team’s review of notes from conference calls and meetings, and e-mail correspondence from the Mission to IREX beginning in January 2012—after the new CoP was installed—indicate that the Mission had protracted communication with IREX HQ about various issues (i.e., the aforementioned high-level support by IREX HQ and, subsequently, the on-site provision of a full-time, expatriate senior advisor). In late April 2012, the Mission requested that IREX HQ initiate the provision of high-level program support to the Belgrade office, which IREX HQ had promised to supply. Throughout the late spring and summer 2012, IREX provided rotating DC- and regionally-based staff on short-term assignments in Belgrade. A full-time, expatriate senior advisor was not deployed until August 2012. According to Mission representatives, USAID/Serbia requested that the DCoP be trained and prepared to assume leadership of SMAP during the initial CoP’s extended tenure. According to the initial CoP, the DCoP was instructed on budgeting and reporting, provided with greater exposure to IREX HQ at international events, and given increasing responsibilities for parts of the program. Weekly debriefs between the CoP and DCoP beginning in Y2 reportedly addressed issues such as personnel matters, the budget and budget tracking, and USAID relations. While in Serbia, however, the team found evidence of several serious problems with program management under the new CoP, including communications and coordination with the Mission and other key implementers, personnel management, program implementation, and work planning.

50 Including as part of the budget revision that the program received for years three and four ($800,000) and requests to extend the CoP’s tenure through Y3 and December 2011 respectively. 51 In an e-mail communication with the evaluation team dated December 20, 2012, IREX confirmed that it did not document these early conversations in writing. The primary AOR also confirmed that concerns were initially (i.e., beginning in the spring of 2011) conveyed verbally by the Mission to IREX. The Mission did, however, provide the team with written documentation, in the form of briefing/conference call notes and e-mails dating from September 2011through July 2012 as evidence of its on-going communications with IREX on issues related to the leadership transition, its concerns about the proposed successor, and project management issues during Y4 of the project. 52 The conference call, held on September 28, 2011, involved the IREX vice-president, IREX contracting officer, IREX Serbia CoP, and the head of the Mission’s DG Office, the alternate AOR, and the activity manager. 53 According to IREX, the original request was submitted in July 2011. The revised request was based on additional information requested by the Mission as per e-mail communications on this issue provided by the Mission. 54 While the DCoP participated in some meetings with the Mission during that timeframe, multiple representatives of the DG Office indicated that the infrequent involvement of the DCoP in important discussions related to program planning and management was a cause for concern and factored into discussions of the leadership transition. PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM 19 USAID PROJECT MANAGEMENT There was considerable turnover of USAID/Serbia personnel during SMAP, including the Mission Director, AOR, and others within the DG Office, as well as among U.S. Embassy staff that were responsible for media-related issues. There were also indications that the USAID DG Office was stretched thin vis-à-vis the demands of managing its various programs, particularly when a member of the team was on leave, as was the case with the primary AOR for SMAP. At various times throughout SMAP’s four-year tenure, there were two different AORs (including the head of the DG office), an acting AOR, an alternate AOR, and an activity manager.55 Feedback provided to the evaluation team by both USAID and IREX as well as a few local partners suggested that personnel changes resulted in shifting program direction and priorities as well as conflicting communication and management styles throughout the life of the program. Multiple stakeholders familiar with program management issues characterized the Mission’s approach as gravitating between informal/permissive and assertive/strict.56 USAID and IREX offered differing perspectives on the effect of the Mission’s management. From USAID’s perspective, shortcomings in program management and a lack of responsiveness on the part of the implementer necessitated stricter management by the Mission. From IREX’s perspective, differing management approaches affected program implementation resulting in miscommunications57 and slowed approval processes, and in the case of the stricter approach, exceeded the substantial involvement clause of the cooperative agreement.58 While the language of the substantial involvement clause, and the very nature of cooperative agreements, limited the Mission’s options for exerting greater control over the program, the team found that there were grounds for the Mis- sion’s concerns about IREX’s management of the program (as discussed previously) and a need for stricter oversight. Regardless of the management approach employed, however, and as the findings under Work Planning and Leadership Transition above indicate, the Mission ultimately signed off on program documents and decisions with which it had lingering concerns. The team found that multiple factors affected the amount of time required for approvals, including increased oversight by the Mission and changes in personnel, as well as the sufficiency of requests, information, and program documentation submitted by the implementer. The team cannot reach a finding concerning misunderstandings or confusion stemming from verbal communications that were not documented in writing at the time or formally conveyed by the implementer to the Mission.59 Based on the written documentation and correspondence provided by the Mission, the team found its concerns and requests were consistent, if communicated with greater seriousness and urgency as problems materialized and/or persisted.60

55 The activity manager, who was not certified as an AOR at the time, served as the point of contact for the program when the primary AOR was on maternity/medical leave. On this issue, the head of the DG Office at the time, sent an e-mail to IREX confirming that if a written approval was forthcoming from the activity manager and the head of the DG Office was copied on the e-mail, this indicated concurrence with the decision and constituted documentation of approval by the head of the DG Office. IREX raised with the team its concerns about this approach, indicating that it sought retroactive reconfirmation of these approvals from the AOR. 56 The informal management approach was also characterized an involving a more deferential attitude toward the SMAP CoP. 57 Including among some program partners with whom the Mission had direct interaction, a point reinforced by several local partners. 58 The substantial involvement clause of the cooperative agreement required Mission approval of work plans and key personnel. The DG Office also required approval of SMAP subgrants to which IREX agreed. 59 Based on documentation (or the lack thereof) available to the team for review. 60 The level of written documentation provided by the Mission including correspondence and notes from meetings and phone calls far exceeded that provided by IREX – at the team’s request – which influenced the findings presented herein. PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM 20 CONCLUSIONS AND LESSONS LEARNED

CONCLUSIONS

OBSTACLES TO IMPLEMENTATION AND PROGRAM ADJUSTMENTS Conclusion: Given the scope of external obstacles SMAP confronted, the overarching goal of the program—namely, the transition of the Serbian media sector into a legally sound, economically viable system providing professional and independent news and information to citizens throughout Serbia—as well as some outcome indicators established in the PMP were overly ambitious.61 Given the timing of program design and procurement, neither USAID nor IREX could have envisioned the occurrence or scope of the worldwide economic crisis and its effects on the program’s ability to achieve and sustain anticipated results, particularly under IR 2 (sustainability of media partners) and IR 3 (access to quality information). The economic crisis dramatically changed the program environment; the effects of the crisis were so severe and protracted that they were beyond the resources and ability of the program to overcome. The crisis accelerated an inevitable consolidation of Serbia’s oversaturated media market, contributing to the collapse of some local media partners (particularly local television), and resulted in backsliding on key business sustainability and professional journalism indicators. In this context, the goal of achieving an economically viable system proved unattainable, despite the program’s determined efforts. The program’s goal of a legally sound media sector overestimated the political will on the part of the government, led by the Democratic Party, among other political actors, to genuinely pursue genuine media reforms (both de jure and de facto) and to respect media independence in Serbia. Lagging political will was further compounded by the lack of an informed and engaged public and weak links62 between advocacy- oriented MSOs and CSOs that shared common interests in free speech, media independence, transparency, and accountability. This limited the scope and, therefore, the influence of the reform constituency. Interna- tional prioritization and support of a free press agenda, including privatization, transparency and consolidation of ownership, and state ownership media is extremely important; however, absent political will and public engagement in country, international assistance is limited in what it can achieve. Conclusion: Adjustments to the original program design were essential for SMAP’s continued relevance. While the diversification of program partners created the possibility to focus assistance on media outlets with the greatest prospects of success, program management appeared slow to move away from some unsustainable partners. Given the dynamic nature of Serbia’s media landscape, the definition of media partners needed to be expanded to allow the program to adapt to changing circumstances on the ground, to enable cooperation with a range of media outlets important to local communities, and to make determinations predicated on the relative financial viability of various types of media. By focusing on media outlets, the program was free to work with local television as envisioned in the original program design as well as to pursue partners in radio, print, and online media. Once it became clear that local television was losing viewers to national stations (as well as to cable and satellite television) and given the growth of radio and the internet, the diversification of media partners proved even more essential to ensure the program’s relevance. This adjustment also created the possibility for the program to focus on media partners with the greatest prospects of success—and where

61 In and of itself, the PMP was highly ambitious, totaling 64 pages and nearly 30 output and outcome indicators. This placed a considerable burden on SMAP staff to collect, analyze, and report data, not all of which was useful to the program or capable of being absorbed by the Mission. 62 As evidenced by the SNA of MSOs and CSOs carried out by Social Impact on behalf of IREX in Y4 of SMAP. PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM 21 the program could make a difference—in a context where the combined pressures of the economic crisis, digitalization, increased competition, unfair market conditions, and an oversaturated market seriously undermined the viability of local television. The expanded definition of potential partners provided SMAP with the options it needed to seize new opportunities and pursue effective programming. Despite the priority given to television in the original cooperative agreement, holding on to local television partners once it became clear that they were losing viewers was one of the mistakes of the program, a view shared by the CoP. IREX was slow to retract its support from local television partners, some of which continued to receive assistance into the latter half of the program. Given the changing media landscape in Ser- bia, SMAP should have transitioned away from local television partners sooner and concentrated on radio and web-based media outlets. Ultimately, local television partners faced obstacles so large that no amount of training, advice, or subgrants could have ensured their survival, despite the good intentions of the immediate beneficiaries and incremental improvements in the quality of program content and production. Given the high costs of operating a television station, IREX support could not have offset the debt, falling revenue, and other adverse conditions faced by local television partners. In this sense, the shift to radio was a sound move as a small grant to a local radio partner could achieve significantly more than an equivalent grant to a television station, given lower operating costs. It remains to be seen, however, whether all SMAP radio and web partners will survive the economic crisis in the long run.

Conclusion: Despite the leading role played by European donors vis-à-vis the Government of Serbia’s digital conversion process, the potential of digitalization to undermine local media diversity and sustainability and, thereby, USAID’s development priorities, values, and past investments justified SMAP engagement of decision-makers and processes. Given the role that digital conversion will play in transforming Serbia’s media landscape, SMAP engagement of decision-makers and processes on this issue, although limited, was justified and helped to facilitate a more informed policy approach to digital conversion. The digital conversion process is behind schedule and, though this is not IREX’s fault, the program was not able to proceed with all planned assistance. As noted in the findings, the comparative and practical knowledge gained through IREX assistance is likely to continue to be applied beyond the life of the program as virtually all program participants remain in decision-making or advisory positions despite the change of government. As such, the program leaves behind some positive effects even though the situation is complex and much work remains to be done. Since the way in which the digitalization process proceeds could threaten USAID’s development priorities, values, and past investments, further U.S. involvement would be warranted and welcomed by responsible authorities, whether at a policy level, through collaboration with other donors, or discrete activities. Conclusion: The expansion of new media activities was the right decision given the rapid growth of the internet in Serbia and media partners’ limited knowledge of and experience with the web and mobile technologies. Assistance in this area helped partners to understand the importance of new media and produced tangible results under all three IRs. However, the program could have done more to help partners further achieve and own conceptual and structural advances. Given the rapid growth of the internet and increasing penetration of mobile phones in Serbia between 2008 and 2012, the inclusion of new media activities in the original program design, the expansion of the definition of partners at the outset of the program, and the decision in 2010 to place greater priority on new media within SMAP were essential to address fundamentally shifting journalistic and business realities. SMAP made great strides in helping partners understand the importance of new media and produced tangible results regardless of media type or partner across all three IRs. The combination of market research, training, networking activities involving marketing and advertising professionals, and support aimed at improving the quality of partner websites and increasing their online presence helped to develop Serbia’s nascent online advertising market and created the prospect of new revenue streams, even if these have yet to be realized by all the program partners. The involvement of advertisers, although not envisioned in the original program design, provided for a more holistic and integrated approach to financial viability. SMAP also supported a PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM 22 number of new media innovators, including legacy partners FoNet and Radio 021 and start-ups such as the online publications Juzne Vesti and Pistaljka. Still, IREX could have done more to help partners further achieve and own conceptual and structural advances utilizing the internet and mobile phones. The team understands the rationale behind IREX’s approach to upgrading partner websites, whereby IREX subcontracted to Inbox to provide partners with new sites, but we are concerned that partners’ in-house capacity may not have been sufficiently developed, especially since much of this assistance was provided in the final year of the program. As a result, some partners may un- derutilize this assistance over time and fail to devise their own creative uses for interactive tools. The program’s late focus on mobile technologies for most partners also meant that they will (presumably) launch Android applications and mobile-optimized versions of their websites only in the final months of the program when further IREX advice and support will no longer be available.63 Given their relatively low levels of knowledge and experience with new media, most media partners would also have benefitted from more iterative and in-depth training combined with ongoing mentoring across a broader range of new media topics in order to better prepare for evolving trends in journalism, information consumption, and business models.

INTERMEDIATE RESULTS

IR1: LEGAL AND REGULATORY FRAMEWORK Conclusion: Coalition-building efforts under SMAP were critical to helping the program’s MSO partners undertake more strategic and effective advocacy initiatives and have a seat at the table. At the same time, however, SMAP missed opportunities to further broaden the constituency and political will for media reform. The formation of the Media Coalition was a critical development within the program. By speaking with one voice, SMAP’s MSO partners were better able to protect their interests and exert greater influence on the Serbian government and its policies. The program’s ongoing support of joint activity is important to further build trust among the members, provide opportunities to achieve incremental and tangible results, and to further consolidate the coalition, even though it is likely to continue to operate on an informal basis for the foreseeable future. At this juncture, the Coalition faces a real test of its advocacy capacity as it attempts to lobby the new Serbian government to implement the Media Strategy. Presuming that the members’ leaders can continue to focus on mutual interests, the Media Coalition appears poised to hold together after the departure of IREX and to sustain its advocacy activities. The main concern, given the importance of individual personalities to organizations (as discussed below), is whether any future leadership changes within either of the journalists’ associations might negatively affect continued collaboration. While recognizing the major achievement signified by the formation of the Media Coalition, the team believes that a broader constituency for media reform is needed to most effectively overcome lagging political will in Serbia, and that SMAP missed several opportunities to push political will along. Specifically, the program would have benefitted from a large-scale and sustained public relations campaign throughout the course of the program to promote the idea of a free press as a public good, to help mobilize broad-based public opinion and activism around media reform, and to further influence public policy agenda-setting on this issue. As the results of the program’s social network analysis (SNA) showed, MSOs also exhibited relatively weak connections to CSOs with common interests, including freedom of speech, independent media, transparency, and accountability. Efforts to facilitate networking and joint action between MSOs and CSOs, however configured, should have been pursued earlier in the program. Such efforts would have bolstered MSO advocacy and lobbying efforts under IR 1 and helped to move SMAP further toward its stated goals.

63 As noted in the Findings section, only Radio 021, which had been receiving new media support under SMAP from YI, had launched a mobile-optimized version of its website two weeks before the evaluation team’s visit with just one quarter left to the program. PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM 23 IR 2: SUSTAINABILITY OF MEDIA PARTNERS Conclusion: By getting competitors to appreciate their common interests and work together to the benefit of the industry, SMAP was able to achieve a multiplier effect that will continue beyond the life of the program. This was an effective approach and a significant achievement of SMAP. Getting competitors to appreciate their common interests and to work together is an approach that is neither easy nor quick, but one that is essential in an environment of declining media assistance. SMAP partners are transferring to other organizations who did not directly participate in program activities, and outlets with greater capacity are providing leadership and support to outlets with less capacity. As such, SMAP achieved an important multiplier effect that will continue beyond the life of the program. In addition, efforts by various trade groups to establish relationships among media outlets, advertising agencies, and marketing professionals—combined with the availability of market research, described in more detail below—are critical to attracting increased advertising to radio and print outlets in local markets and to online media. Partners like RAB and IAB represent the type of entrepreneurial spirit and forward-looking approach that is greatly needed within a sector that had become used to generous funding from donors supportive of their desire for profound political change. Despite the effort dedicated to new media activities in the latter half of the program, it is unfortunate that advertisers were not brought into the training program sooner. Conclusion: The provision of market research was essential to better understanding the market position of local partners, to establish the importance of decision-making and planning on an evidentiary basis, and to approach advertisers. SMAP’s investment in market research was key to efforts to improve prospects of media sustainability by providing an evidentiary basis for planning and decision-making on the part of trade associations and individual media outlets. This research contributed to a marked shift in attitudes and behaviors among SMAP’s media partners, moving them from distrust of data and fear of what the results might indicate to an understanding of the importance of this information to the survival and success of their business. Given that SMAP was the only media assistance program funding market research for programmatic purposes, the team has concerns about the near-term sustainability of this activity given the current economic climate.64 While a few of the program’s partner research firms and trade associations indicated that they intend to continue with research absent IREX support, it is likely that this will be less frequent and involve smaller sample siz- es. Conclusion: Training provided by SMAP was not sufficient to ensure the success of media partners. Tai- lored mentoring and coaching proved more effective and should have been employed more routinely and thoroughly by the program. Participatory approaches to business and sustainability planning also proved to be more conducive to partner ownership than having external experts present business plans on the basis of data provided by local partners. Providing the necessary training elements (e.g., journalism training, management training, sales training, and preparing a business plan) for partners won’t guarantee their success. Given the idiosyncrasies of local and regional markets, the effects of the ongoing financial crisis, and the tectonic shift brought to the media space by the internet and mobile technologies, SMAP could have achieved greater results if it had provided even more direct hands-on mentoring and one-on-one coaching to media partners, particularly in business and sustainability planning. By outsourcing its work related to business development, IREX was not in a position to provide day-to-day support under this core component of the program. In some cases, outsourcing also contributed to a nonparticipatory approach that undermined partner ownership of and follow through on business plans.

64 According to IREX, concerns about sustainability were among the reasons that the program moved from diary to computer- assisted telephone interface (CATI) and the use of internet panels. PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM 24 IR 3: ACCESS TO QUALITY INFORAMTION Conclusion: IREX could not have predicted the negative effect that the economic crisis would have on the quality of professional journalism. The subsequent exodus of trained journalists from the profession exposed the need for more sustainable training solutions. Training was a core component of SMAP, both directly by IREX and indirectly via MSOs, and topics ranged from journalism ethics and niche reporting to technical skills and new media. While IREX could not have predicted the economic crisis and the resulting exodus of trained journalists from the profession, the crisis exposed the need for sustainable training solutions. As noted in the Findings section above, IREX found itself having to “start from scratch” every six months to train new reporters. As media outlets continue to operate under significant financial pressure, the need to train new waves of journalists will likely continue. While SMAP has created some training capacity within MSOs as a means of supporting sector development and generating revenue, the number of topics upon which partners can train remains relatively limited and basic. As such, it appears that MSOs will not be able to meet the training needs of the media sector in the near term. In hindsight, the SMAP might have done more to diversify and expand training capacities within the MSOs/training corps and better leverage universities65 to provide more diverse and sustainable training options. Conclusion: Despite the reporting excellence and public legitimacy of the program’s investigative journalism partners, online publications primarily dedicated to investigative reports will likely remain dependent upon donor funds for the foreseeable future. While it is clear that the program’s investigative journalism partners achieved excellence in reporting as well as legitimacy among their respective audiences and peers, this will not necessarily translate into improved prospects for advertising revenues or funding from domestic sources. Public institutions that could become the subject of investigative reports are unlikely to provide subsidies to the publications. Moreover, private companies are likely to consider whistleblower sites too risky and politically controversial to invest in spon- sorship or advertising. For their part, online publications dedicated to investigative reporting are wary of public or corporate funding that has the potential to compromise actual or perceived journalistic integrity. In this respect, Serbia is not unlike other transitional democracies where investigative reporting remains a difficult and dangerous undertaking. Given the fact that online publications are pursuing high-quality, hard- hitting journalism at a time when the quality of journalism within most traditional media outlets is declining and since the internet offers comparatively greater freedom of speech in Serbia, this is an area where continued donor attention and support could add value.

PROJECT MANAGEMENT AND IMPLEMENTATION

PROJECT LEADERSHIP AND APPROACH Conclusion: Shortcomings in IREX program management and technical leadership adversely affected program implementation and progress toward expected results, particularly during Y4. Conclusion: Due to external and internal factors, SMAP primarily pursued an opportunistic approach. More strategically oriented and focused management could have strengthened program effectiveness despite external obstacles to implementation. While the situation in Serbia required continued programmatic flexibility, SMAP’s opportunistic approach produced mixed results. On one hand, IREX was responsive to its local partners’ needs, contributing to locally driven and locally owned processes. On the other hand, SMAP spent a great deal of its resources on meeting immediate needs at the expense of pursuing a longer-term focus that involved more in-depth, day- to-day technical support to those media partners that had the best chances of surviving the financial crisis and demonstrated an entrepreneurial spirit and ability to adapt to and manage change. This longer-term focus would have allowed IREX to emphasize higher-level technical skills and support oriented around sound

65 SMAP moved in this direction only during the final months of the program (with Singidunum University). PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM 25 business models, to ensure that partners had the internal capacity and strategic vision to sustain and build upon results once SMAP ended. In the case of the program’s local TV partners, for example, IREX never had enough resources to support them in the long run. The involvement of a respected CoP with long-standing relationships and experience in Serbia featured prominently in SMAP’s approach and had both positive and negative effects. On the one hand, it provided program stability and continuity, and evoked the trust of local partners, which was essential to achieving certain coalition-building efforts and activities requiring cooperation among competitors for the benefit of the sector. On the other hand, this history framed the CoP’s point of view vis-à-vis the management and implementation of SMAP and contributed to a reactive approach, hesitancy to leverage local partners’ trust in order to push them hard enough, and ongoing support to some partners that were not sustainable or sufficiently reorienting themselves to new realities. New leadership could have offered fresh perspectives the program needed to innovate and evolve, while enabling it to better exercise “tough love” in tandem with financial support to further strengthen program partners’ (both media outlets and MSOs) prospects for sustainability.

OUTSOURCING Conclusion: While extensive use of outsourcing may have provided for implementation efficiency, it came at the cost of in-house technical capacity and leadership, both within IREX/Serbia and, in some cases, media partners. The program would have benefitted from a combination of broader in-house technical expertise, both expatriate and local, and short-term consultants. While the range of SMAP partners and scope of activities necessitated the use of short-term consultants, having another expatriate on staff could have helped to fill gaps in program management and technical expertise. Areas that would have particularly benefitted from a second advisor include business and sustainability planning, association development, and new media activities. Broader in-house management and technical capacity might also have provided IREX with more insights into the business prospects of its partners, more intervention options, and better chances for a smooth leadership transition. Ongoing reliance on outsourcing to for-profit firms—such as ECG Media and Inbox by IREX and a private law firm by ANEM—came at the expense of in-house capacity and, potentially, organizational sustainability. In some of these cases, SMAP missed opportunities to incrementally build partners’ in-house capacities. For example, IREX could have worked to build the services provided by ECG Media (i.e., financial reviews, business planning, tracking funding opportunities, and training on how to apply for EU funds) within a particular MSO or across a corps of local trainers, thus providing for the development of additional member services and revenue streams.

LEADERSHIP TRANSITION Conclusion: IREX was slow to recognize the seriousness of problems with the leadership transition in Y4 and did not act quickly or decisively enough to respond to the Mission’s concerns and requests, and therefore was unable to minimize damage to program quality. Ineffective program management and technical leadership during Y4 adversely affected the functioning of SMAP’s Belgrade office as well as the quality of program planning and implementation. SMAP was dys- functional for nearly eight months66 at one of the most critical phases of the program, when assistance aimed to step-up advocacy for implementation of the Media Strategy vis-à-vis the new Serbian government, and to reinforce local partners’ prospects for sustainability in anticipation of program close-out. This situation could have been averted and/or mitigated if IREX had been more responsive to the Mission’s repeated and

66 The eight months started with the discussion between the IREX vice president for media and USAID/Serbia Mission Director and DG Office Director on January 26, 2012, to the arrival of the IREX’s senior media advisor in August 2012, and continuing into the time of the evaluation team’s visit, when the ramifications of problems with project and personnel management were still evident, although the new senior advisor was taking tangible steps to turn the situation around. PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM 26 reasonable statements of concern about the leadership transition and its requests for the provision of high- level technical support in Y4. Based on information provided by multiple sources, the team has to conclude that IREX’s efforts were not sufficient to routinely involve the DCoP in strategic decision-making and program planning, foster an effective working relationship with the DG Office team, or prepare the DCoP to successfully manage and close out the program.67 Given that the eventual elevation of the DCoP to the position of CoP was planned from the outset of the program, IREX had three years to ensure that the DCoP had the necessary leadership and program management skills, and to successfully transition the DCoP into the requisite roles and responsibilities. IREX’s handling of the leadership transition resulted in a deterioration of both program quality and donor satisfaction.

USAID PROGRAM MANAGEMENT Conclusion: USAID’s program management was alternately permissive and strict in its oversight of SMAP. In general, the Mission inconsistently and under-utilized its authority over standard program management tools, such as work plans and approval of key personnel. The informal and permissive approach to project management at the outset of the program set a precedent and contributed to IREX’s expectations of the Mission’s management moving forward. This approach led to more stringent oversight exercised by the DG Office later in the program as problems began to emerge and played a role in the difficulties the Mission encountered in getting IREX to take its concerns seriously. While cognizant of the limits placed on the Mission by the substantial involvement clause of the cooperative agreement, the team concludes that more assertive and consistent program management was needed from the outset of the program. Despite the Mission’s continued misgivings about or dissatisfaction with SMAP work plans, the NCE, and the proposed CoP for Y4, it ultimately signed off on these program documents and decisions given the need to move forward with program implementation. In the case of planning documents, although USAID consistently requested more forward-planning details and engaged IREX program management toward this end, it ultimately compromised on technical quality. In the case of the leadership transition, the Mission had the authority to deny approval of IREX’s proposed CoP. However, the bar is not clearly set for denial of a CoP which contributes to differing interpretations as in this case, i.e., whether serious and on-going concerns were sufficient or whether evidence of poor program management and personnel skills was required. It is possible that the program would have been better served if the Mission had denied IREX’s proposed CoP or made its approval contingent upon the provision of a senior media advisor (i.e., rather than leaving this to a later date). Even if the Mission had insisted upon the presence of a senior media advisor sooner than it did, the team acknowledges that program disruptions would likely still have occurred given IREX’s difficulties in rapidly identifying and securing full-time field personnel.

PROGRAM MANAGEMENT LESSONS LEARNED Based on the findings and conclusions presented throughout this report, the team has identified the following lessons learned as they relate to program management and implementation: 1. Including the possibility of working with a variety of types of media into program design (such as in the RFA and cooperative agreement) allows a program to respond both to local developments and broader trends within the media industry, even if a particular medium is planned to be the primary beneficiary. Anticipating the need to provide training for advertising professionals at the program design stage provides for a holistic and integrated approach to improving financial viability. 2. In-house technical capacity not only makes program implementation more efficient, but also en- hances the program’s ability to help local partners more regularly and more quickly.

67 There was concurrence on this within the Mission among various individuals responsible for project management and oversight and with reinforcing input provided be sources outside of the Mission. On the issue of routine involvement of the DCoP vis-à-vis the Mission, the team is referring to direct discussions between the Mission and IREX Serbia. PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM 27 3. Programs require flexibility to respond to local conditions and developments on the ground but need to balance opportunistic and strategic approaches to consolidate results. To ensure adequate program management and implementation, planning documents need to present details that establish the strategic orientation and progression of the program. 4. From the outset of a program, an implementer’s headquarters needs to be sufficiently engaged with the field and the donor to move quickly to limit damage to program quality and donor relations as problems develop. 5. Transition of program leadership to local staff should be not be undertaken until local staff have the necessary skills. If a leadership transition is planned, the implementer should undertake capacity building from the outset. 6. When confronted by ongoing concerns or evidence of poor leadership and an inadequate response by the implementer, the donor should move decisively to deny proposed key personnel/request their removal. 7. The profound changes facing the media sector due to increasing internet and mobile connectivity require implementers to plan farther into the future and utilize cutting-edge technical and economic expertise in designing, planning, and implementing media assistance programs. 8. Subgrants have a limited ability to promote or secure partners’ financial viability. If a local partner is facing an inevitable closure, the implementer must present a clear and strong case for providing continued technical or financial assistance before any subgrants to the partner are approved. 9. Tailored, hands-on, on-site advisory services, such as mentoring and coaching, add value to other activities, including training and subgrants, by helping to ensure that new knowledge, skills, and technologies are synthesized in a way that maximizes the efficacy of assistance for each beneficiary. 10. Fostering cooperation among MSOs on the basis of mutual interests is hard and takes time, but can lead to positive outcomes that benefit MSOs as well as the sector as a whole. 11. Research provides a valuable evidentiary basis for planning and decision-making both by the implementer and by local partners, and can contribute to fundamental shifts in attitude and behavior.

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM 28 RECOMMENDATIONS

Given the timing of the evaluation vis-à-vis the significant and protracted effect of the worldwide economic crisis on the Serbian economy, including the media sector, the Mission may want to consider conducting a review in two to three years to reassess the longer-term effects of SMAP. In particular, this review should focus on results under IRs 2 and 3, which were most adversely affected by economic conditions as evidenced by backsliding on key indicators. Once the economy turns around, local partners should be better able to leverage the knowledge, skills, and tools gained through the program to improve their financial viability and, thereby, to dedicate resources to (re)hiring trained journalists, improving the quality of news and information programming,68 and producing original content.

Although SMAP ended in December 2012, the media sector will continue to confront multiple challenges. Moving forward, the Mission remains committed to advancing media reform in Serbia and has multiple options and opportunities to pursue this goal through existing programs and direct funding to local organizations. These efforts can be further supported by U.S. Government engagement at a policy level to ensure an independent, diverse, and professional media sector in Serbia. In fact, there are several key policy areas where USAID—indeed the entire U.S. Government—should continue to focus efforts and available resources. Many of the challenges facing the Serbian media sector are not new—in fact, SMAP explicitly targeted many of these challenges—but rather are endemic and pervasive and therefore likely not to be fully resolved by a single assistance program. These challenges, many of which relate to Serbia’s slow march toward EU accession, the global economic crisis, and evolving technology, include: 1. Digital conversion. A host of discrete issues are critical to Serbia’s transition from analog to digital television broadcasting. The United States should attempt to understand and keep track of digitalization issues as the Serbian government’s plans become clear. It is possible that the U.S. Government, both in Serbia and the United States, could play a positive role in ensuring a smooth, efficient conversion process, potentially including new assistance. The U.S. ability and willingness to independently voice concerns about media freedom and diversity should be exercised more frequently and with more coordination between USAID and the U.S. Embassy. 2. Transparency of ownership and political capture of media. While it would be useful to know who owns specific Serbian media outlets, this knowledge alone will not solve the main issues underlying media independence. Major political forces continue to exert undue influence on Serbian media and advertising sectors, which stifles political debate and democratic reform. Even though USAID has supported media development programs in Serbia for the past 15 years, these political forces remain opaque and entrenched. With its media program ending, the U.S. Government should now increase diplomatic efforts to attempt to resolve this issue, keeping in mind the central role the media plays in ensuring democratic reforms. 3. Divestment of the state from media ownership and increasing government support for public media. As evidenced by its plan for regional public service stations and behavior over the past several years, the Serbian government appears to have given up on media privatization. USAID should press the GoS to follow its own policies of privatization and advertising, both of which would have a significant effect on the media sector. Despite longstanding state support for local media and the fact that few in Serbia appreciate the concept of donating money to support local media, USAID should support a drastic change to the status quo, in which subsidized public media compete for private advertising revenue and broadcast more advertisements per hour than allowed by law, i.e., local media should either com-

68 For example, local partners could be in a better position to expend more funds on research, investigative reporting, and field-based reporting.

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM 29 pete fairly for public funds via transparent and accountable procurement processes and based on outlets’ tangible commitment to public service. 4. Public awareness and media literacy. Stakeholder feedback suggests a high level of public misperception and uninformed expectations of the media. Without informed and broad-based public support for a free press agenda, efforts directed at media reform are likely to only be effective at the margins. USAID should consider programs that expand the constituency for media reform and raise public awareness of and appreciation for the role of the media, media ethics, and accurate, fair, and balanced reporting. Ser- bian citizens also need media literacy programs to help them better access, analyze, create, and interact with media in various forms to build the skills of inquiry and self-expression that are essential to democracy. In addition to engagement at the policy level, there are several opportunities to continue to advance media reform through existing democracy and governance programs or direct funding of local organizations: 5. The recently awarded Civil Society Forward program can serve as a vehicle for further support of the media sector. The program’s objectives—policy advocacy, CSO and network capacity, and government efforts related to the legal and regulatory environment for civil society—are compatible with various media reform activities. In particular, Media Coalition efforts to foster cooperation between MSOs and undertake broad-based advocacy on issues of mutual interest appear particularly well suited to this program. Along the same lines, this program could be utilized to support recent overtures by the Govern- ment Office on Cooperation with Civil Society to support joint activities by media and CSOs. Media activities carried out as part of this program should provide multiplier effects that benefit the sector as a whole. 6. Future assistance should leverage existing relationships and communication channels with political parties, the National Assembly, and Minority Councils to repeatedly and consistently convey the importance of independent and diverse media in Serbia. Future programs should work with the National Assembly to abandon the practice of adopting laws through special procedures, which would have direct and consequential effects not only for media reform but also for transparent and accountable governance. 7. Efforts by USAID, in tandem with the Commercial Section of the U.S. Embassy, related to digitalization, antimonopoly legislation, and the development of e-commerce as well as government procurement reform, could have important implications for the Serbian media sector and should be pursued. 8. In keeping with USAID Forward, efforts within the Mission to transition to direct funding of local partners provide further opportunities to support media reform. Emphasis should be placed on supporting organizations that can leverage past media investments, achieve a multiplier effect, and best position legacy organizations to sustain support to the sector upon USAID’s departure from Serbia. In keeping with the Mission’s current strategy and efforts to further facilitate Serbia’s movement toward Euro- Atlantic institutions, USAID may also consider direct funding to support the provision of quality news and information programs, particularly investigative reporting. Such support, however, may involve a short-term focus driven by immediate needs rather than ensuring longer-term sustainability.

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM 30 ANNEX A: STATEMENT OF WORK

SECTION C – DESCRIPTION / SPECIFICATIONS/STATEMENT OF

WORK “Performance Evaluation of the Serbia Media Assistance Program

(PE-SMAP)”

C.1 PURPOSE The purpose of this task order is to provide USAID/Serbia with a performance evaluation of the Serbia Media Assistance Program (SMAP) Cooperative Agreement Number 169-A-00-08-00101-00 implemented by the International Research and Exchanges Board (IREX). The purpose of the evaluation is to measure the change in development outcome that is attributable to the Program. Overall, the evaluation shall show how the Program’s performance influenced the Serbian media sector, but also on Program primary and secondary partners and other stakeholders; and how/ if the Program assisted Serbian citizens to have access to professional and independent news and information. The evaluation shall also show how the Recipient implemented the program to provide support to the transition of the Serbian media sector into a legally sound, economically viable system providing professional and independent news and information for citizens throughout Serbia. The evaluation will provide rigorous, evidence- based and independent analysis on IREX’s performance over the life of the program. C.2 BACKGROUND SMAP started on September 24, 2008 and has a current completion date of September 23, 2012. USAID funds obligated to date for the program are $8.8 million. Prior to SMAP, USAID supported two media assistance programs: ProMedia I (1997-1999) and ProMedia II (1999-2007). Both programs were implemented by IREX. USAID media assistance prior to 2000 focused on independent journalists, print, and electronic media in an effort to support the development of alternative sources of information and pro-democracy voices. After 2000, USAID media assistance shifted focus to ensure that these independent media were sufficiently equipped to deliver professional and objective information to Serbia's citizens and to become sustainable in an increasingly competitive market dominated by media that had consolidated their positions during the 1990’s era, and by the new multinational media conglomerates that entered the Serbian media market after 2000. The previous USAID media program, ProMedia II/Serbia (1999-2008) delivered support that included technical assistance toward raising journalism and production standards; training in specialized reporting, such as business and economics; subgrant support for equipment purchases, program productions, and in-depth coverage; advancement of the privatization of municipal media; and the development of skills in media management. SMAP focuses on building a stronger, more transparent, effective and stable legal and regulatory environment for media by developing advocacy and monitoring skills of media support institutions, strengthening capacities to represent their constituencies and advance public interest, and enhancing capacities for coalition building and joint initiatives among media stakeholders. SMAP promotes the sustainability and professional skills of select local and regional broadcast media outlets through customized development plans for each station aimed at improving management and financial capacities, use

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM A-1 of research and ratings surveys, provision of training and production subgrants. Program partners are primarily local outlets covering key markets throughout the country, but some assistance is also targeted at other local media companies. The Program encourages quality media coverage by local and regional media of issues of interest and concern to citizens in their communities. The Serbia Media Assis- tance Program also supports local media outlets in adapting to the new digital environment and using new media, such as mobile phones, the Internet, and social networking sites, to enhance their operations and sustainability, as well as the public’s access to information. The aim of SMAP is to provide support to the transition of the Serbian media sector into a legally sound, economically viable system providing professional and independent news and information for citizens throughout Serbia. The Serbia Media Assistance Program seeks to build on previous USAID media support by: 1. Assisting media support institutions, many of whom have developed as solid organizations with previous USAID assistance, to act as more effective advocacy organizations 2. Using existing Serbian talent and resources to provide technical assistance and training, to the maximum extent possible 3. Replicating the journalism standards established at the national level with USAID assistance at the local and regional level 4. Refocusing support from previous long-term partners who have reached standards of quality journalism and are nearing sustainability. According to its design, the program is to produce the following results: 1. Strengthen the legal and regulatory environment for media, particularly the role of media support institutions and other organizations in ensuring fair, timely, and transparent implementation 2. Increase the managerial and financial capacity of independent media outlets, particularly local and regional television stations following privatization and licensing 3. Enhance the quality and availability of information important to citizens, with a focus on news and program production at the local and regional levels and coverage of reform issues at the national level. Considerable emphasis should have been placed on building sustainable media outlets, organizations, processes, and approaches that will continue to support the health and vitali- ty of the Serbian media sector following the end of USAID assistance. The legacies expected as a result of the Media Assistance Program include: 1. Fair, timely, and transparent implementation of a consistent legal and regulatory framework for media 2. Sustainable capacity of media support institutions and other organizations to effectively advocate for both the media sector and the public interest 3. A core of private local and regional media outlets throughout the country that consistently provide quality coverage of local, regional, minority, and reform issues 4. Strong local media assistance providers supported by the domestic media market.

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM A-2 C.3 RELATIONSHIP TO MISSION STRATEGY From 2001-2011, USAID has provided nearly $663 million in Serbia to address areas that are important to Serbia’s long-term development goals and U.S. interests. The overall strategic goal of this assistance is to foster regional stability and support Serbia’s vision to be democratic, prosperous and engaged in Euro-Atlantic institutions. USAID assistance programs continue to support and advance key reform efforts in these areas. When it was designed, SMAP was originally designated as one of the core programs under the USAID/Serbia Strategic Objective 2.11 “Risk of Political Instability Reduced” within the mission’s strategy for 2006 to 2010. The SO focused on the support of key democratic structures and processes at all levels nation-wide to increase political stability; it also looked to improve conditions in specific, vul- nerable areas, by increasing economic opportunities, civic participation and community interventions. In 2010, that strategy was amended to cover the period of 2011 to 2015, although the mission’s overall strategic objective of supporting Serbia in its vision to be “democratic, prosperous and moving toward Euro-Atlantic Integration” remained the same. Under the amended Country Strategy for Serbia USAID democracy programs are re-focused to reflect the current situation, and assistance is united under one Assistance Objective, Democratic Institutions Strengthened. USAID’s strategy includes two Assistance Objectives: 1. A More Competitive Market Economy and, 2. Democratic Institutions Strengthened. In developing this strategy, USAID worked closely with local stakeholders including counterparts in the Serbian Government, private sector, and civil society, and took into consideration the assistance portfo- lios of other donors and partners. Other broad development factors that are critical to Serbia’s advancement are integrated into the USAID portfolio, as appropriate. These include transparency and accountability, gender, youth, reduced regional imbalances, national minorities’ inclusion, use of public-private partnerships, and regional and cross- border cooperation. Assistance Objective (2) pursues two key elements: improving government operations; and increasing civil society engagement in public. Initiatives addressing civil society engagement in public life area aimed to strengthen media and civil society organizations so they can better represent their constituents, serve as effective watchdogs and advance the public interest. Inter alia, activities include raising the professional and managerial standards of organizations, providing the capacity to conduct successful advocacy campaigns, promoting public-private partnerships and corporate social responsibility, and opening channels for public input to the governing process. Candidates and political parties could be advised on internal governance, constituent outreach, coalition building, and management and leadership. Serbia now has many essential democratic structures in place and a government and public generally committed to EU accession. Through Assistance Objective (2), USAID’s strategic approach shifts toward improving the effectiveness, transparency and accountability of government institutions at the central and local levels; increasing judicial independence; supporting civil society to increase its capacity to engage in public life; and encouraging civil society and media to advance the public interest. USAID’s approach includes assistance to enhance the effectiveness of political parties and their responsiveness to voter concerns, developing a legal and regulatory environment that supports Civil Society Organizations (CSOs), further support for CSO advocacy and fostering the concept of corporate social responsibility.

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM A-3 USAID media assistance falls under the USAID Assistance Objective (2) - Democratic Structures in Serbia Strengthened; and addresses two intermediate results: Civil Society Engagement in Public Life Increased, and Civil Society and Independent Media Strengthened. C.4 OBJECTIVE The objective of this performance evaluation is to conduct a full, evidence-based and independent review of USAID/Serbia SMAP activities and results from September 2008 to the present. Specifically, the purpose of the evaluation of SMAP is to identify and detail the extent to which the program achieved its intended results. This includes the impact the Program has had on the Serbian media sector, but also on Program primary and secondary partners and other stakeholders; and how / if the Program assisted Serbian citizens to have access to professional and independent news and information. The evaluation shall also examine the extent to which the achievement of intended results was affected by implementation approach and effectiveness. The conclusions and recommendations from the evaluation would be used by the Mission management for assessing the results of the investment in media assistance to Serbia. Lessons learned will be shared with other USAID Missions across the world. As USAID Serbia is in the process of designing the new civil society program which may provide limited targeted support to media CSOs, the recommendations, conclusions and experiences from the SMAP evaluation will be valuable for the upcoming development of activities in this field. C.5 DETAILED STATEMENT OF WORK The evaluation will show how the Recipient implemented a program to provide support to the transition of the Serbian media sector into a legally sound, economically viable system providing professional and independent news, and information for citizens throughout Serbia. Specifically, the evaluation shall identify and discuss: (1) How the Program influenced its primary and secondary partners and other stakeholders through its three main primary activities, namely has it successfully strengthened the legal and regulatory framework for media, increased the sustainability of its media partners and enhanced the quality and availability of information for citizens; (2) How the three primary activities contributed to the overall Program goal, namely the transition of the Serbian media sector into a legally sound, economically viable system providing professional and independent news, and information for citizens throughout Serbia; (3) Lessons learned regarding program management and implementation; (4) The extent to which SMAP achieved its intended Program results and legacies, including where appropriate, analysis of resource sufficiency relative to results achieved. Apart from an evaluation of the achievement of the original program goals set by the RFA, the evaluation shall answer the following: (5) Were there factors that were not envisaged in the original design that Program had to respond to in order to achieve its goals; how the program responded to those factors (if any), and what impact (if any) did they have on Program’s work and results, including but not limited to Program’s overall goal and expected results; (6) Estimate if / how new media have been employed by the Program, and the impact of new media activities on Program partners and program goals (if any).

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM A-4 Tasks The specific tasks to be undertaken by the evaluation team to prepare the deliverables include review of the following documents: - List of partners - Current cooperative Agreement - SMAP Program Description (from Cooperative Agreement) - Implementing Partner’s Annual Report for first, second and third year of the Program - Implementing Partner’s performance quarterly reports - Work plan for years 1 - 4 - Implementing Partner’s PMP - Media Sustainability Index reports 2009, 2010 and 2011 - Covington & Burling legal analysis of the amended Public Information Act and the translation of the amended articles (along with the English version of the law with integrated changes and a Brief that was submitted to the Constitutional Court) - Anti-corruption Council report on Media Situation in Serbia 2011 - USAID strategy (FY 2006-2010) - USAID amended strategy (FY 2011-2015) - Draft Strategy For The Development Of The Public Information System In The Republic Of Serbia Until 2016 – unofficial draft translation (2011) - Media Study Report (funded by EU) (2010) Evaluation team should also review the following information sources: 1. Media Sustainability Index reports – available on: http://www.irex.org/msi/ 2. European Commission Key findings on Serbia 2011 - http://www.europa.rs/en/srbijaIEu/kljucni_dokumenti/2010.ht ml 3. Local media laws and other regulations in English: http://www.anem.rs/en/propisi/laws.html; http://www.anem.rs/en/propisi/otherregulations.html 4. Anti-corruption Council report on Media Situation in Serbia 2011 5. ANEM (Association of Independent Electronic Media) reports on Legal monitoring of the Serbian media scene (USAID funded project): http://www.anem.rs/en/aktivnostiAnema/monitoring.html To support the team’s initial literature review, USAID/DGO will provide electronic copies to the contractor of all documents to be reviewed (listed above).

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM A-5 Once in the field, evaluation team should meet with: - USAID/Serbia’s Mission Director, Democracy and Governance Office Director, program AOR, and other DGO staff

- U.S. Embassy-Belgrade staff, - Program partners, including media outlets (primary and secondary partners) and associations, both in Belgrade and throughout Serbia.

A meeting with key Government partners, including the Ministry of Culture, Media and Information Society (formerly Ministry of Telecommunications and Ministry of Culture, combined into one ministry after the Government reconstruction in 2011), Republic Broadcasting Agency, Republic Agency for Telecommunications, and others, may also be scheduled. Methodology: The evaluation team shall start its work with a paper review of all the documents cited in the “Tasks” section above. Key informant interviews will be the primary data source for this evaluation. This is the most appropriate method to obtain relevant data. All findings and conclusions/qualitative analysis should be supported by quantitative data, when available, and/or representative examples. Program addresses different sectors, geographical regions and partners that may have different roles, capacities, impact and interests. Besides evaluating overall Program impact and achievements, analysis/findings should also show disaggregated Program results/impact by the different types of partners, regions and/or sub-sectors included in the Program, supported by quantitative data and/or representative examples. The Team Leader will provide the list of core questions to the USAID prior to starting interviews. USAID reserves the right to approve /suggest additional questions. The evaluation will summarize the findings from field visits, discussions, and meetings with respective reports. Pitfalls and gaps, if any, should be identified and supported by recommendations. Reports shall be submitted to the Task Order COR and Democracy and Governance Office (DGO) Director for review, and an out-brief and summary of findings provided to Mission Management. The evaluation should be backed up by relevant data and information gathered from meetings with program partners and all the relevant stakeholders. The evaluation team should clearly state their methodology in producing evaluation and clearly demonstrate links between the recommendations/conclusions and data available. In addition to these approaches, the Mission is open to new, creative suggestions regarding this evaluation. The implementer is invited to provide a more detailed explanation of the proposed methodology for carrying out the work. The body of the final evaluation (description of the project, evaluation purpose and methodology, findings, conclusions and recommendations) should not exceed 25 pages.

[END OF SECTION C]

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM A-6 SECTION F – DELIVERIES O PERFORMANCE

F.1 PERIOD OF PERFORMANCE

a) The estimated period of performance for this task order is from September 1, 2012 to November 15, 2012.

(b) Subject to the ceiling price of this task order, the TOCOR may extend the estimated completion date, provided that the extension does not cause the elapsed time for completion of the work, including the furnishing of all deliverables, to extend beyond 60 calendar days from the original estimated completion date. Prior to the original estimated completion date, the contractor shall provide a copy of the TOCOR’s written approval for any extension of the term of this task order to the Task Order Contract- ing Officer (TOCO); in addition, the contractor shall attach a copy of the TOCOR's approval to the final voucher submitted for payment.

(c) It is the contractor's responsibility to ensure that the TOCOR-approved adjustments to the original estimated completion date do not result in costs incurred that exceed the ceiling price of this task order. Under no circumstances shall such adjustments authorize the contractor to be paid any sum in ex- cess of the task order amount.

(d) Adjustments that will cause the elapsed time for completion of the work to exceed the original estimated completion date by more than 60 calendar days must be approved in advance by the TOCO.

F.2. DELIVERABLES

In addition to the requirements set forth for submission of reports in Sections H, I and J, and in accordance with AIDAR clause 752.242-70, Periodic Progress Reports, the Contractor shall submit reports, deliverables or outputs as further described below to the COR (referenced in Sections F.2 and G).

The evaluation team will produce the following deliverables:

1) Work Plan: A detailed workplan will be submitted within 5 calendar days after the effective date of the Task Order, to be approved by USAID/Serbia, in which the timeline, methodology, and activities of the evaluation are clearly stated.

2) Draft Evaluation Report: The Contractor will submit the Draft Evaluation Report of findings and recommendations to the Task Order COR and USAID/Serbia within 26 calendar days after the effective date of the Task Order prior to the team’s De-briefing with the Mission (see below), but before the departure of the evaluation team from Serbia. The Draft Report shall clearly describe findings, conclusions and recommendations and shall not exceed fifteen pages. USAID/Serbia will share the draft report with its implementing partners and request their comments.

The Contractor will receive feedback from USAID/Serbia and its implementing partners on the draft report within 30 working days from the submission of the draft evaluation report. Con- tractor shall incorporate these findings and recommendations into the Final Evaluation Report as appropriate.

3) Debriefing with USAID: The team will present the major findings of the evaluation to

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM A-7 USAID/Serbia after submission of the Draft Evaluation Report, but before the team’s departure from the country. The debriefing will include a discussion of achievements and issues as well as any recommendations the team has for possible modifications to project approaches, results, or activities. The team will consider USAID comments and revise the Draft Report accordingly, as appropriate.

4) Final Evaluation Report: The Contractor will have eight work-days after receipt of USAID’s comments on the Draft Report to submit the Final Evaluation Report. The Final Report is to be delivered to USAID/Serbia/Democracy and Governance Office (DGO). The Final Report shall include an executive summary (up to two pages), a table of contents (one page), list of acronyms and abbreviations (one page), the body of the report (not to exceed 25 pages) and applicable appendices. It is anticipated that the Final Report would be longer than the Draft in order to include fuller analysis and recommendations from the Team. The body of the report should contain description of the project, evaluation purpose and methodology, findings, conclusions and recommendations. Attached to the report in annexes will be the evaluation statement of work, a full presentation of the evaluation design and methodology, list of interviewees, and list of documents reviewed. Additional documents used in the preparation of the evaluation can also be included. The report will be submitted in English, electronically. The report will be disseminated by USAID.

Agency criteria to ensure quality in the evaluation final report are presented in Attachment J.1. The contractor shall use all of these criteria in the preparation of the final report.

The evaluation must be in compliance with USAID Bureau for Policy, Planning and Learn- ing Evaluation Policy dated in January, 2011 as link is provided herein: http://www.usaid.gov/evaluation/USAID_Evaluation_Policy.pdf

5) Weekly Meetings: It is expected that the evaluation team will be in regular contact with the Task Order COR throughout the period of performance. In addition, the team should schedule weekly meetings (as needed) with the Task Order COR and Democracy and Governance Of- fice staff (at a time convenient to mission staff) to update them on the progress of the evaluation and to ask any clarifying questions on the project as the evaluation progresses.

The evaluation team is required to submit all records from the evaluation and all quantitative data in an organized fashion and fully documented for use by those not familiar with the project or evaluation.

In accordance with Section F.7 herein, the evaluation team is required to submit all records from the evaluation and all quantitative data in an organized fashion and fully documented for use by those not familiar with the project or evaluation. The Final Evaluation Report shall be submitted to the Development Experience Clearinghouse (DEC) within three months of completion by the USAID Serbia Program Office at http://dec.usaid.gov. The report must be approved by USAID/Serbia before submission to the DEC.

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM A-8 ANNEX B: EVALUATION SCHEDULE

Performance Evaluation - Serbia Media Assistance Program September 10–28, 2012

Thursday, 6 September 14:00 Conference Call with USAID E&E (DGST) Monday, 10 September 10:30 Team Meeting 14:00 Meeting with IREX 16:00 USAID In-Brief Tuesday, 11 September 10:30 MPAS 12:00 Pistaljka 14:00 Media Association 16:00 B92 Wednesday, 12 September 10:00 USAID DG Office 11:00 Media Coalition Round Table 13:00 UNS 15:00 IAB 16:30 EU Delegation Thursday, 13 September 9:00 RATEL 10:00 Travel to Novi Sad 11:30 RBA 13:30 Radio 021 (Novi Sad) 15:00 Media Art (Novi Sad) 16:00 RTV Vojvodina (Novi Sad) 17:30 Travel to Belgrade 20:30 FoNet News Agency Friday, 14 September 9:00 U.S. Embassy, Public Diplomacy Section 10:00 NUNS 11:30 ABC 12:30 Travel to Aranđjelovac 13:00 Multiple meetings at the Aranđjelovac Summer School (Aranđjelovac) Monday, 17 September 12:00 TV5 (Niš) 14:00 Radio City (Niš) 15:00 OK Radio Vranje (in Niš) 16:00 Juzne Vesti (Niš) Tuesday, 18 September 9:00 TV Zaječar (in Niš) (cancelled by TV Zaječar)

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM B-1 10:30 Media Centar Niš (Niš) 13:00 Travel to Kragujevac 16:00 Sumadijapres (Kragujevac) 17:00 Travel to Požarevac 18:00 Radio Boom 93 (Požarevac) 19:30 Travel to Belgrade Wednesday, 19 September 10:00 Open Society 12:00 Ministry of Telecommunications 14:00 ANEM 16:00 Ipsos Thursday, 20 September 10:00 Former Minister of Telecommunications 11:00 OSCE Regional Media Conference “Main challenges to media freedom in South East Europe Shaping Policy for the Future” 12:00 NDNV (in Belgrade) 13:00 Radio Plus Pirot (in Belgrade) 14:00 Institute for Sustainable Communities 15:00 Local Press (in Belgrade) Friday, 21 September 10:00 U.S. Embassy, Commercial Section 12:00 Media Center Belgrade 13:30 European Consulting Group 15:00 USAID SMAP AOR Sunday, 23 September 11:00 Team Debrief 18:00 Preliminary De-Brief for Head of DG Office Monday, 24 September Report Writing 19:00 OSCE Tuesday, 25 September Report Writing 11:00 Ministry of Culture 16:00 Debrief for USAID Mission Director Wednesday, 26 September Report Writing Thursday, 27 September Report Writing 16:00 Outbrief with Mission Friday, 5 October 09:00 IREX-Washington, DC

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM B-2 ANNEX C: DOCUMENT REVIEW

STRATEGY AND PROGRAM DOCUMENTS  USAID/Serbia’s FY 2006 – 2010 strategy  USAID/Serbia’s amended strategy for FY 2011 – 2015  Cooperative agreement No. 169-A-00-08-00101-00  SMAP program description  SMAP annual program reports (October 2008-September 2009, October 2009 – September 2010, October 2010 – September 2011)  SMAP quarterly reports (October 2008 – present)  SMAP performance monitoring plan; and  SMAP annual work plans (October 2008 – September 2009, October 2009 – September 2010, Oc- tober 2010 – September 2011)

OTHER RESOURCES  Draft Strategy for the Development of the Public Information System in the Republic of Serbia until 2016 (unofficial draft translation)  Covington and Burlington legal analysis of amended Public Information Act  English translations of the Public Information Act, local media laws, and pertinent regulations  Anti-Corruption Council Report on the Media Situation in Serbia (2011)  Media Study Report (2010)  Media Sustainability Index reports (2008 – 2011)  European Commission Key Findings On Serbia (2011)  Freedom House Freedom in the World reports for Serbia (2008 – 2012)  Final Report of the OSCE-ODIHR Election Observation Mission to the Republic of Serbia Early Parliamentary Elections 2008 (section VII. on media)  Final Report of the OSCE-ODIHR Election Observation Mission to the Republic of Serbia Parlia- mentary and Presidential Elections 2012  IREX Serbia RFP – Economic Study on the Impact of Conversion of Serbia to Digital TV  RAB Serbia Founding Assembly  IREX list of SMAP digitalization study tour participants  IREX list of SMAP trainings  IREX list of SMAP grants  San Ng consultant report on ANEM

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM C-1  Media & New Technologies Among Young Generation – Strategic Marketing Research (commissioned by IREX)  2011 MDLF Media Development Impact Dashboard  RATEL Publication on Mobile Phone and Internet Penetration  Serbia Rapid Media Assessment – Meg Gaydosik and Troy Etulain, 2010  European Integration - Media Fund Summaries of awarded projects 2010, European Union Delega- tion to Serbia  Republic of Serbia Civil Society Assessment Report, Prepared By Catherine Barnes For USAID Serbia And Montenegro  Serbian Media Support Network Report, Maureen Taylor, Social Impact, April, 2012  IREX, Serbia Media Assistance Program, Proposed Extension Period Activities  ANEM Monitoring Newsletters, a selection between August 13, 2009-July 30, 2012  “Confidence in the media as source of information,” Strategic Marketing Research (commissioned by IREX)  “New Media Usage Amongst Youth,” Ipsos (commissioned by IREX), December 2011  “Report On The Pressures And Control Of The Media In Serbia,” Anti-Corruption Council of the Serbian Government, August 29, 2011  “Audience Of Major News Shows RTV 1 from 01.01.2008 to 15.09.2012,” RTV Vojvodina  “New Media Usage,” Ipsos (commissioned by IREX), October 2011  “Serbia - Estimates Of Total Market Value Of Real Mass-Media Advertisement For 2011,” Nielsen

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM C-2 ANNEX D: PERSONS INTERVIEVED

Organization Name Title Location USAID Fritz Susan Mission Director Belgrade USAID Peter Wiebler Director, DG Office Belgrade USAID Walter Doetsch (former) Director, DG Office Via phone USAID Ellen Kelly Senior Rule of Law Advisor Via skype USAID Sibina Golubovic Media Manager / AOR Belgrade USAID Marko Pjevic Activity Manager Belgrade

USAID Miodrag Bogdanovic Monitoring and Evaluation Belgrade Specialist USAID Ivan Vukojevic Program Development Special- Belgrade ist USAID Aleksandar Djurei- Project Management Specialist Belgrade novic USAID Dragana Stevanovic Project Management Specialist Belgrade

USAID Milena Zivkovic Project Management Assistant Belgrade USAID Meg Gaydosik Senior Media Development Washington, DC Advisor, E&E/DGST IREX Serbia Marina Paunovic Chief of Party Belgrade IREX Serbia Vladan Nikolic Deputy CoP Belgrade IREX Serbia Tatjana Antic Administrative staff Belgrade IREX Serbia Rich McClear Former CoP Via e-mail IREX Serbia Wes Burd Senior Media Advisor Belgrade IREX WDC Mark Whitehouse Vice President, Media Programs Washington, DC IREX WDC Lisa Kovack Program Officer Washington, DC IREX WDC Erin Murrock Program Coordinator Washington, DC U.S. Embassy Stark Stephen Press Attache Belgrade U.S. Embassy Zorica Mihajlovic Senior Commercial Specialist Belgrade ABC Srbija (Audit Neda Lukic Executive Manager Belgrade Bureau of Circula- tion) AIM (Alliance Inter- Ruza Ristanovic Executive Director Belgrade national Media) ANEM Sasa Mirkovic Chairman Belgrade ANEM Jasna Milanovic Coordinator Belgrade ANEM Slobodan Kremenjak Lawyer Belgrade Association of Media Zoran Papic General Manager Belgrade BETA Jana Zaric Project Manager Belgrade

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM D-1 European Consulting Sanja Bakoc Account Manager Belgrade Group EU Delegation Svetlana Djukic Project Manager-Operations Belgrade FoNet Agency Zoran Sekulic Director Belgrade IAB Gabrijela Stjepanovic Executive Director Belgrade Institute for Sustaina- Bogdan Gavanski Deputy Chief of Party for Pro- Belgrade ble Communities grams

Ipsos Strategic Mar- Srdjan Bogosavljevic Managing Director Belgrade keting Juzne Vesti Predrag Blagojevic Editor in Chief Niš Local Press Snezana Milosevic General Secretary Kragujevac Dejan Miladinovic Media Art Rastislav Durman Director Novi Sad Media Center Beo- Sami Ofelija Backo- Director Belgrade grad vic Media Center Beo- Nevenka Vucenovic Head of Financial Department Belgrade grad Media Center Beo- Gordana Mladenovic Deputy Director Belgrade grad Media Centar Niš Danijela Maravic Media Relations Manager Niš Media Photographers' Zvezdan Mancic President Belgrade Association of Serbia Ministry of Culture Dragan Kolarevic Assistant Minister Belgrade Ministry of Culture Snezana Pecencic Senior Advisor Belgrade Ministry of Foreign Stefan Lazarevic State Secretary for electronic Belgrade and Internal Trade communications, information and Telecommunica- society and postal services tions Ministry of Foreign Irini Reljin Assistant Minister Belgrade and Internal Trade and Telecommunica- tions Ministry of Foreign Katarina Tomic Advisor Belgrade and Internal Trade and Telecommunica- tions Ministry of Foreign Vladana Radis- Advisor Belgrade and Internal Trade avljevic and Telecommunica- tions Ministry of Foreign Jovana Matic Head of Unit for Programming Belgrade and Internal Trade and Telecommunica- tions

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM D-2 Ministry of Tele- Jasna Matic Former Minister of Telecom- communications munications NDNV (Independent Dinko Gruhonjic President Novi Sad Journalist Association of Vojvodina) NUNS (Independent Vukasin Obradovic President Belgrade Journalists’ Associa- tion of Serbia) OK Radio Goran Vladkovic Director Vranje Open Society Fund Dragan Kremer Media Program Coordinator Belgrade Organized Crime and Drew Sullivan Advising Editor Corruption Reporting Project OSCE Dragana Nikolic Sol- Head of the Media Department Belgrade omon Pistaljka (Eutopia) Vladimir Radomiro- Director Belgrade vic Pistaljka (Eutopia) Dragana Matovic Editor Belgrade RAB Maja Rakovic Chairman Man Board Belgrade Radio 021 Slobodan Stojssic Director Novi Sad Radio 021 Slobodan Krajanovic Editor Novi Sad Radio 021 Ivica Milaric Creative Editor (website) Novi Sad Radio Boom 93 Milorad Tadic Director Požarevac Radio City Niš Dragan Kocic Director Niš Radio Luna Aleksandar Rankovic Director Užice Radio Plus Nenad Paunovic Director Pirot RATEL (Republic Milan Jankovic Director Belgrade Telecommunication Agency) RRA (Republic Goran Karadzic Vice-President of Coun- Belgrade Broadcasting Agen- cil/Zamenik Predsednika RRA cy) RTV B92 Veran Matic President, Board of Directors Belgrade RTV Cac ak Svetlana Kojanovic Director Čačak RTV Vojvodina Sinisa Isakov General Manager Novi Sad Sumadijapres Ljiljana Palibrk Director Kragujevac TV 5 Sladjana Ostojic Editor in Chief Niš UNS (Journalists’ Ljiljana Smajlovic President Belgrade Association of Ser- bia) UNS Nino Brajovic General Secretary Belgrade

PG Mreza Lila Radonjic Aranđjelovac

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM D-3 Deputy Prime Minis- Suzana Vasiljevic Aranđjelovac ter's PR Radio-Television Antonela Riha Aranđjelovac Serbia City Radio Niš Branislava Cvetkovic Aranđjelovac

RTV Vojvodina Tanja Jordovic Aranđjelovac

Boom 93 Jelena Petkovic Aranđjelovac

Ipsos Strategic Mar- Jasna Milosevic Aranđjelovac keting Radio 9 Kragujevac Bojana Ljiljak Aranđjelovac

UNS Olivera Stosic Aranđjelovac

Faculty for Commu- Nada Perisic Popovic Aranđjelovac nications and Media

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM D-4 ANNEX E: SMAP PROJECT EVALUA- TION WORK PLAN

PURPOSE Democracy International, Inc. (DI) will conduct a comprehensive performance evaluation of USAID’s Serbia Media Assistance Program (SMAP) implemented by the International Research and Exchanges Board (IREX). As outlined in the Statement of Work, the purpose of the evaluation is to measure the changes in development outcomes that are attributable to IREX’s Serbia Media Assistance Program. The DI team will assess how the program’s performance has influenced the Serbian media sector as well as IREX’s primary and secondary partners and other stakeholders. The evaluation will provide a detailed examination of the progress SMAP has achieved toward the following three results:  Strengthen the legal and regulatory environment for media;  Increase the managerial and financial capacity of independent media outlets; and  Enhance the quality and availability of information important to citizens with a focus on: o News and program production at the local level; and o Coverage of reform issues at the national level. The evaluation will also analyze where the program was successful and where it was not, identify the factors affecting program implementation, identify obstacles to program success, and examine the effect of the program on USAID/Serbia’s strategic objectives. The team will also formulate lessons learned regarding program management and implementation, including identifying unanticipated factors that were not envisaged in the original design, whether resources were sufficient for program objectives and how the program responded and adapted to these factors, and the effect of these factors on the program’s goals, activities, and results. The team will also evaluate whether the use and impact of new media by the program and its partners positively or negatively affected program outcomes. METHODOLOGY OVERVIEW Based on the time available to undertake the evaluation and the size of the team, DI will rely on Rapid Appraisal, an approach that utilizes several evaluation methods to quickly, yet systematically, collect data. This will include a review of documents, key informant interviews, and observation during site visits. Given the limited number of individuals who will be able to provide substantive feedback on the program, the evaluation design will feature purposive, nonrandom sampling methodologies, including:  A modified version of quota sampling, where the team will identify categories within the popula- tion to be included in the evaluation design (e.g., USAID and the U.S. Embassy; other donors; IREX; other civil society and media implementers; media outlets; intermediary organizations; such as professional media associations and media support nongovernmental organizations; policy- makers; media regulators; and independent experts)  Judgment sampling, where the team selects the sample based on expert judgment and understanding of the program; and  Snowball sampling, which relies on referrals from initial subjects to generate additional subjects. Further discussion of the evaluation methodology, by activity, is presented in the subsequent sections.

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM E-1 PHASE I: PREPARATION AND INITIAL DESK REVIEW Before arriving in Serbia, the evaluation team will conduct a detailed desk review of relevant documents. This initial review will help the team to better understand the social, political, and political context in which Serbia’s media has developed, stalled, or regressed (and related challenges and opportunities) as well as SMAP’s goals, implementation plans, and performance monitoring efforts. Through the desk review, the team will be able to identify relevant and measurable indicators of program outcomes and to develop and refine its evaluation methodology, work plan, and interview guide based on a more thorough understanding of the program’s activities. As part of the desk review, the team will:  Review all relevant USAID strategy and SMAP program documents; and  Examine academic literature, expert analysis, survey research, media indices, legislation, and policy documents as well as informal publications (e.g., media reports, web publications, unpublished studies, internal memos) on the history and current status of the Serbian media sector and other pertinent topics. The desk review will include, but not be limited to the following resources:

STRATEGY AND PROGRAM DOCUMENTS  USAID/Serbia’s FY 2006 – 2010 strategy;  USAID/Serbia’s amended strategy for FY 2011 – 2015;  Cooperative agreement No. 169-A-00-08-00101-00;  SMAP program description;  SMAP annual program reports (October 2008-September 2009, October 2009 – September 2010, October 2010 – September 2011);  SMAP quarterly reports (October 2008 – present);  SMAP performance monitoring plan; and  SMAP annual work plans (October 2008 – September 2009, October 2009 – September 2010, Oc- tober 2010 – September 2011). NOTE: The evaluation team will also review documents from SMAP’s partner organization, the Euro- pean Consulting Group (ECG) and other relevant USAID-funded projects, such as the Association of Independent Electronic Media’s reports on legal monitoring of the Serbian media scene.

OTHER RESOURCES  Draft Strategy for the Development of the Public Information System in the Republic of Serbia until 2016 (unofficial draft translation);  Covington and Burlington legal analysis of amended Public Information Act;  Public Information Act, local media laws, and pertinent regulations available in English;  Anti-Corruption Council Report on the Media Situation in Serbia (2011);  Media Study Report (2010);  Media Sustainability Index reports (2008 – 2011);  European Commission Key Findings On Serbia (2011);  Freedom House Freedom in the World reports for Serbia (2008 – 2012);  Final Report of the OSCE-ODIHR Election Observation Mission to the Republic of Serbia Early Parliamentary Elections 2008 (section VII. on media); and  Final Report of the OSCE-ODIHR Election Observation Mission to the Republic of Serbia Parlia- mentary and Presidential Elections 2012 (section X. on media). NOTE: In an effort to maximize synergies and avoid duplication of effort, the team will also examine documents from other donors.

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM E-2 During this phase the Team Leader will prepare the evaluation work plan and will work with the Media Expert to prepare an interview guide for the key informant interviews. Both the evaluation work plan and the interview guide will be subject to review by the Mission and finalized before the start of interviews in the field. Prior to departure, the Team Leader will meet with DI staff, USAID, and IREX representatives in Washington, DC. During this phase, the Interpreter/Local Logistics Coordinator and DI staff will begin making logistical arrangements in preparation for the fieldwork, including international flights, lodging in Belgrade, in- country translation and transportation, and scheduling.

PHASE II: FIELDWORK AND DATA COLLECTION Phase II primarily encompasses fieldwork and data collection.

IN-BRIEF AND CONSULTATIONS WITH THE MISSION Upon arrival in Belgrade, the team will participate in an in-brief with the Mission to develop a more nuanced and detailed understanding of USAID’s goals for the evaluation as well as to define and clarify the evaluation approach. During the in-brief, the team will identify and prioritize interviewees who can provide substantive feedback on the program and represent diverse views, determine the cities to be visited by the team, and finalize evaluation activities, list of interviewees, planned deployments, work plan, and interview guide.

KEY INFORMANT INTERVIEWS Following the in-briefing and finalization of evaluation planning, the team will conduct key informant interviews in Belgrade and outside the capital. On-site key informant interviews will provide the team with a better appreciation of program stakeholders’ attitudes and behaviors as well as in-depth information on specific issues based on individuals’ perspectives and experience. The interviews can also be expected to yield insights that may not have been fully captured by formal programmatic reporting, including challenges, lessons learned, and suggestions for improving such assistance in the future. The team will conduct interviews based on the interview guide described above. The interview guide will serve to structure discussions with various key informants and will provide for conversational yet focused communication. The interview guide will be tailored to each of the SMAP program objectives and media topics more generally. The specific mix of questions to be used in a given interview will be based on the background and expertise of interviewees and demographic considerations as appropriate. By using a standardized questionnaire, the team will be able to qualitatively compare the answers and reactions of different stakeholders and beneficiaries in order to facilitate inferences on program performance and impact. At the same time, a flexible approach will allow the team to explore unanticipated topics and adjust data collection to respond to conditions on the ground.

SITE VISITS SMAP institutional partners are located throughout Serbia, including in Novi Sad in the north and Niš, Novi Pazar, Zaječar, Užice, and Sandžak in the south. Given the size of the team, the timeframe for the evaluation, and budget considerations, the evaluation team will not be able to visit all locations where SMAP has institutional partners, grantees, and beneficiaries. Due to the timing of IREX’s summer school from 12 – 15 September in Aranđjelovac and the OSCE’s second Southeast Europe Media Conference on 20 and 21 September in Belgrade, some partners will be travelling to and from these events and staying at event locales. This has necessitated some changes in planned trips to the region. As a result, the team will meet some regional partners while they are in Aranđjelovac and in Belgrade.

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM E-3 At present, the team plans to travel to Novi Sad and to Niš to interview partners from those towns and surrounding areas. Priorities for meetings during the IREX summer school and the OSCE conference include regional partners from Novi Pazar, Užice, and Kragujevac.

As the team is still collecting information about which partners will be where and when, some continued flexibility in planning is likely to be required.

PHASE III: DATA ANALYSIS AND DEVELOPMENT OF RECOMMENDATIONS

DATA ANALYSIS Following two and a half weeks of fieldwork and prior to departure, the evaluation team will review and analyze information and data collected during the desk research, key stakeholder interviews, and observations during site visits in order to hone the key findings, conclusions, and recommendations. The team will seek to determine the extent to which the program:  Strengthened the legal and regulatory framework for media, the effect of advocacy to reform and strengthen the legal and regulatory environment as well as the capacity media support organizations to build coalitions, advocate for the media sector, and advance the public interest;  Increased the sustainability of its media partners, e.g., whether training helped media outlets reach a wider audience, obtain new sources of revenue, and strengthen their financial or managerial capacity; and  Enhanced the quality and availability of information provided to Serbian citizens, e.g., whether targeted primary and secondary partners are self-sustaining organizations that order Serbian citizens balanced and comprehensive news and information and whether training increased the coverage of minority issues. The team’s data analysis will be informed by a contextual appreciation of the political, social, and economic history of Serbia and a robust understanding of SMAP and the Serbian media sector as well as being grounded in SMAP’s theory of change. Using the theory of change will allow the team to determine how effective the program has been in achieving its intended results and why, whether SMAP was based on accurate assumptions about the enabling environment, and whether the program’s design was appropriate to strengthen the media sector in Serbia. Based on responses to the standardized questions in the interview guide, the team will identify and analyze important trends, latent relationships, and commonalities and differences in responses based on age, gender, ethnicity, partner type, and region.

PREPARATION OF DRAFT REPORT AND MISSION DE-BRIEF On 26 September 2012, the Team will submit a partial draft report (not to exceed 15 pages) to USAID/Serbia that includes the evaluation’s key findings, conclusions, and recommendations. The team will strive to ensure that all findings, conclusions, and recommendations are based on accurate, objective, and reliable data. On 27 September 2012, during the debrief, the team will present achievements, issues, and lessons learned. Also on the 27th, the team will meet with the DG Office to discuss the partial draft report. On the basis of these discussions and verbal feedback provided by the Mission, DI will prepare a full draft report to be submitted to USAID / Serbia by 9 October 2012. The full draft report will include:  Executive Summary (up to two pages);  Table of Contents (one page);  List of Acronyms and Abbreviations (one page);  Body of the Report (not to exceed 25 pages) to comprise;

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM E-4 o Project description o Evaluation purpose and methodology o Findings o Conclusions o Recommendations  Applicable Appendices o Evaluation statement of work o Full presentation of evaluation design o List of interviewees o List of documents reviewed o Other relevant evaluation documents

REVIEW AND FINALIZATION OF THE EVALUATION REPORT USAID/Serbia will review the full draft report and provide written comments to DI by 20 November 2012. The team will then prepare and submit the final evaluation report. DI will submit the final evaluation report to USAID/Serbia by 30 November 2012. DI will ensure that the final evaluation report will meet quality assurance criteria as defined by USAID and will comply with USAID Bureau for Policy, Planning, and Learning Evaluation Policy (January 2011). The final evaluation report will be submitted electronically as stipulated by the task order.

ILLUSTRATIVE INTERVIEWEE LIST The team anticipates holding interviews with the Serbian institutions and organizations based on the availability of key stakeholders and the time available to conduct interviews during the first and second week of the evaluation.

DONORS AND IMPLEMENTERS  USAID  U.S. Embassy  OSCE (media department)  EU Commission (Support for Media Capacity in the Area of EU Integration project)  UNDP Serbia  Open Society Institute  Institute for Sustainable Communities  National Endowment for Democracy (in Washington, DC)

GOVERNMENT OF SERBIA  The Ministry of Culture Media, and Information Society  Republic Broadcasting Agency  Republic Agency for Telecommunications

IREX AND ITS PARTNERS This will include IREX’s primary and secondary partners and grantees, including but not limited to:  Audit Bureaux of Circulations (ABC Srbija)  Association of Independent Electronic Media (ANEM)  B92  Beta Press d.o.o.  City Radio

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM E-5  FoNet  IAB Serbia  Juzne Vesti  Lokal Press  Media Art Service International  Media Center Belgrade  Media Center Niš  The Independent Journalists’ Association of Vojvodina (NDNV)  The Independent Journalists’ Association of Serbia (NUNS)  OK Radio  Radio Advertising Bureau Serbia (RAB Serbia);  Radio Boom 93  Radio Luna  Radio Sto Plus  Radio 021  Timocka RTV  The Journalists Association of Serbia (UNS)  Sumadijapress  Radio 9  TV Čačak  RTV Vojvodina  Radio City Niš  TV 5

ILLUSTRATIVE INTERVIEW GUIDE The team will tailor its interview questions to particular types of organizations and the expertise, responsibilities, and characteristics of those being interviewed and prioritized based on the time available for each interview. Questions will be drawn from an interview guide that includes overarching issues and each of SMAP’s program objectives as illustrated below:

OVERARCHING ISSUES

 How would USAID assess its relationship with IREX, the responsiveness of IREX to the Mis- sion, and the programmatic success and management effectiveness of SMAP?  [For partners] How was your experience with your Memorandum of Understanding with IREX?  What’s the current status of the minority councils with relationship to the media?  How was IREX’s relationship with European Consulting Group (ECG)? o How accessible was your mentor at IREX or ECG?  Can IREX provide the team with updated program performance data?  What is the residual capacity of the advocacy organizations? Did IREX change their disposition toward collaboration? How? o What is the residual capacity of media support organizations more broadly, e.g., with respect to members services and capacity building?  What are the key legacies of the SMAP program? o Enabling environment? o Traditional media? o New media? o Access to quality information?

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM E-6  What kinds of support to media might be provided under the new civil society programming? o Does USAID intend to support media organizations via direct funding (either during the pilot phase or upon completion of the new civil society project? Issues?  With respect to the media, perceptions of the EU’s ability to move things forward from here.  With respect to lessons learned and with an eye to applying lessons learned under similar circumstances, how might USAID and / IREX approached programming assistance differently to address the significant challenges of (to get further along with respect to the transition): o Privatization and issues related to . . .  Minority council control of minority language media;  State funding (conflicting directives related to state support)  Political capture o Digital conversion (both law / policy and technical aspects) o Policy and legal framework? (both reforms and implementations, e.g., piracy) o Other?

LEGAL AND REGULATORY ENVIRONMENT  What where the most significant developments in the legal and regulatory environment for media during the course of the project? o Did support under the SMAP program play a decisive role in influencing these developments? o Please specify how the program impacted the legal and regulatory environment (e.g., in terms of contributing to positive change or preventing or mitigating negative change). o In your assessment, was SMAP able to influence the outcome of any new legislation or reconsideration of any existing legislation (including by-laws, policy papers, etc.?)? Could you give examples? o Are there certain legal and regulatory goals under the program that have not been achieved yet? Elaborate. o How would you assess the ability of the SMAP partners to achieve these goals within the current political context?  Did IREX provide legal and/or advocacy support at a point in time when it was most needed?  Did you benefit from the media law monitoring service (through ANEM)? How did you use this information?  How useful was IREX’s monitoring of the legal enabling environment to you? How could it have been improved?  Did IREX help you keep track of, and/or understand legal and/or legislative issues in any other way? How did you use this knowledge?  Did you find any of the written legal materials produced under SMAP to be useful? How did you use this information?  Was IREX effective at keeping the rest of Serbian media informed about legislative and regulatory events, and/or ongoing advocacy efforts? What was the result?  Were they effective at informing the general public about these issues? Did this make a difference in the outcome of any advocacy efforts?  In terms of the needs and capabilities of your organization, was the customized advocacy training IREX provided to your organization timely? Appropriate? Sufficient? Please elaborate.  How effective was SMAP in helping media organizations to develop their own ability to exercise and/or defend their legal rights? Please provide examples.  Was the IREX program at the center of most legal and advocacy activities during the past 5 years or was it more on the sidelines?

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM E-7  Did assistance you received under SMAP help you and/or your organization improve the effectiveness of your advocacy efforts? How did you assess/measure any improvement?  Did assistance you received under SMAP help you and/or your organization cooperate more effectively with other media support organizations?  Was IREX balanced and fair in its relations with the various media advocacy organizations and associations?  Was IREX successful at bringing in the best possible legal expertise, both from within Serbia and from other countries?  If you have faced legal challenges over the past 5 years, please describe the assistance provided by IREX, if any. What was the impact of this assistance? o Please assess the performance of the various media associations over the past 5 years. How do you think SMAP influenced their performance? o How would you rate the quality of services offered to members? o How does membership in media associations benefit you/your organization, if at all? o Describe your involvement in their activities, if any.  In terms of its legal and advocacy activities, what lessons did you draw from your experience under the SMAP program? o Would you have approached activities differently? o Should more of an emphasis been placed on certain activities and less on others? o Where there gaps in support?  How would you assess SMAP’s role in helping facilitate the digitalization process? Please elaborate.  [For ANEM] Please assess your experience and effectiveness advocating before RATEL and RBA.  Has SMAP significantly enhanced media support organizations’ capacities for coalition building and joint initiatives among media stakeholders? How so?  How effective was SMAP in helping media support organizations build relations among each other and establish more permanent channels of communication and networking habits.

MANAGERIAL AND FINANCIAL CAPACITY  In terms of the needs and capabilities of your organization, was customized assistance in support of strategic planning and business planning to your organization timely? Appropriate? Sufficient? Please elaborate.  Please assess the mentoring you received under SMAP that was aimed at improving your organizations’ managerial and financial capacity. o How has your capacity improved as a result of this assistance, if at all? o How do you track/measure improvements in your managerial and financial capacity?  Do you feel more or less financially stable now than in 2008? Can attribute any changes to your financial stability to the SMAP program? o Has your revenue increased during the course of the program? Data? o Has your resource base become more diversified during the course of the program? Data?  Please describe the usefulness of the development plan prepared for your organization under SMAP. How central is the plan to your day-to-day decision-making?  What changes have you made to your business plan as a result of any aspect of the SMAP program?  Could you please describe any topic-specific consultations you were provided by IREX? o How useful were they? o As a result of these consultations, did you change your strategies or behavior? How? What was the outcome?  Has the SMAP program made you feel more independent or effective as a manager? If so, please explain.  Did you increase your sales as a result of IREX assistance? Data?

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM E-8  In what ways could IREX have helped the Serbian media become more profitable or at least financially stable?  How would you assess your organization’s sustainability (particularly TV / radio outlets)  Is there specific assistance that would have benefitted your organization’s financial sustainability, but that you did not receive under SMAP? Please elaborate.  Do you feel the ratings material IREX provided to you was timely and accurate? How did you use this information? Outcome?  Could you please describe any major changes you made to expenditures as a result of assistance from IREX? o Did you spend more on advertising? Salaries? Equipment? Ratings? Staff Development? o If you did make significant spending changes, what were the results?  Given Serbia’s crowded media environment, was SMAP assistance able to help you distinguish your organization and/or its content from your competitors? o Please describe how this may have contributed to your station’s financial status. o How did SMAP address drastic changes in the media environment, such as the economic crisis, decline in TV viewership, and increase of new media / Internet usage? Has SMAP properly responded to these challenges?  Given Serbia’s crowded media environment, was SMAP assistance able to help you distinguish your organization and/or its content from your competitors?  Are media organizations familiar with the Balkan Trust for Democracy? Other sources of funding?  (As applicable), what has been your experience with the Media Development Loan Fund?

QUALITY AND AVAILABILITY OF INFORMATION  Can you describe changes you made to content as a result of technical assistance provided by IREX as part of the SMAP program?  Are you aware of any efforts by IREX to promote the role of women in Serbia’s media? If so, could you please assess the success of these efforts?  What feedback have you received, if any, about changes you have made to your content? What was the source (e.g., peers, audience, members)? Please describe.  Could you assess the impact of SMAP’s efforts to provide extra coverage of the ICTY?  What improvement in your organization’s content are you most proud?  Do you think the SMAP program has affected ethical standards among Serbian journalists? If yes, give examples.  Do you think the assistance IREX provided affected journalistic standards in one form of media over the other, e.g., radio over TV? o Is there a difference in the journalistic standards of media organizations receiving assistance under SMAP and those that did not? Evidence?  What effect has the SMAP program had on the character of minority language program and content availability? What is the impact of this?  Please describe the mentoring relationship IREX had with your organization. o How often did IREX provide mentoring? o What did you learn that was new, if anything? o How did you apply any newfound knowledge or skills? o What evidence is there that the mentoring effort produced results?  Did you ever receive a content development sub grant from IREX? o If so, please describe what content the grant was for and whether you think the activity was successful. What was the result? o How did you assess/measure the success of the activity?  Has SMAP enabled your organization to reduce any self-censorship that may have been occurring?

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM E-9  Has SMAP assistance helped improve the quality of your organization’s content? Which types of programs improved? How have they improved: o Daily news? o News magazines? o Talk shows? o Live productions? o Features? o Creative productions? o Other?  Has SMAP affected the ability of your organization to conduct investigative reporting? If so, please explain.  Do you believe that SMAP contributed to the quality and quantity of investigative journalism stories? Can you give examples (TV shows, radio programs, web content)?  Do you measure the quality of your content differently now as a result of the SMAP program? If so, please explain.  Did the SMAP program impact your use of new media in any way? o How has this affected your perspectives, your content, your business decisions, your financial prospects?  Was there a particular area of journalistic skills development in which the SMAP program was particularly successful? Can you give examples of this success?  As far as you know, have any media outlets greatly improved the quality of their content and/or productions as a direct result of assistance from IREX under the SMAP program?  According to your perspective, do Serbians—across the country or in your town/city have access to enough sources of information to give them a balanced perspective on current events in Serbia?  Were the study tours to the UK, the Netherlands, and Spain and participation in international con- ferences / events helpful to your organization? How? What new knowledge or skills did you ac- quire? And, how did you apply these within your organization and relative to its activities?  How effectively has SMAP supported local partners in adapting to the new digital environment, so as to enhance operations and sustainability and to increase public access to information?  How useful and effective were SMAP activities in monetizing online content?  How useful has development of a web measurement system proved to be? Has it improved your business? How?  How would you assess the quality and innovation demonstrated in SMAP multi-media events (election coverage / election night, data software visualization program, etc.)?  How would you assess the impact of SMAP’s work on protection of whistleblowers (Juzne, Vesti, Pistaljka)?

DELIVERABLES As part of the evaluation, DI will provide the following deliverables to USAID: 1. Work Plan – to include methodology, question guide, activities, and timeline by 5 September 2012. On the basis of USAID/Serbia feedback, DI will submit a revised Work Plan by 11 Sep- tember 2012. 2. Weekly Meetings – with the TO COR and DG Office staff to provide progress reports and seek clarifications as needed. 3. Draft Evaluation Report a. A partial draft report (up to 15 pages), including findings, conclusions, and recommendations to be provided on 26 September 2012. b. A full draft report (up to 25 pages) will be provided to USAID/Serbia on 9 October 2012. .

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM E-10 4. Debriefing with USAID – to be held on 27 September 2012, during which the team will present major findings, including a discussion of achievements, issues, and recommendations. 5. Final Evaluation Report – full report to be submitted to the Mission by 30 November 2012 following receipt of written comments from USAID/Serbia (on 20 September 2012). Tasks and deliverables will be completed according to the implementation plan that can be found in Ap- pendix A.

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM E-11 ANNEX F: ILLUSTRATIVE TIMELINE

Tasks 1 – 8 10 – 16 17 – 23 24 – 30 1 – 7 8 – 14 15 – 21 22 – 28 29 Oct – 5 – 11 12 – 18 19 – 28 26 Nov Sep Sep Sep Sep Oct Oct Oct Oct 4 Nov Nov Nov Nov – 3 Dec Prepare work plan and inter‐  view guide Make logistical arrangements     and schedule meetings Conduct initial desk research 

Conduct DC meetings (USAID,  IREX) Travel to Belgrade 

In‐brief with USAID/Serbia 

Finalize work plan and inter‐  view guide with Mission Conduct interviews in Bel‐   grade and the regions (N, SE, SW) Attend OSCE 2nd Southeast 20‐21 Europe Media Conference Sept.

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM F-1 Tasks 1 – 8 10 – 16 17 – 23 24 – 30 1 – 7 8 – 14 15 – 21 22 – 28 29 Oct – 5 – 11 12 – 18 19 – 28 26 Nov Sep Sep Sep Sep Oct Oct Oct Oct 4 Nov Nov Nov Nov – 3 Dec Prepare partial draft report ‐  findings and recommenda‐ tions De‐brief with USAID/Serbia 

Depart Belgrade 

Prepare full draft report  

Submit full draft report to  USAID/Serbia USAID/Serbia reviews full        draft report Prepare final report  

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM F-2 ANNEX G: DIGITAL CONVERSION AND THE DIGITAL DIVIDEND

Serbia has an international obligation to convert from analog to digital television broadcasting no later than June 17, 2015, because it is a signatory of the International Telecommunications Union GE-06 Agreement, adopted by in 2006 at the ITU Regional Radio Communication Conference. The conversion to digital television broadcasting aims to improve the quality of broadcast signals and to free up spectrum for other consumer services, most notably mobile broadband. This liberated spectrum, known as the “digital dividend” will be 72 Mhz in size in Serbia’s case, and will come from channels currently being broadcast via UHF. In order to facilitate this switchover, the Serbian government is planning to set up a state-run national digital broadcasting infrastructure known as a multiplex. The system will be a monopoly and all TV broadcasters will use the system. There will actually be two multiplexes, one designated for only national channels, the other which will show the local and regional stations in areas where multiplex transmitters are located. Local TV (understood as a TV station owning and controlling its own transmitter) will effectively cease to exist. According to the government’s plan, all existing licenses of all current TV broadcasters will expire before the planned conversion date. Despite concerns that the government might use the licensing process to eliminate unwanted stations, both RATEL and the Ministry of Foreign and Home Trade and Telecommunications confirmed to the team that all broadcasters will be eligible for new digital broadcasting licenses and that the system will have enough space to accommodate all broadcasters. The government will also use the transition endeav- or to close any remaining pirate TV stations. Serbia’s approach of using multiplexes is not the only possible one. In the United States, for example, TV stations applied for and were issued frequencies and purchased their own transmitters. The U.S. approach also differs from the Serbian approach because it allowed for a multiyear grace period for low-power broadcasters. Serbia plans to “turn on” digital television broadcasting one region at a time, but does not plan to allow any continued analog broadcasting. The Serbian government does not plan to provide any assistance to broadcasters for digital conversion, but it does plan to subsidize 300,000 converter boxes for households that do not own and cannot afford to buy a television set with the DVB-T2 standard planned for Serbia’s system. They also plan to undertake a public information campaign to inform citizens about the digitalization process and are already consider- ing several communications plans, although officials the team met with indicated that there is not yet a plan in place with which they are satisfied. The good news for TV stations that do survive until 2015 is that their inclusion in the multiplex will mean a much greater reach for their signals than their transmitters currently provide. The coming multiplex monopoly also frees stations from having to buy their own digital transmitters, which can cost hundreds of thousands of dollars. Nevertheless, stations will have to pay to be on the system, and although prices have not yet been set, it is not clear how many stations will be able to afford what could be a heavy recurring cost. The evaluation team also found that some TV stations remain uninformed about many of the realities of digital conversion and there are many unanswered questions as much depends on the government’s choice of strategy.

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM G-1 The Serbian approach of using a monopoly broadcaster is viewed by many as a threat to press freedom because of the ease in which the government can shut off a station’s broadcasts. When asked about the future costs faced and business models required for regional and local TV stations, multiple government officials raised an idea that apparently has gained traction in Belgrade—independent stations joining forces to form regional networks as single companies. “I personally will try to push for the systematic networking of these stations. I think that only if they create such a network can they become profitable,” said a high-ranking official at the Ministry of Telecommunications. None of the TV stations interviewed seemed to be aware of this idea, however. A senior government official remarked, “Few people have noticed that there are no local TV stations in the strategy and it will be a big surprise. The truth is that we have too many stations for such a small country.” Aside from television stations’ need to adjust to new cost and audience paradigms, the irony is that the process of converting to digital broadcasting threatens its continued existence. The “digital dividend” of a liberated spectrum provided by a national switch from analog to digital broadcasting will enable the in- troduction of mobile broadband services, allowing citizens to stream or download videos on their mobile devices. Officials told the evaluation team that the GoS is eager to proceed with a 4G spectrum auction and is waiting for digital conversion to occur in order to do so. The switch will also facilitate the spread of broadband internet access. Furthermore, RATEL said more than 50 percent of Serbians already have access to cable TV (according to the 2012 MSI, Serbians currently have 39 cable channels to choose from) and may be content to remain with it instead of buying any new equipment. The issue remains complex. A new public company has split off from Radio-Television-Serbia (RTS) to oversee the transformation to and management of Serbia’s digital broadcasting system. The company is reported to have very little state funding and is behind on its work. The transformation plan the company must produce must include plans for where towers will be placed (the current RTS infrastructure is insufficient), how much of which equipment is needed, etc. At the same time this company chooses where transmitters will be placed to reach various regions of the country, the GoS is choosing where the main studios for its six regional public service broadcasters will be placed. The studio location decision implic- itly also divides the country into spheres of influence, as regional minority councils are expected to domi- nate the content of regional broadcasters. This is because the control in situ of the studios will mean in- person influence over editors and journalists and their journalistic decisions. The potential for overlap or incongruities between regional public service broadcaster locations and target audiences with the place- ment of multiplex transmitters makes the government’s calculations more complicated. Another issue is that the infrastructure the government is setting up could be used for provision of other services. Communication among the various nodes of the government’s planned system will use internet protocol, which can carry any data format. It is possible, for example, that the multiplex could help spread broadband internet or cable television around the country. The digitalization plan already appears to be out-of-date, assuming that each of the seven national stations will soon transition to broadcasting in high definition (HD). This is because the data connection between towers will be 36 megabits, while the Ministry of Telecommunications said that one HD channel requires six megabits. Thus, the multiplex planned for national channels will be six megabits short. It appears that either the government will move the second national public TV channel to the second multiplex or end its operations, allowing for regional public service broadcasters to take its place, or decide to give less band- width to HD channels. Nevertheless, multiple government sources affirmed that the new system will have space for all existing broadcasters.

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM G-2 From their perspective, stations will have to replace any remaining analog TV equipment (such as standard VHS or Super-VHS). For most of the stations, however, this shouldn’t present too big of a cost. Many stations, some of them with support from IREX, already shoot and edit using digital equipment. It is not clear that the disappearance of local TV will mean that IREX’s investment in these stations will have been wasted. Some stations may succeed in making the transition. Individuals trained by IREX may continue to utilize their skills elsewhere. Nevertheless, several of those interviewed during the evaluation expressed fear that the transition will unavoidably lead to a decrease in media plurality, a reduction in coverage of local issues, and the loss of a whole genre of independent media.

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM G-3 ANNEX H: CONSISTENCY OF STORY AND PROGRAM QUALITY STAND- ARDS The findings of IREX’s content analysis indicate improvements in substance and production quality during the course of SMAP, particularly in Y3 and Y4. The team undertook a review of the content analysis scores to determine whether or not partner radio stations were able to achieve and consistently maintain the following: (1) improvements in scores for story categories and program components from quarter to quarter; (2) story and program scores of at least three “adequate” on the five-point scale used for the content analysis; and (3) story and program targets as established in the PMP. In general, we found considerable fluctuation of scores from quarter to quarter, suggesting that some stations had difficulty maintaining higher standards once they achieved them. With respect to story, none of the six radio stations monitored achieved their story target during the course of assistance. Only Radio Luna was able to achieve a score of at least three “adequate,” which means it was good enough for broadcast on any national radio station. However, several stations were able to achieve and maintain improvements in story quality from quarter to quarter as follows:  City Radio showed six quarters of sustained story improvements (out of 12 monitored). As of the third quarter of year four, City Radio’s story score was moving downward.  Radio Luna demonstrated five quarters of sustained story improvements (out of 12 monitored). As of the third quarter of year four, Radio Luna’s story score was moving downward.  021 was able to sustain story improvements for four quarters (out of 12 monitored). As of the third quarter of year four, 021’s story score had notched downward.  Radio Zaječar sustained story improvements for three quarters (out of 10 monitored). As of the last quarter monitored (the first quarter of year four), Radio Zaječar’s story score was moving upward.  Boom 93 was not able to sustain story improvements for at least three quarters (out of nine monitored). As of the third quarter of year four, Boom Radio’s story score was moving upward.  OK Radio did not achieve story improvements (during two quarters monitored). As of the third quarter of year four, Ok Radio’s story score had moved downward. A comparison of program scores by quarter revealed comparatively better performance. At some point during SMAP, five of six radio stations were at least able to achieve a rating of 3 “adequate” for programming. In addition, four of the stations were able to meet their program target at some point during SMAP. Radio Zaječar and Radio Luna showed the most sustained program improvements when compared to the other stations:  Radio Zaječar showed sustained program improvements for eight quarters (out of 10 monitored) and was able to achieve and maintain its program target for six quarters. As of the last quarter monitored (the first quarter of year four), Radio Zaječar’s program score was moving downward. Radio Luna also demonstrated sustained program improvements for eight quarters and was able to achieve its program target for three quarters (out of 12 monitored). As of the third quarter of Y4, Radio Luna’s program score was moving downward.

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM H-1

 Boom 93 showed sustained program improvements for four quarters (out of nine monitored) and met its program target in the third quarter of Y4. At that time, its program score was moving upward.  Radio 021 showed sustained program improvements for four quarters (out of 12 monitored), but did not meet its program target. As of the third quarter of Y4, 021’s program score was moving downward.  OK Radio achieved its program target in the third quarter of Y4 (within the second quarter of monitoring of this station) and at that time its program score was moving upward.  City Radio was not able to sustain program improvements for at least three quarters (out of 12 monitored) and never achieved its program target. As of the third quarter of Y4, City Radio’s program score had leveled off.

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM H-2

ANNEX I: TYPES OF ASSISTANCE UN- DER SMAP SMAP assistance involved technical assistance, training, mentor- SMAP AT A GLANCE ing/coaching, subgrants (including in-kind grants), research, coalition-  80 grants awarded to 33 local building/networking activities, and study tours. Core SMAP partners re- organizations (25 under IR 1, ceived multiple forms of assistance in various technical areas, while oth- 35 under IR 2, and 28 under er beneficiaries may have only received training or networking opportu- IR 3) nities. Local partners and beneficiaries the team met with highly appreci-  124 purchase orders ated the support they received under SMAP. They described IREX assistance as being what they needed. Below, the team considers in greater  80+ training events detail how IREX sought to achieve the program objectives under SMAP  1,329 media professionals and the efficacy of various approaches, including training, coaching and trained mentoring, subgrants, research, and a variety of networking activities.

TRAINING, MENTORING, AND COACHING In pursuit of all of its intermediate results, IREX provided training on basic journalism skills, journalistic ethics, investigative journalism, niche reporting, sales and marketing, package production and presenting for television, radio formatting and programming, and a range of topics related to online media, including basics of new media, writing for the web, online editing, and multiplatform authoring. It employed a combination of centrally- and regionally-based group trainings, SMAP TRAINING ACTIVITIES ongoing mentoring, and provision of consultants to partners for one-on-one coaching. Participation in the many roundtables According to information provided by convened by IREX partners also appears to have had an educa- IREX, it provided approximately 84 training tional character for participants. activities under SMAP either directly, through local partners, or in cooperation This approach seemed to work for most partners, though in with the OSCE including: some cases more training opportunities outside of Belgrade would have been appreciated. A former IREX trainer under the  Economic/financial market reporting (6 events) ProMedia projects indicated that he believed training under  EU accession topics (2 events) SMAP could have been better in terms of in-house training ca-  Investigative journalism/reporting on pacity, more and repeated training opportunities, and refinement public finances/corruption (17 events) of training modules through testing. As discussed in the Effica-  Training of trainers (2 events) cy of New Media Activities section, some one-off training ses-  Basic radio journalism (4 events) sions for partners were insufficient to address their needs ade-  Radio formats (3 events) quately and build their capacity. In some cases, especially for  Photo journalism (4 events) smaller media outlets with only a handful of staff, trainings  Television package and presenter might have been more than they could absorb. “We got a lot of skills (17 events) trainings, even though we didn’t ask for all of them,” said a rep-  Various topics related to online/multi- resentative of a local TV station. platform media and advertising (23 events) Several stakeholders noted the perpetual need for training be-  Sales and business development (6 cause the profession is changing and due to high turnover in the events) sector, particularly since the economic crisis. Given the extent  Training for women media managers of ongoing training needs, both basic and advanced, more diver- (through the summer school programs) sified training capacity within MSOs and local training corps would have been beneficial along with earlier efforts by the program to leverage universities to provide for greater sustainability of training activities.

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM I-1 Local partners particularly valued mentoring and on-site coaching and consultations. The provision of this hands-on and tailored assistance added value to other forms of assistance (e.g., training and subgranting) and evidence suggests that this helped local partners to better apply and adapt assistance to the benefit of their organizations. While praising the quality of consultants provided by IREX, several partners agreed that having more on-site expertise would have been helpful. According to one association leader, “It would have been better if there had been more people here day-to-day to provide technical advice rather than having consultants come back and forth. The head of another membership association that received training absent mentoring or coaching told the team, “Long-term projects would have been better. Still, we had some progress as a result of the workshops.” Despite a reliance on consultants as opposed to in- house training and advisory capacity, the relationships developed between SMAP consultants and local partners continued long after their face-to-face interactions. Several partners also acknowledged the limits of training and coaching received under SMAP. Specifical- ly, and as discussed in greater detail in the report under IR 2, the financial status of some media outlets was so fragile that assistance could not have made a difference.

SUB GRANTS SMAP SUB GRANTS At the time of the evaluation, SMAP had awarded 80 subgrants across four years (see text box, right). Grants were distributed Multiple grant recipients: across each of the IRs: (1) 25 grants supported legal reform  ANEM = 10 grants totaling $575,691 activities under IR 1; (2) 28 grants sought to build financial  Radio 021 = 6 grants totaling $176, and managerial capacity under IR 2; and (3) 35 grants were 69 223 designed to increase access to quality information under IR 3.  City Radio = 5 grants totaling $62,186 Grants covered a wide range of activities from legal monitor-  UNS = 5 grants totaling $86,733 ing, advocacy, Hague reporting, content development and pro-  Simplicity = 5 grants totaling $88,175 duction, and sales and marketing support to website develop-  B92 = 4 grants totaling $162,812 ment, equipment and software purchases, premises relocation,  NUNS = 4 grants totaling $74,654 software development, and promotional activities. Typically,  Lokal Press = 4 grants totaling $86,971 program partners and beneficiaries came to IREX with ideas  Luna = 4 grants totaling $53,927 that were funded or not. This fostered locally-driven program-  BETA = 3 grants totaling $74,147 ming, the value of which was highly regarded by grantees. At  FoNet = 3 grants totaling $80,909 the same time, however, subgranting in response to immediate  Boom 93 = 2 grants totaling $37,842 needs contributed to short-term fixes that did not always con-  MTV = 2 grants totaling $127,554 tributed to a strategic or synergistic approach.  Scena = 3 grants totaling $41,795  RTV = 2 grants totaling $29,846 RESEARCH Note: An additional 18 groups received IREX placed an emphasis on research, including legal moni- one grant each. toring and analysis, polling on youth and new media, market research and media audits, web measurement systems, financial reviews, content analysis, social network analysis, and studies (e.g., most recently, related to public financing of local media). This emphasis provided an evidentiary basis for SMAP programming and for decision-making by key partners. “What we lack in Serbia is statistics. Because of IREX, we have statistics to act upon,” a representative of the international community told the team. Representatives of various research firms indicated that local media were initially afraid of what research results might show, because that they would be exposed or disadvantaged vis-à-vis their competition. “Now they understand the value of this research,” one researcher noted. Several radio industry representatives confirmed this observation.

69 Nine grants simultaneously supported IR 2 and IR 3. There were two equipment grants that were not tied to any IR.

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM I-2 While there are concerns about the ability of local actors to invest precious funds in research once SMAP ends, one market research firm and an industry group noted that they are already investing in research and plan to do so in the future, even if the frequency of the research will be less and the sample size will be smaller. According to another researcher, “commercial stations may continue to invest in research because they now understand that their financial existence depends on it.” SMAP’s investment in market research was an important to the effort to build media sustainability. In addition to market research and research related to media policy and the media landscape, SMAP also undertook various forms of research and analysis as part of its monitoring and evaluation efforts. The program’s 64-page PMP contained nearly 30 output and outcome indicators. The team found that SMAP did not use all of its indicators and that the PMP’s placed a considerable administrative burden on the program team to collect, analyze, and report data. Ultimately, the Mission could not absorb or utilize all of the data provided and reported only outcome indicator data to USAID/Washington. Throughout the program, some indicators were dropped or added, the definitions of some indicators were modified or qualified,70 and monitoring methodologies changed.71 In these cases, the team found that reliable and sustained tracking of data from reporting period to reporting period proved difficult. With respect to the most costly component of the PMP, content analysis, while the Mission indicated that this was too technically complex for their use and that they could not place the results in context, evidence suggests that the data did help to inform training plans and the evaluation team found that it proved very useful to understanding the extent to which SMAP did and did not contribute to improved story and program quality (with the exception of elections reporting by partner radio stations), and to the capacity of partner outlets to sustain higher quality standards.

NETWORKING ACTIVITIES The program also undertook a variety of networking activities and exchanges. These included Digital Days and Radio Days in Serbia and Europe, UK study tours on digitalization, and trade events, including Diskop Sales Market and the Weekend Media Festival that helped production partners to sell their inven- tory on world markets. These events were highly valued by SMAP partners. “IREX provided a bridge to the international community,” one radio station manager told the team. The team also found evidence that the knowledge and skills gained through various exchanges and networking opportunities was being applied and shared with others, providing an important multiplier effect. “My mission now is to share my knowledge with others,” a participant in Radio Days told the team. Efforts by various groups, including IAB, RAB, and ABC, to bring together media practitioners, advertisers, and sales and marketing representatives have already produced results. The greater understanding and relationships developed through these events will be critical to the future growth and sustainability of the sector.

70 For example in relation to the Advocacy Index. 71 For example, from the use of diaries to the CATI method for IRs 2.1C and 2.1D.

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM I-3 ANNEX J: DONORS’ FUTURE PLANS

The EU has a large assistance portfolio in Serbia under its Instrument for Pre-Accession Assistance (IPA) that provides direct funding of media, training and capacity building (mostly to help media cover EU issues), and work on media legislation. Media organizations are also able to access funding through EU calls for applications to support civil society programs. It also has a separate program to support the legal and technical (including equipment) aspects of the digitalization process. Beginning in the first quarter of 2013, the EU will initiate a new program to support the implementation of the Media Strategy. While this assistance will be carried out in cooperation with the GoS, the EU representative the team spoke with confirmed that public consultations would be factored into the program. The EU’s top priorities through next year include corruption, the judiciary, and media. In light of these priorities, there appear to be multiple opportunities for MSOs and media outlets to access training and grants and engage in dialogue with the government on key media reforms. Work in investigative reporting and, potentially, engaging in further work with the media and the judiciary relating to the adjudication of media cases, would also to appear to fit within the parameters of EU media support. At the same time, multiple stakeholders, particularly those from small organizations and organizations outside Belgrade, expressed doubts about their ability to access EU funding despite the fact that IREX provided training through ECG on how to prepare applications for EU funds. “The problem with EU funds is that they sometimes look like banks—they set such criteria that we are not eligible. They end up funding larger organizations that are more established.” EU funding was also deemed to be less nimble than that provided by the United States. “The EU is so complicated. You apply now, you might be discussing the project in two years,” a former high-ranking official noted. In addition to the EU, the OSCE has a very small budget, approximately $400,000 per calendar year, to work on issues relating to media policy, freedom of speech, and media training. In addition, OSI will continue to support program activities relating to media and new technologies and will continue to provide support for media associations, media reform initiatives, and the next round of broadcast licensing, which, by necessity, will involve OSI in digitalization issues. In terms of direct support to individual media outlets, “We don’t have much good news here. The budget is decreasing dramatically,” the OSI Media Pro- ject Coordinator told the team. “Compared to the past, there will also be less money for media content. We’ve seen a decrease in ideas and initiatives related to content. There’s very little worth of support,” he said. Due to the scarcity and inadequacy of proposals to develop media content in key issues, despite a clear need, OSI recently began developing economic content in house in partnership with the media centers in Niš and Belgrade and FoNet. In term of a key aspect of USAID media assistance via SMAP, namely capacity building through organizational development and business planning, it does not appear that this will be taken up by other donors.

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM J-1

METHODOLOGY

U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523 Tel: (202) 712-0000 Fax: (202) 216-3524 www.usaid.gov

PERFORMANCE EVALUATION OF THE SERBIA MEDIA ASSISTANCE PROGRAM G-1

February 22, 2013

Statement of Differences Performance Evaluation of the Serbia Media Assistance Program

IREX appreciates the information provided in the assessment and believes that on balance it captures key dynamics of the implementation of this program, while also providing important lessons learned of both successes and challenges that can benefit USAID and implementers for future programs.

IREX would like to make several points that we believe provide a fuller and more accurate understanding of the implementation of the program as it relates to specific conclusions presented in the evaluation.

 Shift from Television – The evaluation correctly notes that the original RFA issued by USAID called for a heavy emphasis on television and that USAID and IREX jointly discussed shifting the emphasis of the program to those media which were most viable and best served their local audiences. It should be noted that this shift was also strategically related to an understanding that all media outlets would be undergoing change as they responded to the growth of new media technologies, coming digitalization, and the economic and political environment.

 IR 3: Access to Quality Information. Investigative reporting: The evaluation correctly notes the difficulty in sustainability facing key online media supported by SMAP. It should be noted that this is a challenge faced by online media – particularly investigative - throughout the world. By leaving this out, the implication is that this is a problem specific to Serbia and/or the program. Leading investigative websites in developed countries such as the US are also donor dependent and struggling to find sustainable models. Serbian online media face an even more precarious situation - limited online advertising market, the potential discouragement of advertisers due to political pressures, and the lack of domestic philanthropy (which in the US provides substantial support to online investigative reporting).

 Project Management and Implementation:  IREX notes that the evaluation stated (footnote 47) that the ProMedia office closed in 2007 before the extension of ProMedia II. IREX would like to clarify that its agreement with USAID had expired and as such it had to close the office and release staff. Three months later, USAID issued a brief extension of the program during which time IREX had to re-staff entirely and restart operations. It was in the period of office closing that key technical staff found other stable jobs.  Work Planning – IREX takes note of the evaluation’s statement that work planning could have been more strategic and detailed and we believe this is the

goal of all work plans – to balance the strategic goals of the program and link them with the explanation of activities. We acknowledge that elements of the workplans could have been better prepared and better suited to the point of contact at the time of submission. However, the evaluation states that IREX was reactive without providing examples or evidence of how this was manifested in activities or design. In many cases, differences were in fact strategic or related to a difference of opinion on what constituted activities contributing towards program strategy. This is not captured in the evaluation and this lack of discussion of real differences of opinion that were debated in strategic and tactical approach fails to accurately capture the full dynamic of discussions between USAID and IREX and reduces them to a lack of detail and/or a reactive approach without substantiating information.

IREX continued to follow the basic strategic direction of the program, as encapsulated in the IRs and expected legacies of program (see Project Description), even when it was (rightly, as the evaluation notes) reacting to the economic crisis and its impact on its partners.

IREX also notes a lack of examples or evidence when it states that IREX did not provide sufficient detail in workplans and modifications. IREX, based on its prior experience with USAID projects in dozens of countries, deployed a template, style, and level of detail accepted at many other USAID missions. IREX also notes that reactions to workplans were at times presented piecemeal, including providing further new comments after IREX had responded.

Further, the evaluation team downplays the real impact that frequent staff changes as USAID had on project implementation. The USAID point of contact frequently changed and at times IREX did not have an AOR engaged with the program in any substantive manner. IREX understands these frequent changes were in part beyond USAID’s control, but they did represent a significant impact on the program as each point of contact brought a different interpersonal style, different approaches to engaging partners, different views on the suitability of proposed activities and partners, different levels of experience and understanding of media development, and different approaches to handling suggestions and comments verbally or in writing. It should be noted, as it was to the evaluation team, that the tone and nature of relations with USAID (and with partners) would change with certain specific changes in AOR/point of contact at USAID in a manner that did affect the goal of creating a cooperative working relationship between IREX and USAID.

IREX believes a lesson learned from this would be not only the need for the implementer to better detail and strategically relate activities in its workplanning, but for USAID to provide more consistent AOR support with skills in media and media development, clarity in the format and level of detail of the workplan, and timely and comprehensive feedback on the workplan in one written document.

IREX would also like to clarify the comment on accurate and up to date pipeline data. IREX acknowledges that there were at times inconsistencies in how certain planned expenses were reported in pipelines and how these were explained to USAID, but it also notes that at other times there was confusion by USAID staff on the differences between a pipeline, a formal financial report which reflect only actual expenditures, and accruals (actual and estimated) which USAID requested quarterly and which IREX willingly supplied despite this not being a formal reporting requirement. This led to repeated requests for clarification and repeated explanations of the differences between these different types of financial documents. IREX notes that, at project closing, it spent funds as planned and with minimal variation - as we consistently indicated would happen via our pipelines.

 Work Planning – COP. While IREX appreciates the recognition of the value the first COP brought to the program, it finds that certain statements regarding the ability of the COP (which the evaluation bases in part on his presence in Serbia under Milosevic) to move forward with partners on sustainability issues unfairly discredits the good work done with partners on sustainability and the abilities of the COP and lacks specific examples.

As noted in the evaluation, in retrospect even the COP acknowledges that different approaches could have been tried. However, the evaluation unfairly ties this this to his previous experience in Serbia with the implication that he favored old partners and did not understand the current environment. The lack of specific examples only exacerbates this issue and calls into question what the real lessons learned are. Examples of what IREX (largely in the person of this COP) did do to push certain partners in the direction of sustainability, even when met with resistance, are missing. Also missing are examples of why certain partners were recommended for continued participation in the program based on the value of their work to the media community even when they were resistant to changes towards improved sustainability.  Leadership transition –Clarification is required from IREX’s perspective, even while acknowledging that elevation of the DCOP to the COP position did not work out as planned and affected the program. . As the Evaluation notes, the IREX COP, with support from IREX DC, did provide extended mentoring, training, and guidance to the DCOP. This occurred from the beginning of the program and was implemented more intensely in year two – even prior to any request from USAID that this occur. Whether that training was successful do to the DCOP’s ability to take on these responsibilities is an issue but the evaluation can leave the impression that this was not done despite the fact that IREX did in good faith undertake to provide the needed training. . IREX would also like to clarify that references to discussions about the DCOP candidate did not occur with IREX DC leadership until the fall of 2011. IREX notes that (as the evaluation noted), the transition from DCOP was clear in the original CA and in modifications, including the modification with specific approval submitted in July 2011 from the DC office. IREX also notes that this relates to another conclusion of the evaluation that there needed to be better communication between IREX DC and USAID as there were prior opportunities for USAID to discuss the matter with IREX DC sooner based on submissions of modifications and requests for approval. . IREX would like to clarify a misperception regarding the provision of senior support to the Serbia office as noted on page 19. The evaluation implies (in parts of the evaluation) that as early as the fall of 2011, IREX agreed to supply a full-time senior expat to the Belgrade office. However, this was not proposed by USAID or IREX at that time but was proposed only later in the spring of 2012. When proposed and while looking for that person, IREX supplied different STTA to support the program office, including COP level support. This was in addition to enhanced support for the COP in terms of mentoring in person and virtually that occurred prior to that (and is nowhere mentioned in the evaluation). . IREX has learned lessons from this experience it will apply to future programs. IREX also believes that more frank and direct conversations from USAID would have benefited the program and believes that at times concerns were veiled and non-specific. In other programs, frank and direct conversations have helped IREX respond quickly to address concerns in management and personnel. It is not entirely, as the evaluators state, that USAID needs to better use its key personnel approval but that it needs to be specific about concerns and raise those early in a specific manner. Calls and meetings referenced in the Fall of 2011 did not provide specific concerns but general questions about the suitability of the DCOP. . IREX also notes that if USAID had earlier requested a full-time senior advisor, it could have more easily filled that position as a recruiting difficulty was the length of time for the assignment with a person of the qualifications needed. The evaluation fails to note this and states that the difficulty in finding someone for a short-term assignment to effectively shadow a COP (as candidates approached correctly understood it) would have also transferred into difficulty finding a suitable person to take on a longer assignment with a clearer technical role (as the position might have been designed if it had been proposed by USAID in the Fall of 2011).