occasionalpapers Volume IV No 4 • Autumn 2003

The Occasional Papers are pub- lished quarterly by NISPAcee and dis- Content tributed together with the NISPAcee Newsletter. The series launches academic publications that discuss Kristin Morse, results of policy research in differ- ent fields of public administration. Raymond Struyk It provides a forum for the analysis and discussion of contemporary The Policy Fellows Program: problems and possible solutions, models and methods of public ad- In-Service Training in Public ministration and public policy with assessments of empirical evidence Policy for Russian Local from regional experience in public Officials /p. 3 sector reform.

The main goal is to enhance the quality and quantity of intellectual Karine Astvatsaturova exchange among researchers, educators, scholars and practitio- ners dealing with major issues of Armenian Labor Market public administration and public Dynamics During The Last policy in the Central and East Euro- pean regions. Two Decades /p. 22 Volume IV • No 4 • Autumn 2003 papers NISPAcee occasional occasional 3 NISPAcee Occasional Papers are published with the support of the Local Government and Public published with the support are of the Local Government Papers Occasional NISPAcee , 1525 ut. 11, (affiliated with the Open Society Nador Institute), Initiative Service Reform . ISSN 1335-5945 Subscriptions be placed to without charge. Requests available additional copies are and limited Subscriptions Change of the deputy to additional copies should be addressed or for editor. on the mailing list including in advance, weeks four the publisher at least should be sent to notifications address addresses. the old and new both for publication will receive a modest honorarium. a modest publication will receive for Mr. to submission, should be addressed for including manuscripts correspondence, Editorial 42, 840 02 Bratislava 163, 5/B, P.O.Box Hanulova Secretariat, NISPAcee deputy editor, Sklenar, Juraj should be sent in Manuscripts e-mail: [email protected]. 5557, +421-2-6428 phone/fax: Republic, Slovak form. electronic Papers should be written on relevant public administration and public policy issues based on and public policy public administration on relevant should be written Papers should not countries. The papers carried European and East out in Central research empirical forty than English, a three-page in a language other exceed pages in length. If a paper is written of the length of and a description English summary with a bibliography should be submitted their submission, but the final for reviewers two propose author may document. Each the entire selected are whose papers Those authors of the editor. is at the discretion selection of reviewers Dragoljub Kavran, Government of Serbia, Yugoslavia of Serbia, Government Kavran, Dragoljub English Language Editor Language Services, Hungary Anvil Acme G. Williams, Todd Submissions Pawel Swianiewicz, University of Warsaw, Poland Poland of Warsaw, University Swianiewicz, Pawel Poland University, Maria Curie-Sklodowska Agnieszka Pawlowska, Romania of Economic Studies, Gabriela Cecilia Stanciulescu, Academy Russia University, State Ural Sergei Pushkarev, Republic of Economics, Slovak Milan Bucek, University Slovenia Ljubljana, of University Stanka Setnikar-Cankar, Mzia Mikeladze, Georgian Technical University, Georgia University, Georgian Technical Mzia Mikeladze, of Economic Sciences, Hungary University Jenei, Budapest Gyorgy Planning, Golam Mostafa, Strategic of Management, Economics and Institute Kazakhstan Kyrghyzstan of Management, Academy Salih Murzaev, Lithuania University, Kaunas Technological Domarkas, Vladislav Editorial Board Members Editorial Board Bulgaria Union, of the European Information Centre Ingrid Shikova, Republic Czech Masaryk University, Maly, Ivan Republic Czech of Sciences, Academy Michal Illner, Tartu, of University Drechsler, Wolfgang Editor Office of of the President of Public Administration, Academy Ukrainian Krawchenko, Bohdan Ukraine Kiev, Ukraine, Deputy Editor Slovakia Bratislava, Secretariat, NISPAcee Sklenar, Juraj Copyright © 2003 by NISPAcee © 2003 by Copyright Volume IV • No 4 • Autumn 2003 papers - - - - - Rus 1 NISPAcee occasional occasional For a discussion of decentralization in the former Soviet bloc, in the former Soviet a discussion of decentralization For Ebel, Bird, (1994); Wallich (1999); Kirchner example, see, for (1999); and Titov Treisman, Freinkman, (1995); and Wallich Horvath (2000).

new and valuable information that would information that would and valuable new work. them in their currentassist and future with a conducted were on interviews Follow approximately of 25 graduates sample random to of the course months after completion five on the course of the impact further evaluate performance similar produced professional of the of the evaluation The results results. that it fills a indicate course Fellows Policy local officials need for training definite and NGOs in Russia. staff at advocacy Introduction in of local The responsibilities of the delivery for republics the former Soviet transformed been utterly public services have period. since the beginning of the transition in all countries there varies, While the extent has been substantial of re decentralization localities. to sponsibility the center from were Local governments sia is no exception. task, particularly in their new for ill prepared The situa capacity. terms of policy-making of government officials and NGO analysts work work analysts NGO and officials government of the of part core a are that exercises on together analyti to introduced are Participants course. how taught and methods decision-making cal course The programs. evaluate and monitor to making, policy in concepts key covers also aspects distributional the considering as such developing benefits/pays), (who policies of balancing criteria, decision-making explicit public defining factors, technical and political competi introducing and services, and goods forms In evaluation sector. public the to tion participants during the course, completed with them reported provided that the course 1 3 ------

The Urban Institute, Washington DC, USA Washington Institute, The Urban

Kristin Morse, Raymond Struyk * Struyk Raymond Morse, Kristin Russian Local Government Officials Local Government Russian The Policy Fellows Program: Program: Fellows Policy The for Policy in Public In-Service Training * workshops, each offered over two or three three or two over offered each workshops, problem-solving a uses workshop Each days. comprised participants of groups and format, en the policy development skills of local local of skills development policy the en representa NGO and officials government four of series a of consists course The tives. Abstract strength to seeks Program Fellows Policy The ily those of the Urban Institute, the Institute the Institute Institute, ily those of the Urban Economics, or USAID. Urban for ful comments from USAID-Moscow staff at a USAID-Moscow ful comments from The opinions presentation of these materials. necessar and not the authors’ are expressed analysis; both work at IUE. We warmly thank warmly at IUE. We work both analysis; comments valuable provided Newman Sandra use on a draft. also received The authors to the course. Polina Philippova conducted conducted Philippova Polina the course. to and the evaluation for needed the interviews ably handled the statistical Tomanov Andrey in the first offering described of the course Margarita and Marina Shapiro herein. Pine contributions valuable gina made numerous for the course. The authors sincerely thank sincerely The authors the course. for the local officials and NGO staff in Chebok who participated and Saratov Kirov, sary, tinues active involvement there. Ms. Morse, Ms. Morse, there. involvement tinues active is a consultant to in Russia, currently resident the lead teacher and was Institute the Urban Mr. Struyk, Senior Fellow at the Urban Insti at the Urban Struyk, Senior Fellow Mr. direct – 98) in Russia (1992 resident was tute, and con program reform ing a housing sector gina and Marina Shapiro, who are on the staff who are gina and Marina Shapiro, were Economics, Urban for at the Institute described. of the course co-teachers Agreement between USAID and the Institute and the Institute USAID Agreement between Economics (IUE-Moscow), Urban for Margarita Pine – 00135. – A – 00 – 01 118 No. This course was developed and offered with and offered developed was This course S. Agencythe support Interna for U. of the under the Cooperative tional Development, The Policy Fellows Program: In-Service Training NISPAcee occasional papers

tion has improved modestly, mostly through passed the course. The third part reviews the a learning-by-doing process. Nevertheless, findings of two evaluations of the course – one policy analysis, program formulation, and pro- taking place during and at the conclusion of gram monitoring and evaluation skills remain the course and the second five months after very limited. its conclusion. The third section offers some At the same time local advocacy NGOs, conclusions and addresses the possibility of including think tanks in some cities, have using the curriculum in other countries in the emerged to champion specific causes and Commonwealth of Independent States. to promote more efficient and open govern- ment. 2 In a number of cities, there is active Course Overview cooperation between local government and certain NGOs, including NGOs acting as At a general level, we followed the conven- service providers under contract to the city, tional approach to teaching policy analysis in as well as those being involved in the policy the U. S. – to equip students “with intellectual making process. Nevertheless, there is still tools to aid practitioners in the identification significant wariness by many local officials and specification of policy problems and the to NGOs. Overlaying this scene is the general development of sensible, useful, and politi- cally viable solutions” (deLeon and Steelman lack of public policy skills at the local level. 3 The result is that the quality of decision mak- (2001, p. 164). A consistent market-oriented ing is impaired compared to its potential. paradigm was employed. The course described and assessed here Four Principles. is designed to improve local decision making The following principles guided the team in by raising skill levels. A second objective is designing the curriculum and teaching the to strengthen working relationships between classes. NGOs and government bodies through the inclusion of persons from both groups among Maximize the use of problems and case studies. the students. The course consists of a series Participants are adults, often persons ap- of four workshops, each of two or three days proaching or already middle-aged. Most have devoted to strengthening policy analysis skills. not taken a formal class for years. Many hold

Each workshop uses a problem-solving for- Volume IV • No 4 Autumn 2003 senior positions and are used to frequently mat, and groups of participants composed of expressing their views. These points argue officials and analysts work together on exer- for classes that minimize formal lecturing and cises that are a core part of the course. maximize the time devoted to working on con- During the spring of 2002 the course was crete problems, case studies, and role plays offered in three municipalities in European in order to sustain student interest (Lynn, Russia: Cheboksary, Kirov and Saratov. A 1999, p. 33). The majority of problems and workshop was offered in each city about once examples were drawn from Russia. a month. The information presented is based on this experience. The course is now being Decide at the outset on the main analytic skills and presented in other cities. policy principles to be conveyed and build the course The balance of the article consists of four around these. parts. First is the course overview, which begins with a statement of principles that The team first determined the set of about underpinned the development and teaching ten skills and policy principles that students of the curriculum. Next, the contents of the should master through participation in the curriculum are outlined. Finally, the issue of course that are summarized in Table 1. These institutionalization is briefly addressed. The topics were selected based on the experience second part of the paper describes the partici- of the Urban Institute (UI-Washington) and the pants and gives information on the share that Institute for Urban Economics (IUE-Moscow) in working with Russian local and national

2 For a comparative overview of developments in Russia, see 3 This is consistent with similar views expressed by Lynn USAID (2002); Charities Aid Foundation (1997). (2001) and Romero (2001).

4 5 The Policy Fellows Program: In-Service Training NISPAcee occasional papers

officials on policy issues over the past dec- enforced. The class is informed of the scor- ade, combined with our knowledge of public ing procedures at the outset: the minimum policy curricula. 4 passing grade is 40 points out of total of 65 possible. 5 Tests are administered at the end The ultimate objective is critical thinking. of each workshop and three homework as- Many officials – in Russia and elsewhere signments are distributed. Students receive – tend to operate at what might be termed continual feedback on their performance in “the descriptive level.” In other words, in the form of graded tests and homework as- discussing a problem they can describe a situ- signments. Strong homework assignments are ation and outline a proposed policy, but they reviewed in class. The faculty discusses the seldom reach the “analytic level” where the general quality of the work submitted and is problem and potential solutions are analyzed straightforward in their criticism where this is in terms of incentives that affect behavior. appropriate. Hence, the task of the course in general, and At the conclusion of the course, students the faculty in particular, is to constantly chal- who passed the course were awarded cer- lenge participants to defend their statements tificates at a graduation ceremony. It is not in terms of hard analysis of incentives, behav- clear what value the certificates themselves ioral relations, and consistency with general may have towards graduates future career principles of good management. For example, advancement. To further increase the value of when discussing the structure of a program successful course completion, graduates are that provides income support and training to eligible to participate in an alumni association. unemployed individuals, one must address This association of professional policy mak- the issue of the incentives on program partici- ers receives periodic newsletters and other pants’ effort to look for work embodied in the interesting policy materials. The initial plan way the subsidy is structured. Or, when pro- for the alumni association envisioned periodic posing to hold competitions to acquire certain events for graduates, but it proved too difficult services now provided on a monopoly basis to assemble enough representatives from by municipal firms, what are the incentives each city. to support the initiative on the part of various The Curriculum. municipal and private entities, and how could Volume IV • No 4 Autumn 2003 these affect the final outcome of the competi- The course outline is shown in Table 2. The tions? order in which topics are covered is designed to help students accumulate skills. One ses- Rigorous scoring of homework and tests, and a sion builds on the knowledge developed minimum total score to pass the course. in the previous sessions. Therefore, it is As seasoned administrators, the participants extremely important for students to attend are used to attending seminars and confer- regularly. The final workshop – devoted ences. Tests are seldom given, and certificates to writing concise, effective policy recom- are distributed to all attending regardless of mendations – presents students with policy the level of participation. This can create a re- problems similar to those they have encoun- laxed attitude about the necessity of applying tered in previous sessions. In each policy one’s self to learning the material. The Policy problem/case study, the policy issue is stated, Fellows course seeks to create greater com- its political context outlined, and options for mitment of students by increasing the value of addressing it developed. The student is then receiving a certificate at the conclusion of the charged with the task of preparing a short course. (two-to-three page) recommendation – and to To encourage mastery of the skills being defend the recommended action. This strong taught requires that rules be announced and emphasis on writing skills and preparation of policy recommendations is consistent with

4 An idea of the extensiveness of the experience can be gained 5 The 65 points are made up as follows: up to 10 points for from IUE’s Annual Report and other items on its web site, www. each of four tests; up to 7 points for each of three homework urbaneconomics.ru and from the descriptions of UI projects in assignments; and 1 point for each of five evaluation forms Russia presented at www. urban. org/TPN. that the student submits.

4 5 The Policy Fellows Program: In-Service Training NISPAcee occasional papers

Table 1 Course Objectives: Policy Principles and Skills

Subsidies. Different forms of subsidies; strengths and weaknesses of each.Illustrate various principles, e. g., consumer primacy.

Targeting. Strengths and weaknesses of alternative structures.Actual examples.

Incentives. Identifying and taking proper account of incentives to stakeholders as a key determinant of success in policy and program design. Stakeholder analysis.

Basic policy analysis process. This builds on the topics already discussed.An advantage of this order is that the students are exposed to immediately usefulmaterial before getting the more didactic presentation on the process of policy analysis.Includes defining the problem and weighting policy options against well-defined criteria.

Efficiency in the production of goods and services. Presentation of the basic economic concept. Stress proper role of government as setting the right environment (e.g., enforceable contracts) for most production but with a highly minimized actual production role. Introduce contracting out as an alternative to direct government delivery of services. Make arguments about the virtue of competition, etc.

Program monitoring. Rationale for program monitoring; give specific examples of use of monitoring information being useful to program management.Introduce modified log-frame for use in deciding what information should be collected and what reports to be produced for whom.

Data assessment techniques. Quality control in data assembly.This may be thought of as a sub- topic under monitoring or evaluation but the experience is that local officials do not review statistical tables for obvious errors that simple logical checks would identify.

Program implementation evaluation. Types of questions that can be addressed with process

evaluation and why the answers are important for good program management. Examples of good Volume IV • No 4 Autumn 2003 practices. Class exercises for defining such evaluations.

Writing policy recommendations. Hones ability to analyze problems and clearly present Recommendations.Opportunity to practice writing and critical thinking skills. Analyses of case studies require participants to use concepts from previous workshops.

what is generally viewed as good practice in at American universities, with a heavy empha- policy analysis courses. 6 (Examples of policy sis on international best practices and exam- problems are provided in the annex.) ples from Russia. The Russian team ensured The curriculum is the result of collabora- that the materials and ideas were relevant to tion between Russian and American policy participants by adding numerous local exam- experts. The team jointly identified the core ples of policies, social and economic prob- topics based on their experience of working lems, and current practice. with local and national government officials The curriculum was prepared in English and numerous think tanks in the region since and Russian, complete with instructor’s notes, 1991. These are topics that arise frequently in exercises, class handouts and problems, tests actual policy development in these countries and evaluation forms. and where officials and analysts often lack a Course format. basic grounding. The course is in keeping with the basic discipline of policy training as taught Each workshop was offered over two-full days or three-half days, with cities given the op- portunity to select the schedule that best met 6 See, for example, Musso, Biller, and Myrtle (2000).

6 7 The Policy Fellows Program: In-Service Training NISPAcee occasional papers

their needs. The course was spread over four pursue social or economic reforms. Letters months with one seminar offered in each city were sent and/or meetings were arranged with each month. the mayors of the selected cities. One large city The primary language of instruction was was not interested in the training, with local Russian. The American instructor spoke in officials claiming that their staff already under- English with one of the Russian instructors stood the topics. The course received strong serving as a translator. Formal presentations support from the administration of the selected were limited to no more than 45 minutes, with cities. This support was very important and ample time allotted for group discussions, signaled to participants that city leaders valued exercises, and student presentations. the skills and ideas and had expectations for the time spent in the course. Table 2 Policy Fellows Course Outline

Workshop 1: Critical Thinking about Public Programs and Subsidies  Types of policy actions– focusing primarily on targeting and subsidies  Stakeholder analysis – evaluating the interests/influence of different parties  Policy Analysis Model– a six step process for analytical decision- making

Workshop 2: Efficient Public Programs  The roles and responsibilities of different levels of government  Models for carrying out government functions: direct provision, contracting out, and divestiture/ privatization  Designing contracting out for a concrete municipal service

Workshop 3: Program Monitoring and Evaluation  Program monitoring– what to track, why, how?  Program evaluation– assessing the implementation process and/or outcomes

 Using data effectively Volume IV • No 4 Autumn 2003

Workshop 4: Preparing Policy Recommendations  Writing and presenting policy recommendations

This workshop also serves as an opportunity to review the Policy Analysis Model covered in the first workshop and allow participants to practice and demonstrate skills and ideas from throughout the course.

City Selection. Institutionalization. The course is part of a broader initiative to A critical objective of the Policy Fellows strengthen local governments in the Volga program is to insure that a Russian institution Federal District (VFD) – and all cities selected will be fully capable of teaching this course to participate were VFD cities. The Institute for in the future. The project was conceived as a Urban Economics identified a handful of cities joint undertaking of the Urban Institute and to approach about the course. The criteria the Institute for Urban Economics. Work was for city selection were: cities with whom the facilitated by the fact that the two organiza- organization had some prior experience, those tions have worked closely together since IUE that would not have a mayoral election within was founded in 1995. the next year (which would make it very hard In addressing institutionalization, the for officials to attend the course), and those project team has the advantage that the that had demonstrated some willingness to Policy Fellows program begins with a three-

6 7 The Policy Fellows Program: In-Service Training NISPAcee occasional papers

year funding commitment and program plan. ment officials. The small groups often pro- During the first year, UI took the lead on vided an informal environment where good curriculum development and provided the ideas prevailed and position did not appear to lead instructor for the workshops in the three matter much. Willingness to participate was a cities. The curriculum was refined with the significant factor, and several NGO representa- help of two IUE staff who also team-taught the tives clearly saw the course as an opportunity workshops with the UI instructor. In project for them to engage local officials as much as Year Two, IUE is teaching the course with possible. Others made smaller contributions, some input and monitoring by UI. In the third which seemed largely driven by their limited year, IUE will teach the course on its own. skills and experience. In one city, NGO repre- In the last year IUE will also search for sentatives were simply not as experienced as continued funding for the course. By that the city officials. time the relation between a version of the Nearly 72 percent of participants success- Policy Fellows course for national officials and fully completed the course requirements. the version for local officials will be better Unlike many professional training programs in defined, as will be alternative institutional ar- Russia, Policy Fellows required participants to rangements for possible further dissemination submit work and demonstrate mastery of the of the course. skills/materials covered in the course. That certificates were not guaranteed to Participants and Graduates participants provided incentives for students Sixty-seven students participated in the to attend and apply themselves. The majority course. Participants from the municipal gov- of those that failed the course attended only ernments in each city were appointed through one workshop – seeming to select out of the a process directed by a deputy mayor. Most course. Others that failed the course were of the city officials held senior administrative those that were unable to make the necessary positions, including that of deputy mayor, de- time commitment to attend the workshops partment chief, and deputy department head. and complete the work. Only a few partici- We believe that most students had little infor- pants who applied themselves failed to accrue mation on the course before they attended the the points required to complete the course. first class, despite the team providing descrip- Government officials and NGO representatives Volume IV • No 4 Autumn 2003 tive materials to the deputy mayor. To ensure seemed to pass or fail at similar rates. strong support from the city administrations, The course received strong support from city officials comprised more than three- the administration in each city. IUE staff met fourths of participants. NGOs were recruited with the mayor or vice mayor in each city by IUE with some input from cities. during the city selection process and the city The NGOs were a small minority in each leadership cooperated by appointing vice class. The curriculum mostly describes mayors and department heads to participate policymaking and program development from in the course. Cheboksary and Kirov organ- the perspective of local government, with the ized press conferences during and at the end understanding that this is relevant to NGOs of the course. with an interest in participating in public policy and that sections on program manage- Evaluation ment and monitoring are relevant for any The team conducted two types of evalua- organization. The expectation was that adding tion of the course. First, at the conclusion some NGO representatives to a course largely of each workshop, students were requested for public administrators would elevate the to complete an evaluation form tailored to policy skills of NGOs, help government of- that workshop. To encourage submission of ficials recognize the potential role for NGOs, the evaluations, students received one point and provide a few NGOs with access to public toward graduation for each evaluation submit- officials. ted. In addition to inquiring about the format The group exercises helped to facilitate and quality of the presentations and materials, interaction between the NGOs and govern- the form also asked questions about whether

8 9 The Policy Fellows Program: In-Service Training NISPAcee occasional papers

Table 3 Number of Participants and Graduates

Participants Graduates

% successfully City Government NGOTotal Government NGOTotal completing course

Cheboksary 14 5 19 13 3 16 84.2%

Kirov 15 3 18 12 2 14 77.8%

Saratov7 23 7 30 14 4 18 60.0%

Total 52 15 67 39 9 48 71.6%

the material covered on specific topics was lent) based on the quality of the materials, new to the student and how relevant students organization, exercises, and presentations. judged the topic to their work. Individual topics were also highly rated (no Second, to obtain further information on topic received a rating lower than 4. 4) and the the impact of the course, the team tried to topics deemed most relevant included, in rank interview a random sample of students and order: their bosses in each city about five months program evaluation after the completion of the course. The objec- policy analysis of public programs tive was to determine if the students had been developing policy recommendations able to use the materials they learned in their program monitoring daily work. The interviews also asked the data assessment students for their views about topics that are stakeholder analysis appropriate for the course. options for carrying out local government

First Evaluation functions Volume IV • No 4 Autumn 2003 As noted, participants completed evaluation government functions in a market economy forms at the end of each workshop and a final targeting evaluation at the close of the course. Based subsidies on these forms, participants reported that the contracting out course provided them with new and valuable We were concerned that the course might information that would assist them in their be too sophisticated or too simple and par- current and future work. They also praised ticipants were asked for feedback on the level the structure of the course and stated that the of the material/presentation of each major group discussions and exercises provided topic. Participants rated over 80 percent of the a unique opportunity to interact with their course topics as being presented at the right colleagues, to better understand the ideas level. Ninety-eight percent of participants said presented, and to connect the course material that they would recommend the course to a with the problems currently confronting their colleague. cities. Participants were asked to assess each Quality of Course. topic based on whether they deemed it use- Using a five-point scale, participants rated the ful to their current work, future work, or not course with an average score of 4. 86 (5=excel- relevant. These assessments are aggregated by workshop and included in Table 4. Over 7 A large number of new people showed up for the second 90 percent of all workshop topics were judged workshop in Saratov – only some of whom continued to useful to current or future work. The compara- participate in the course. Any attendee that completed a test or homework assignment was considered a participant and is tively large percentage of participants rating included in this table.

8 9 The Policy Fellows Program: In-Service Training NISPAcee occasional papers

Table 4 Workshop Ratings and Relevance

Topics Useful to Work Rating 8 Current Future Not Useful

Workshop 1: Critical Thinking about Public Programs 4.6 72.2% 27.3% 0.6% and Subsidies

Workshop 2: 4.9 53.6 45.0 1.4 Efficient Public Programs

Workshop 3: 4.7 71.2 19.8 9.0 Program Monitoring and Evaluation

Workshop 4: 4.9 82.6 13.0 4.4 Preparing Policy Recommendations

Workshop 3, on program monitoring and “When I presented at the Cheboksary Regional evaluation, as “not useful” may well reflect the Sport Committee the program on community sports near total absence of program monitoring and facilities development ‘My yard is a sport yard.’ This evaluation by municipal administrations. program was awarded first place in the regional contest of innovative programs in youth policy.” Course Impact. (Cheboksary) The ultimate goal of the course is to improve how participants perform their jobs, specifi- “Providing subsidies for housing and communal cally, that policy decisions will be made more services and providing subsidies for individual analytically and programs will be designed housing construction.” (Cheboksary) Volume IV • No 4 Autumn 2003 and implemented more effectively. In evalua- “Preparing a regional small-business development tion forms completed after the final workshop, program.” (Kirov) a surprising 89 percent of participants said that they had already applied skills derived “Reexamined current programs. We also conducted from the course. Some of their specific exam- a competition of social programs called ‘Kirov is a ples of when they used the course are high- Cultural Capital for Youth’ (utilizing the following lighted below: topics: targeting and stakeholder analysis).” (Kirov) “At the present time I work on socio-economic pro- “Calculation of tariffs. Preparing socio-economic gram development for Chuvashia municipalities. development programs.” (Saratov) During the last three months my work was based on your course. On March 14, 2002, I led a meeting of “Monitoring targeting of regional social programs.” directors of economic departments from different (Saratov) Chuvashia municipalities where we established a structure for policy/program development with the “Preparing for municipal procurement competi- use of workshop materials. For the city of Chebok- tion.” (Saratov) sary we worked out the city socio-economic pro- The examples provided included items gram with the help of IUE experts”. (Cheboksary) that were directly covered in the course “Calculation of tuition fee for evening schools and (evaluation, targeting, competitive procure- calculation of prices for tickets to city entertain- ment, etc.), as well topics that were not specifi- ment facilities.” (Cheboksary) cally covered such as calculating tariffs – but for which presentations on benefit calculation, stakeholder analysis, and policy analysis may 8 The scale ranges from 5 (excellent) to 1 (poor).

10 11 The Policy Fellows Program: In-Service Training NISPAcee occasional papers

have contributed. That participants credited skills, and suggestions for improving the the course for helping them to prepare city course. The in-person interviews were con- and regional budgets, calculate tariffs, and ducted by an IUE staff member familiar with analyze regional problems suggests that the the course but who had not been a teacher. broader principle of the course – rigorous ana- Participants were asked specifically how lytical thinking – was successfully conveyed. they applied ideas/skills from the course to This is a dramatic change from early work- their work and asked to offer specific recom- shops when participants asked for examples mendations. They were also asked if they per- specific to their fields and many seemed un- formed any aspects of their jobs differently, able to apply key concepts or best practices to or if they assumed any new responsibilities a range of problems. that they attributed to skills/knowledge gained from the course. A similar interview question- Follow-on Evaluation naire was designed for the supervisors of While the foregoing information is definitely course participants, and asked these supervi- useful in assessing the success of the course sors if they had observed any change in staff in communicating information to the students, performance after completing the course. it does not address the more important ques- Table 5 shows the number of students and tion of the sustained impact of the training on supervisors interviewed for the study. The re- on-the-job performance. This is the case for search plan was to interview 7 – 10 graduates two reasons. First, at the time of the initial as- and 3 – 4 supervisors in each city. Graduates sessment the students had not had much time were selected for interviews randomly, with or occasion to use the material gained through the exception that we made certain each sam- the course, although a surprising volume was ple included representation from the NGOs. reported. Second, all the information came Supervisor feedback was extremely limit- from the participants, and they may have an ed and has not been included in our analyses. incentive to overstate the utility of the course Many participants were supervised directly to please the instructors or may simply have by the mayor, and mayors were generally as- inflated expectations about how they might be sumed to be unavailable for such interviews. able to use the new techniques in their jobs. However, several supervisors (including the In designing the follow-up evaluation, we deputy mayors of Cheboksary and Kirov) par- Volume IV • No 4 Autumn 2003 generally followed the methods outlined by ticipated in the course along with members of Taschereau (1998). In short, we planned a fol- their staffs. Two supervisors were interviewed low-up survey of a random sample of students and provided feedback on three participants. and their supervisors several months after the Although their comments were generally completion of the course, with some ques- positive about the course and they stated tions specifically aimed at finding out if and that their employees had benefited from the how the course had been used in their work, course, they offered few specific examples. impediments to utilizing course materials/

Table 5 Total Graduates and Sample Sizes

Number Number interviewed City of Graduates Graduates Supervisors 1 (supervisor Cheboksary 16 7 of 2 participants)

Kirov 14 8 1

Saratov 18 10 0

10 11 The Policy Fellows Program: In-Service Training NISPAcee occasional papers

Of the 25 graduates interviewed, 19 (76%) Forty-four percent of graduates inter- worked for government and 5 worked for viewed reported that they had encountered NGOs. Graduates were high-ranking officials, obstacles to their attempts to implement with 84 percent serving as department chiefs ideas/skills from the course. Lack of support or deputy mayors. Thirty-three percent had from supervisors and colleagues were the been trained as economists and the remain- obstacles most frequently cited. der held degrees in social sciences, math- As in the preliminary evaluation, gradu- ematics, engineering, education, medicine, ates identified the group exercises as among psychology, physics, or literature. the most useful elements of the course. Graduates appeared to be quite satisfied When asked if they performed any aspects of with the course, rating the overall quality and their jobs differently as a result of taking the usefulness of the course 4. 65 (5=excellent) course, 28 percent reported that they were and 96 percent reporting that they would refer more likely to work with others. a colleague to the course. Ninety-six percent A secondary goal of the course was to reported that they had been able to apply encourage collaboration among government ideas/skills from the course to their work, with officials and NGOs, and each class included roughly half offering competitive procurement several representatives from local NGOs. Ap- and implementing monitoring programs as proximately half of those interviewed report- specific examples (Table 6). Many participants ed that the course had changed their impres- credited the course with helping them to sion of how these groups might collaborate. systemize their understanding or approach to Several mentioned that the course facilitated policy issues. Sixty-four percent said that they greater understanding and that there had had shared information or materials from the been more cooperation. course with others.

Table 6 Application of Course Materials

What specific things from the course have you Volume IV • No 4 Autumn 2003 Number Percent been able to apply to your work?

Analyze public policy programs and make informed 7 28 decisions (including developing new programs)

Consider stakeholder interests (how programs/ policies 9 36 affect different groups)

Establish criteria (formally or informally) for making 3 12 decisions

Present recommendations on policy issues 3 12

Make recommendations about how to involve 3 16 the private sector

Facilitate a competition 14 56

Implement or improve monitoring program 12 48

Design or conduct an evaluation 4 16

12 13 The Policy Fellows Program: In-Service Training NISPAcee occasional papers

As a final step, we decided to undertake were less likely to report positively on their some exploratory multivariate analysis of the experience after the course. variation in success in applying the knowl- edge gained from the course. In general terms, “Reform resistant cities.” success can be considered to depend on One or another of the cities may have been three broad factors: the ability to formulate a conservative and non reform-oriented. change in policy (in our case, the student’s Motivation for introducing reform mastery of the material), opportunities for making change (or conversely resistance to Negative student attitude during the course. change), and the professional’s motivation to Students who were clearly bored, uninter- introduce change. 9 This general model was ested, and/or disruptive during the course operationalized through the following hypoth- would be less likely to master the course eses and corresponding variables. materials and to judge the newly acquired skills valuable for their jobs. 10 Mastery of the material Younger participants are more likely to apply the Score for the course. newly acquired skills. Participants who demonstrated a stronger Younger students are on average more mastery of the course materials will be bet- open to new methods and concepts and ter equipped to apply the newly acquired more flexible in their thinking about how to techniques in their jobs. apply them. Education. Among government workers, students with fewer The course introduces a number of eco- years of service nomic concepts. So participants with train- are likely to be more open to change. ing in economics were more likely to apply the lessons of the course in their work. Gender. It is possible that either women or men will Opportunities for and obstacles to systematically be more assertive introduc- introducing changes ing change. A higher share of women gradu- Students from NGOs use more of the skills acquired. ated from the course, suggesting greater Volume IV • No 4 Autumn 2003 The informal environment and frequently interest, although they did not score higher shifting agenda of most advocacy NGOs overall than men. affords more opportunities to employ the We tested whether there was support for new skills than the corresponding situation these hypotheses by using multivariate tech- in government offices. niques to estimate a model where the unit of Participants in higher government positions were analysis is the participant and the dependent more likely to apply the newly acquired skills. variable was the participant’s overall rating “of Persons at higher levels in an administra- the quality and usefulness of the course.” Re- tive apparatus have greater possibilities to spondents gave a numerical rating, with a 1 to apply new concepts, techniques, and skills. 5 range, 5 being the highest score. The mean Perhaps most importantly, they may have value of the response was 4. 65. This question the power to introduce some concepts to was asked at the end of the interview, i. e., functions for which they are responsible. after specific questions on how the participant had been able to apply the course to his/her Obstacles. work, whether they did some aspects of his/ Those who reported obstacles to using the her job differently and related questions. So skills gained in the course, such as a lack of the response should give a summary judg- support from fellow workers or supervisors, 10 An analysis of the course scores for the 12 percent of students who were so classified by the instructors did not 9 This model is consistent with standard textbook treatments show that they scored significantly differently from other of policy development. See, for example, Patton and Sawicki students. In fact there was little significant variation in scores (1993), Chapter 2. among groups of students.

12 13 The Policy Fellows Program: In-Service Training NISPAcee occasional papers

ment with strengths and weaknesses freshly Conclusions recalled. The results of the evaluation of the Policy It is worth noting at the outset two limita- Fellows course indicates that it was well-re- tions associated with the dependent variable ceived by participants and that it fills a defi- that limit the findings. First, the variable com- nite training need for local officials and staff at bines rating on course quality and usefulness; advocacy NGOs in Russia. More specifically, this introduces some potential measurement the topics were relevant and useful to partici- error since the analysis is emphasizing the pants’ work and the method of instruction and usefulness aspect when a respondent may organization of the course were highly rated. have focused his or her rating on course qual- Participants credited the course with help- ity. Second, there is limited variation in the ing to “systematize” their approach to public scores. Respondents scored the course at 4, policy issues, suggesting that the course’s 4. 5, and 5. The coefficient of variation for the ultimate objective of fostering critical thinking dependent variable is about 70. has been met. In addition to this general shift The independent variables included cor- towards analytical thinking, many participants respond to the hypotheses stated above. In offered concrete examples of how the course particular they were: had assisted them in developing programs, designing monitoring programs, and facilitat- • student’s score for the course; ing competitions. • a dummy variable indicating a participant Since the staff of municipal governments who had appeared not to be very interest- throughout the CIS share the problem of ed during the course; their Russian counterparts in being poorly • a dummy variable indicating the student equipped for many of the duties that have worked for an NGO; been thrust upon them during the transition, • the student’s age (dummy variable for an obvious question is whether the Policy over 50); Fellows course could be adopted for use • a dummy variable for the number of years elsewhere in the region. The short answer government staff had worked for the gov- is “yes, with some effort.” Success will likely ernment; rest on two elements: the curriculum and the

• a dummy variable indicating whether the Volume IV • No 4 Autumn 2003 instructors. With respect to the curriculum, student claimed education in economics the course is oriented to Russia in its contex- • a dummy variable indicating the student tual material and examples in order to gener- was a female; ate student interest. To be effective, even in • two dummy variables for cities; and other CIS countries, moderate changes would • a dummy variable indicating a government be needed in these areas so that the mate- worker in the position of department head rial would be immediately relevant to course or higher. participants. Indeed, one area where the team So there were a total of 10 independent is introducing changes in the course is in variables and 24 observations (a variable was widening the array of countries from which missing for one respondent, and so this ob- cases are drawn. Somewhat to our surprise, servation was excluded), leaving 13 degrees of freedom. This is clearly a small sample for 11 The multiple regression performed marginally better and the analysis. This factor and the measure- these results can be summarized as follows. The estimated model was weak: an adjusted r2 of. 32 and significance at ment error in the dependent variable require the. 07 level. Only three of the hypotheses were actually that the results be treated strictly as sugges- supported, and two of these only at the 10 percent level of tive. The model was estimated with multiple confidence. First, those who demonstrated a poor attitude during the course were less likely to use the results in their regression and multinomial logit techniques work or rate the course highly. Second, female participants – the latter is preferred given the distribution gave higher ratings. We queried the instructors for why this might be the case, but they reported no evident differences of the dependent variable. In the event, the in participation between male and female participants. Third, results for the multinomial logit models were students in the Saratov offering gave significantly higher wholly insignificant, suggesting little support ratings. The city and region are known for being progressive 11 and this fact is apparently reflected in their ability to use the for the hypotheses stated above. course materials in practice.

14 15 The Policy Fellows Program: In-Service Training NISPAcee occasional papers

students asked for a wider range – both in the sectors and the countries used as examples and cases. Regarding the instructors, it would be wrong to assume that it would be possible for a university professor in a CIS country to take the Russian version of the course materials and effectively teach the course. Even with the large volume of notes and supplemental readings provided in the course materials, these materials are not a substitute for close study of textbooks or, even better, careful mentoring by an experienced policy practi- tioner. The team teaching method used by UI-IUE in the initial course offering provided the essential mentoring to two senior IUE staff members who already possessed a great deal of experience working with local officials on policy development and program implementa- tion. Our recommendation is that this team- teaching model be followed in other countries. Volume IV • No 4 Autumn 2003

14 15 The Policy Fellows Program: In-Service Training NISPAcee occasional papers

References ton, DC: USAID, Bureau for Europe and Eura- sia, Office of Democracy and . Bird, R. M., R. D. Ebel, C. I. Wallich. (1995), De- centralization of the Socialist State (Washington, DC: Wallich, C. (ed. ) (1994), Russia and the Challenge , Regional and Sectoral Study). of Fiscal Federalism. Washington, DC: World Bank. Charities Aid Foundation International. (1997), Working with the Nonprofit Sector in Russia. London: Basic Books, 1997. deLeon, P., and T. A. Steelman. (2001), “Making Public Policy Programs Effective and Relevant: The Role of the Policy Sciences,” Journal of Policy Analysis and Management, vol. 20, no. 1, pp. 163 – 71 Freinkman, L., D. Treisman, and S. Titov. (1999), Subnational Budgeting in Russia: Preempting a Potential Crisis. Washington, DC: World Bank Technical Paper no. 452. Horvath, T. M. (2000), Decentralization: Experi- ments and Reforms. Budapest: LGI Press. Kirchner, E. J. (ed.) (1999), Decentralization and Transition in the Vizegrad (New York: St. Martin’s Press. Lynn, L. E. (2001), “The Changing Public Policy Cur- riculum,” Journal of Policy Analysis and Man- agement, vol. 20, no. 1, pp. 161 – 71 Lynn, L. E. (1999), Teaching & Learning with Cases: A Guidebook. New York: Chatham House. Morse, K., M. Pinegina, C. Romanik, M. Sha- piro, and R. Struyk. (2002), “In-Service Train-

ing in Public Policy for Russian Local Govern- Volume IV • No 4 Autumn 2003 ment Civil Servants and Advacacy NGO Staff: The Policy Fellows Program.” Washington, DC: Urban Institute, Report to USAID, processed. Musso, J., R. Biller and R. Myrtle. (2000), “Tradecraft: Professional Writing as Problem Solving,” Journal of Policy Analysis and Management, vol. 19, no. 4, pp. 635 – 46. Patton, C., and D. S. Sawicki. (1993), Basic Methods of Policy Analysis and Planning. Englewood Cliffs, NJ: Prentice Hall. Romero, F. S. (2001), “The Policy Analysis Course: Toward a Discipline Consensus”, Journal of Policy Analysis and Management, vol. 20, no. 4, pp. 771 – 79. Taschereau, S. (1998), Evaluating the Impact of Training and Institutional Development Programs: A Collaborative Approach. Washington, DC: World Bank, Education Development Institute. U. S. Agency for International Development [USAID]. (2002), The 2001 NGO Sustainability Index for Central and Eastern Europe and Eurasia. Washing-

16 17