External Evaluation of the EU-funded

Project "Supporting socio-economic

Integration of IDPs in Samegrelo - Zemo

Svaneti Region"

Tbilisi March 2011

Executive summary

The report presents the results of evaluation of the project "Supporting socio-economic Integration of IDPs in Samegrelo-Zemo Svaneti Region" implemented by dvv internation in partnership with Acción contra el Hambre (ACF), through the funding of the European Commission grant, starting from September 2009 for the period of 18 months, and with the overall budget of EUR 1 mln.

The project was based on the use of an integrated approach to community development, extending opportunities and economical support to both IDPs and the host populations, and aimed at promoting social integration of the IDPs, their empowerment, and the enhanced dialogue with authorities. It was implemented in three settlements in Samegrelo – Zemo Svaneti Region, namely in the town of (), formerly Military Settlement in (Senaki MS) and the village of Menji (Senaki municipality).

The expected results of the project implementation were supposed to be: 1) Integration and social cohesion within the target communities are improved and \IDPs and representatives of the host communities are empowered to play an active role in the public, economic, social and cultural life of the region and the country; 2) Policy dialogue and communication between IDP-related stakeholders and authorities on local/regional/national levels are improved, awareness on IDP issues raised, and a contribution to better governance has been made. These results were to be achieved through a number of activities, namely:

1.1. Creation of Community Development Groups (CDG) and implementation of Community Projects (CP); 1.2. Implementation of Community Agricultural Support Programme (CASP); 1.3. Establishment and operation of Community Education Centres; 1.4. Delivery of educational programmes (vocational training and personal development courses), and social services to address the identified constraints in the targeted communities; 1.5. Implementation of Youth & Citizenship Programmes; 1.6. Support of local civil society initiatives 2. 1. Capacity building of local administrations (LAs), including provision of consultancy, to support coordination and improved response to IDP-related issues; 2.2. Training of community leaders; 2.3. Promotion of awareness raising on IDP-related issues. The overall goal of the evaluation was to produce information on the outcomes of the project and to assess the overall efficiency of the project interventions, their sustainability and effectiveness. This information and the relevant recommendations should be made useful as a basis for designing, planning and implementation of further programmes and activities contributing to the integration of internally displaced persons. Respectively, the objective of the evaluation can be also formulated as to provide an independent assessment of the project's relevance, quality and impact of activities implemented by dvv international and its partner – Acción contra el Hambre (ACF) – on the lives of internally displaced people and their integration into local society.

The evaluation methodology was based on a number of objectives and principles, following the guidelines of “Evaluation Methods for the European Union’s External Assistance” , prepared by the EuropaAid Co-operation Office. The following evaluation criteria have been applied to assess the outcomes of the project: Relevance; Effectiveness; Efficiency; Sustainability; Impact; Replicability; Demonstrative effect; Coherence/Complementarity.

Desktop study of available relevant documents, such as: dvv international and ACF project documents, needs assessment and programmatic reports, site-visit reports, and related documents; as well as other materials that provided additional information on the region and IDP needs, was followed by study trips to the region, to project management and implementation sites, meeting of dvv international and ACF project staff, also interviews with beneficiaries, municipality officials and civil activists involved in the project. The main instrument was face to face in depth semi-structured interview (audio-recorded), based on a several versions of a general questionnaire, adapted to different types of respondents. An auxiliary instrument was group sessions/interviews, with fewer discussion themes selected but based on the same questionnaire.

All the activities were planned to be implemented by the Project Implementation Team (PIT) created jointly by both implementing partners, and through highly participatory approach, involving local host and IDP communities. This approach was to draw from successful experience of projects implemented by both partners in various regions of , including Samegrelo – Zemo Svaneti, Samtskhe- Javakheti and .

The evaluation revealed a very positive impact of the project implementation, and its outcomes fully correspond to the project proposal approved for EU funding. It should be stressed that the evaluation revealed a very positive impact of the project implementation, and its outcomes fully correspond to the project proposal approved for EU funding. In addition, all beneficiaries and representatives of the population or municipality leadership the evaluator has talked to, expressed appreciation and satisfaction with regards to the project, so far, no negative words whatsoever with regards to the project have been voiced by any party. At the same time, the evaluation mostly focused on the opportunities and ideas as to how improve the project further, in case a similar undertaking was carried out in the future. Such suggestions address both the project design and the project implementation.

The evaluation attempted to answer to a number of following questions:

• Are the objectives of the project met and to what extent?

Both the overall objective of the project – “to contribute to increased social cohesion in Samegrelo- Zemo Svaneti region”; and the specific objective – “to support the integration of IDPs in Samegrelo- Zemo Svaneti region through an integrated programme of community development, capacity building initiatives (for local authorities and community members) and socio-economic assistance” are definitely met.

• Do the activities respond to the needs and requirements of the project beneficiaries?

The activities do respond to the needs and the requirements of the project beneficiaries, i.e. needs for better integration, education, improving quality of life, and getting access to business opportunities.

• Are the activities and outputs of the program consistent with the intended impacts and effects?

The activities and outputs of the program are fully consistent with the intended impacts and effects, namely with the two expected results: 1) “Integration and social cohesion within the target communities are improved and IDPs and representatives of the host communities are empowered to play an active role in the public, economic, social and cultural life of the region and the country”; and 2) “Policy dialogue and communication between IDP-related stakeholders and authorities on local/regional/national levels are improved, awareness on IDP issues raised, and a contribution to better governance has been made”.

• How efficient have resources (costs, timeframe, personnel, etc.) been used in achieving the objectives of the project?

The evaluation has revealed that resources (costs, timeframe, personnel, etc.) been used efficiently in achieving the objectives of the project.

• What proportion of the target groups was reached by the project?

Significantly more than initially planned number of beneficiaries has been reached by the project. So, e.g. 9 agro-business groups have been created and supported instead of 6; or, in addition to educational courses, legal, medical, and psychological consultancy have been provided for free to all those who needed these.

• To what degree have been project partner organisations successful in developing and maintaining relationship with relevant institutions and actors in order to increase the impact of the project on the targeted population?

The project partner organisations have been quite successful in developing and maintaining relationship with relevant institutions (e.g. municipalities, media) and actors (partners and subcontractors such as CTC or CSRDG) in order to increase the impact of the project on the targeted population.

• Are there any established channels of communication between the target groups and the partner organization that will remain in force after the end of the project? There are established channels of communication between the target groups and the partner organization, however it remains to be seen for how long these will stay in force after the end of the project, partly due to its limited duration, and partly due to the risks to the sustainability of at least some outcomes of the project, as discussed above.

• Will the outputs produced within of the project accessible to the target groups beyond the duration of the project?

Some of the outputs produced within of the project will be accessible to the target groups beyond the duration of the project, such as CECs and services provided through these, or the outputs of Community Projects (e.g. such as the kindergarten in Senaki MS).

• Are there any unintended or unexpected impacts, positive or negative?

There any a few unintended or unexpected positive impacts, e.g. those related to the possibility to replicate success with the project and some of its activities in other projects and different environments. With regards to negative impacts, it remains to be seen whether hope and self- confidence brought in by the project implementation do not turn into disappointment and learned helplessness if the lives of beneficiaries do not improve after the finalisation of the involvement in project activities. Still, there is quite high probability of sustainability of achieved outcomes.

• What are the general strengths and the weaknesses of the project?

The general strengths of the project is its complex approach, complementarity and synergy of different activities, and the focus on participatory approach; and of course, the activation and dramatic improvement of the quality of life for many beneficiaries of depressed communities.

The weaknesses may be related to its duration (e.g. to short compared to agricultural development cycle); limited capacity to help finding jobs after passing through educational course; leaving out the most vulnerable in the income generation/business support activities; relying too much on personal opinion of potential beneficiaries when planning activities (such as educational courses, or community projects) without pro-actively informing them about possible options that they know little about.

• Was there any effective needs assessment conducted before starting the project, and how well were its findings taken into account?

There have been conducted needs assessment before starting the project, and their findings have been taken into account. However, the question remains how far needs assessment based solely on opinion surveys can be effective, without also involving expert opinion.

• How effective are feedback mechanisms, if any, incorporated in the project? What are mechanisms to consider complaints?

Due to permanent contact between the project staff there are effective channels of communication with beneficiaries and feedback mechanisms that characterise the project. However, there do not seem to exist any explicit mechanism for considering complaints, even though this has not created any problem to date.

• Are any benchmarks and quantitative/qualitative criteria of success/failure explicitly formulated?

Yes, there are explicitly formulated benchmarks and quantitative criteria of success/failure. However, more could be done to design effective qualitative criteria as well.

• Are the achieved results sustainable? When dvv international/ACF withdraw, what will happen?

The biggest question is how far all the achieved results are sustainable when dvv international/ACF withdraw. Some of the results are definitely sustainable and the impact long term, but in other cases things are more open to change.

• Is there any effective information strategy behind the project?

There is certain information strategy behind the project at different levels (local/regional/national, but its effectiveness may have left some room for improvement.

• How flexible should the project be to better adjust to regional differences in case of its replication elsewhere?

Some of the activities within the framework of the project are sufficiently flexible and can easily adjust to regional differences in case of its replication elsewhere. In case of other activities, changes may appear necessary (e.g. restriction to agricultural business may not be equally efficient in urban environment, or when land and other resources are too scarce).

Based on the answers received for the above questions, the following criteria of success/failure were considered:

Relevance:

The project concept and approach applied by dvv international and ACF are fully relevant, and the objectives of the intervention are fully consistent with beneficiaries/community/country needs, also partners' and donors' policies.

Effectiveness:

All the objectives of the project have been achieved. Efficiency: While this was not thoroughly evaluated, still the evaluation demonstrated that the resources and inputs (funds, expertise, time, etc.) have been cost-effectively converted into results, i.e. in achieving project objectives Sustainability: The continuation of benefits from the project implementation after it has been completed, including the probability of continued long-term benefits, and the resilience to risk of the net benefits over time is different for different activities and outcomes. While the expectation is that most of impacts are sustainable, further observation may be needed to check the issue of sustainability. Impact: The positive short and long-term effects produced by the project implementation are evident, both direct and indirect. However, the scale and positive quality of the impact of the project implementation on social integration of the target groups may be bigger if more attention was paid to multiplier/demonstration effects.

Replicability:

Possibility for this or similar approach to be effectively applied in different location and environment, although with the similar set of problems and needs. Still, some activities, such as restriction to agro- business, may appear less effective in different (e.g. more urbanised) environment.

Demonstrative effect:

There is an impression that more effort is needed so that the implementation of the project could serve as a demonstration of more innovative technologies/ideas/approaches that could be independently replicated by other community groups with similar needs and resources. Coherence/complementarity: The project design and implementation are fully consistent and seem to well complement other programmes and projects funded by the EU (e.g. as stated in ENP AP), and also well correspond with the country's policies and development strategies.

As a result of evaluation, a number of general and activity-specific recommendations were developed based on the data obtained during the evaluation, and drawing from the main findings and lessons learned. Some of the more general recommendations are as follows: o Sustainability issue should be thoroughly re-assessed for each of the different activities, and is of particular importance in case of agricultural projects, or training community leaders that may appear unable to apply learned skills. o As the limited duration of the project implementation (18 months) would not allow for ensuring effective long-term results in at least some of activities, the future project designers (and especially donors) may think of reconsidering either the duration, or the most time-sensitive activities, also thinking of how to incorporate follow-up in the project design. o When several partners are working together within the framework of the project, one could expect more built-in synergy between different activities carried out by the partners, and restructuring proposals so as to avoid overlap of responsibilities. o The location of the three project implementation settlements should be carefully selected (by future project designers and donors) so as to optimise the conducted activities, i.e. more rural communities with better land access in case of focus on agriculture (even with lower IDP concentration), or more urban in other cases. o Complementarily of expertise and activities of dvv international and ACF may be possibly supplemented by another partner/subcontractor that could contribute to fill in the missing capacity in such areas as e.g.: professional orientation (helping to better understand ones potential and options), labour market analysis, employment support, marketing of produce, and capacity building of newly created NGOs. This once again leads to the issue of project duration, which may appear too short for such work. o One may consider structuring of the future project in a more consistent way around such synergic goals as: a) promoting civic activism/competence (involving partnership and exchange of experience with more mature CSOs, possibly outside of the area); b) promoting (and institutionalising, e.g. in the form of advisory councils) policy dialogue with authorities; c) adult education for personal development (with thorough discussion of possible options, unknown innovations and existing experience); and d) income generation/socio-economic adjustment (taking into account labour market trends, in close cooperation with municipalities and business communities). o When designing the project, particular importance should be ascribed to the multiplier effect, i.e. by demonstration of the effectiveness of innovative approaches creating the possibility to replicate innovations and achievements of the project results, particularly in the case of such activities as agro-business support, promoting civil activism, or supporting youth self-organisation. o Multiplier effect can be achieved through at least two types of supported initiatives – a) those that help other initiatives to develop through providing necessary assistance, services or materials; b) through the effect of demonstrating new effective options, technologies or approaches that may be easily replicated by other groups, maybe even without external support. This would mean more focus on innovation, and diversifying the range of options instead of relying on proven but less innovative approaches that more limited demonstration effect. o When selecting project or business proposals/ideas for support, it an alternative could be considered that while the selection criteria remain explicit and transparent, as should be the formal requirements for proposals, effective selection may be based on general considerations (such as general social impact) that may be unfair to demand from the applicants to incorporate into their proposals. o When aimed at improving livelihood, a project by consider attempting to find some compromise or balance between the two objectives – supporting the most needy, and making the support sustainable, and respectively, the issue of vulnerability of applicants may get explicitly including respective selection criteria, on the other – applying a good deal of social work in order to understand how the most vulnerable could be included in this or some other scheme. o It may be prudent reconsider, obviously required by the donor, existing restriction of small business support to agricultural activities, as IDPs in many locations (particularly urban and semi- urban settlements) have very limited access to (fertile) agricultural land which is a necessary requirement for getting a business proposal supported. o For all elements of the project, it seems prudent to supplement opinion surveys of potential beneficiaries by other instruments such as working with experts, and also preliminary work with respondents explaining the broader range of existing options. Thus, it is not just the opinion of the potential beneficiaries that should be taken into account, but it must be informed opinion. o Any project design should deal with the policy dilemma in educating people in order to help them to deal with the challenges of the socio-economic integration in the situation when there are virtually no jobs easily available, and the job market is extremely underdeveloped. One may consider involving also a service that would help with either finding employment or creating opportunities through supporting sort of business incubators for small businesses. o When the project is supporting to develop and incorporate a new CSO/CBO, partnering of less experienced new organizations with more developed CSOs may be considered as an effective approach to support the newly established CSOs. It should be also recognised that NGOs in less developed regions can hardly survive unless they diversify their activities, and are helped with that. o As sensitising the municipalities towards IDP-related problems is important, one might consider an approach based on publicised rating of the municipalities within the country in accordance to their responsiveness to IDP needs by effort of international organisations and local CSOs may bring in competitiveness, motivation, and sometimes even the “name and shame” factor.

The evaluation document is supplemented by annexes, containing the list of persons and groups interviewed, and the main instrument/questionnaire, used in the semi-structured interviews and developed in three versions, according to the type of respondents.