Final Performance Evaluation of the Transition Initiatives for Stabilization Project Idiq Aid-623-I-14-00009 Task Order Aid-623-T0-15-00014

SOMALIA PROGRAM SUPPORT SERVICES FINAL PERFORMANCE EVALUATION OF THE TRANSITION INITIATIVES FOR STABILIZATION PROJECT IDIQ AID-623-I-14-00009 TASK ORDER AID-623-T0-15-00014

June 4, 2016 This publication was produced at the request of the United States Agency for International Development. It was prepared by International Business & Technical Consultants, Inc. under IDIQ AID-623-I-14-00009, Task Order /AID- 623-TO-15-00014 1

SOMALIA PROGRAM SUPPORT SERVICES

FINAL PERFORMANCE EVALUATION OF THE TRANSITION INITIATIVES FOR STABILIZATION PROJECT

IDIQ AID-623-I-14-00009, TASK ORDER /AID-623-TO-15-00014 June 4, 2016

Authors: Roger Pipe, Jama Egal, Jane Meme, Lucas Malla, Zamzam Billow and Aden Abdi. International Business & Technical Consultants, Inc. (IBTCI)

Additional input was provided by Mamuka Shatirishvili, Jeffrey Swedberg, and Gayla Cook of IBTCI, and consultant David Sip.

In the US: IBTCI Home Office 8618 Westwood Center Drive Suite 400 Vienna, VA 22182 USA +1.703.749.0100

In Kenya : Park Office Suites, #9, Ground Floor Parklands, Nairobi +254.786.866.793

IDIQ AID-623-I-14-00009

DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government.

COVER PHOTO: Women celebrate during a Community Dialogue Event in Hodan, Mogadishu. Photo: Alinur Hassan, IOM.

CONTENTS CONTENTS ...... II TABLE OF TABLES ...... III TABLE OF FIGURES ...... III ACRONYMS ...... IV ACKNOWLEDGMENTS ...... V IN MEMORIAM ...... VI EXECUTIVE SUMMARY ...... VII I. INTRODUCTION ...... 1 II. EVALUATION PURPOSE...... 1 III. DESCRIPTION OF THE STABILIZATION PROJECT, INCLUDING GOAL AND TIS OBJECTIVES . 1 IV. KEY EVALUATION QUESTIONS ...... 3 V. METHODOLOGY ...... 4 EVALUATION LIMITATIONS ...... 5 VI. QUESTION 1 FINDINGS: ACHIEVEMENT OF IP OBJECTIVES ...... 5 TIS OBJECTIVE 1: INCREASE CONFIDENCE IN ALL LEVELS OF GOVERNANCE ...... 6 CONCLUSIONS REGARDING OBJECTIVE 1 ...... 10 ACHIEVEMENT OF OBJECTIVE 2: SUPPORT COLLABORATION BETWEEN GOVERNMENT, THE PRIVATE SECTOR, AND CIVIL SOCIETY ...... 11 CONCLUSIONS REGARDING OBJECTIVE 2 ...... 15 ACHIEVEMENT OF OBJECTIVE 3: INCREASED DIALOGUE ON PEACE, RECOVERY, AND DEVELOPMENT IN SOMALIA ...... 15 CONCLUSIONS REGARDING OBJECTIVE 3 ...... 19 INDEPENDENT EFFECT ON THE ACHIEVEMENT OF TIS OBJECTIVES ...... 19 VII. EVALUATION QUESTION 2: TIS CONTRIBUTION TO USAID GOAL & OBJECTIVES ...... 20 TIS CONTRIBUTION TO INCREASING STABILITY ...... 20 WERE ANY TIS APPROACHES ESPECIALLY EFFECTIVE IN CONTRIBUTING TO STABILITY OR TO THE ACHIEVEMENT OF USAID OBJECTIVES? ...... 22 QUESTION 2 CONCLUSIONS ...... 23 VIII. EVALUATION QUESTION 3: INSTITUTIONALIZATION OF THE TIS PROCESS AND COMMUNITY USE OF TIS PHYSICAL STRUCTURES ...... 23 INSTITUTIONALIZATION OF THE TIS PROCESSES ...... 23 CONTINUED USE OF TIS PHYSICAL STRUCTURES ...... 24 QUESTION 3 CONCLUSIONS ...... 26 IX. EVALUATION QUESTION 4: TIS SUCCESS AT ADDRESSING GENDER ISSUES ...... 26 INCREASED INCLUSION OF WOMEN IN COMMUNITY-LEVEL DECISION-MAKING ...... 26 ENABLING WOMEN TO ADVOCATE FOR THEMSELVES AND TAKE ACTION ...... 27 ENSURING BOTH MEN AND WOMEN BENEFIT FROM THE ACTIVITY ...... 29

EVALUATION QUESTION 4 CONCLUSIONS ...... 29 X. EVALUATION QUESTION 5: EFFECTIVENESS OF COMMUNITY PLANNING SESSIONS ...... 29 XI. RECOMMENDATIONS ...... 31 Annex A: Statement of Objectives for TIS Final Performance Evaluation ...... 1 Annex B: Map of Somalia Showing TIS Settlements Surveyed during the Final Evaluation ...... 11 Annex C: Evaluation Design and Methodology ...... 12 Annex D: Evaluation Tools ...... 53 Annex E: Conflict of Interest Disclosures ...... 79 Annex F: Evaluation Team CVs ...... 80 Annex G: Places Visited & List of Organizations and People Interviewed ...... 81 Annex H: Change in Perceptions Regarding Local Government ...... 93 Annex I: Level of Satisfaction with Basic Services ...... 94 Annex J: Correlations between Indicators of Good Governance ...... 95 Annex K: Detailed Stability Index ...... 96 Annex L: Summary of Findings from Focus Groups Regarding TIS Physical Structure ...... 97 Annex M: Findings related to Evaluation Question 5 from Focus Group Discussions and Key Informant Interviews and from the Household Survey ...... 100 Annex N: Summary of Findings from Focus Groups Evaluation Question 4...... 107

TABLE OF TABLES Table 1: Comparison of Average Stability Indexes by Geographic Area ...... 19 Table 2: Results for Comparison of Means (TIS alone and TIS plus other DPs)...... 19 Table 3: Knowledge of CPS and Perception of CPS Effectiveness ...... 31

TABLE OF FIGURES Figure 1: Percentage Change in Citizen Perceptions of Local Government from 2014-2016 viii Figure 2: Stability Index in TIS Settlements: 2014 and 2016 ix Figure 3: TIS Results Framework 2 Figure 4: Number of Grants and Disbursements by IP 2 Figure 5: Confidence in Government 7 Figure 6: Indicators of Collaboration 12 Figure 7: Correlation between Indicators of Good Governance (Country Level) 14 Figure 8: Peace Committees and Community Dialogue 17 Figure 9: Stability Index (Partial Measurement) 21 Figure 10: Grant Bundles with Highest Stability Index 22 Figure 11: Issues Reported on TIS Physical Structures 25 Figure 12: Increased Inclusion of Women in Community-Level Decision-Making and in Representation of Women’s Interests 27 Figure 13: Perception Of FGDs Regarding Women’s Advocacy 29 Figure 14: Perceptions of the Effectiveness of TIS Community Planning 30

iii

ACRONYMS

ADESO African Development Solutions ASWJ Ahlu Sunna Wal Jama’a CDC Community Development Committee CDE Community Dialogue Event CPS Community Planning Session DAI Development Alternatives Incorporated DC District Commissioner DP Development Partners EFS Emerging Federal States FGD Focus Group Discussion FGS Federal Government of Somalia IBTCI International Business & Technical Consultants, Inc. IDP Internally Displaced Person IOM International Organization for Migration IP Implementing Partner KEA Kenya and East Africa KII Key Informant Interview LOP Life of Project MCH Maternal Child Health MIDA IOM's Migration for Development in Africa MOA Ministry of Agriculture POC Project Oversight Committee SPSS Somalia Program Support Services TIS Transition Initiatives for Stabilization USAID United States Agency for International Development USAID/Somalia USAID/Kenya and East Africa (KEA)/Somalia Office

ACKNOWLEDGMENTS IBTCI and the evaluation team would like to start by thanking the respondents - the beneficiaries of Transition Initiatives for Stabilization (TIS) activities – and the many key informants for their time. They answered our lengthy surveys and continue to struggle in a fragile environment with hope and determination. The evaluation team would also like to thank the staff of TIS Implementing Partners; Government Officials in Somaliland, Puntland, the Federal Government of Somalia, and the Emerging Federal States; USAID/Somalia Staff; as well as the many Somali fieldworkers who played an essential role in the data collection process.

IN MEMORIAM This Final Evaluation is dedicated to the memory of two TIS engineers who lost their lives while working on the project: A TIS Engineering Team Leader, was tragically killed in Mogadishu on April 7, 2015 on his way to work. He had worked on the TIS project since May 1, 2013. His responsibilities included overall supervision of technical and engineering components of the program, including development of designs and bill of quantities, tendering of construction grants, regular site visits, and working with government and contractors on infrastructure related projects. A TIS Migration for Development in Africa (MIDA) expert, was killed in Mogadishu on November 17, 2014. He returned to his homeland of Somalia to contribute to the reconstruction of his country.

EXECUTIVE SUMMARY INTRODUCTION The Transition Initiatives for Stabilization (TIS) project was a five-year, US$102.8 million project of the United States Agency for International Development (USAID). The goal of TIS was to increase Somalia's stability through targeted interventions that foster good governance and economic recovery, and reduce the appeal of extremism in the communities in which the activities were undertaken. TIS provided grants to finance the implementation of projects in various sectors at the settlement level throughout Somalia. The project’s implementing partners (IP) were Development Alternatives Incorporated (DAI) and the International Organization for Migration (IOM). This final TIS performance evaluation of TIS aims to assess the extent to which the project contributed to achieving USAID’s goal and TIS objectives. It aims to determine what worked well and what could be improved upon, and to identify lessons about conducting stabilization activities in Somalia in order to improve future programming. The evaluation was designed to provide comprehensive answers to five key questions: 1. To what extent have TIS/DAI and TIS/IOM achieved their objectives? Did the type of approach used have an independent effect on the achievement of TIS objectives? 2. To what extent has the project contributed to the goal and/or to the achievement of USAID objectives? Were any TIS approaches especially effective in contributing to stability or to the achievement of USAID objectives? 3. To what extent did the process modeled by TIS become institutionalized, and did physical structures constructed by TIS continue to be used by communities? 4. How successful was the project in addressing gender equity and empowerment issues, such as increasing the inclusion of women in community-level decision-making, promoting representation of women’s issues, enabling women to advocate for themselves and take action, and ensuring both men and women benefit from the activity? 5. To what extent were Community Planning Sessions (CPS) effective in producing decisions that were inclusive, consensus-based and reflective of community priorities?

Data for this evaluation were collected through a 2016 survey of individuals at the household level, as well as through focus groups and key informant interviews. In addition, the findings from a USAID 2014 Stabilization Survey were used as a basis of comparison to determine the change over time in key project indicators.

FINDINGS

 TIS’s innovative essence was the consultative and participatory process that brought communities (civil society and business people), local administration and central government together to discuss and identify the community’s priority needs.  TIS proved to be a strong catalyst in encouraging greater engagement between communities and government. This is corroborated by the findings from the household level survey, the Focus Group Discussions (FGDs) and Key Informant Interviews (KIIs).  In TIS settlements people are now more involved than before in decisions that improve their lives. Local administrations listen more to people’s views and are more responsive.  TIS contributed significantly toward increasing collaboration between local government and stakeholders.  However, this increased collaboration with local administration has not yet generated increased confidence in all regions.

vii

 The impact of TIS has been at the district level where it helped increase confidence in the ability of local administration to improve lives, particularly in Puntland and Mogadishu.  TIS contributed to an improvement in the local government’s level of professionalism and satisfaction with basic services. This increased people’s confidence in local government in TIS settlements.

Figure 1: Percentage Change in Citizen Perceptions of Local Government from 2014-2016

Level of Confidence in collabora on Sa sfac on ability of with local gov’t with Basic Professionalism Absence of Absence of local gov’t to Services of local gov’t Corrup on Nepo sm improve lives −

Somaliland Decrease Increase

Puntland Less than 5%

5% - 9% Emerging Federal States (Excluding Mogadishu) 10% or more Mogadishu

Three aspects of collaboration were analyzed at the local government level:

 The extent to which people living in the settlement felt they had a say in decisions that improve their lives;  The extent to which local government listens to people’s views;  The extent to which local government is responsive to people’s views. Between 2014 and 2016 there has been a significant increase in all four geographic areas in the percentage of respondents who feel that

 They now have a greater say in decision-making;  Local government listens more to people’s views; and  Local government is now more responsive to people’s views. In summary, in Somaliland, Puntland and Mogadishu, respondents believe there is now greater collaboration with local government in terms of people’s participation in decisions, the extent to which local administration listens to them, and the responsiveness of local government. In Somaliland, the percentage of respondents who believe there is now greater collaboration has increased. However, there has been a decline in the percentage of respondents who strongly agree that this is the case. TIS facilitated dialogue on peace and recovery and was a strong catalyst in encouraging greater engagement between communities and government. TIS-supported community infrastructure has been a key enabler of peace and development. Peace committees and peace meetings that TIS supported and facilitated served as forums for discussion of issues and for preventing and/or resolving conflicts within communities. TIS- supported dialogue activities brought diverse sectors of the community together and contributed to social integration and economic recovery.

viii

Although local Peace Committees existed in settlements before TIS, many people living in the communities where TIS worked tended to associate the Peace Committees with TIS. TIS took advantage of their existence and built upon their organizational structure. Members of Peace Committees were often also community leaders. When TIS created Community Development Committees (CDCs) as part of its consultative, participatory process, many of the Peace Committee members also participated in the CDCs. TIS’s support of Dispute Resolution Centers is also credited with having contributed to peace. Projects funded by TIS were started in a consultative manner and this may have encouraged the government to embrace more consultative approaches to dealing with matters of peace, security and development. To analyze TIS’s contribution to increasing stability in Somalia, the evaluation team designed a Stability Index based on seven characteristics of stability defined by USAID for the purpose of this evaluation.1 Using a comparative approach to compare this 2016 household survey data with existing USAID 2014 data, it was found that communities’ scores on the Stability Index rose significantly in all four geographic areas.

Figure 2: Stability Index in TIS Settlements: 2014 and 2016

100%

80%

60% 66% 67% 69% 65% 2014 51% 52% 40% 48% 46% 2016

20%

0% Somaliland Puntland Emerging Mogadishu Federal States

Institutionalization of the TIS Process Based on key informant interviews with Somali government officials, the evaluation team found that TIS processes have been or will be adopted by important entities and actors. In addition, local communities like the TIS approach and have started challenging other agencies (NGOs and UN) to work like TIS with grassroots consultation. Community Use of TIS Physical Structures Most of the physical structures built by TIS continue to be put to good use, are used as intended, and are equitably accessible. TIS infrastructures are perceived to be public facilities under the custody of government, with the government having the primary responsibility for protecting and maintaining them. In general, FGD participants spoke positively of the contribution TIS infrastructure has made to the improvement of living conditions in their respective settlements. Notwithstanding the overall positive view of TIS physical structures, the following issues were reported and/or observed:

 Ownership: some TIS structures were reported as privately owned, rather than government owned;  Incompleteness: at the time of the FGDs, a number of TIS structures had not yet been completed (It is noted that, at the time of the FGDs, DAI work was still ongoing);

1 See Annex A, “Statement of Objectives for the TIS Evaluation”, page 2.

 Location/inaccessibility: there were cases where the TIS structure was built in a location that women did not consider to be safe;  Size: some FGD participants considered the TIS structures to be too small;  Quality: a number of quality-related issues were reported.

TIS Success at Addressing Gender Issues

 Through the process of gender inclusive, community-based consultation, TIS has contributed to increasing the role of women in community-level decision-making.  Female focus group participants say they do not feel confined to their homes anymore and are now able to go out and work. Participants said they felt empowered as a result of participating in the TIS-supported CPS and skills trainings.  Women said they have created a culture of self-mobilization. After receiving training from TIS, they started to take on more initiatives. Thanks to TIS trainings on governance, women are playing a more active role in local politics.  Women FGD participants said that the increased participation of women in economic activities has raised their status, as they are now better able to access and control economic resources.  TIS economic/livelihood training and grants, markets and street lighting increased the economic activity of women, earning them more respect as well as income.  According to FGD participants and Key Informants, there has been increased respect and support for women’s interests, both at the community and government levels.  TIS directly contributed to a change in the way that society perceives women, thanks to the inclusive policy that encouraged women’s participation in CPS and Peace Committees.

Effectiveness of Community Planning Sessions

 According to FGD participants, most CPS were both representative and effective in producing decisions that were inclusive and reflective of community priorities.  The projects (grants) prioritized by the communities were the ones implemented under TIS that contributed to support for and ownership of those projects by local government and the community.  Multi-stakeholder involvement in CPS was hailed as one of the strengths of the TIS approach.  In some areas in Puntland and Somaliland a majority of youth FGD participants felt they were not represented. A minority of participants also said that they considered the choice of representatives in CPS was not always inclusive.  Most focus group participants felt that TIS projects reflected community priorities. On average, 87% of household survey respondents also said that TIS community planning sessions were effective in selecting projects that reflect priorities in the settlement.

RECOMMENDATIONS The recommendations are intended to be action-oriented, practical and specific, with defined responsibility for the action.

1. Include a strategic communication component and strategic communication activities in each grant: In a stabilization project that, through its projects and processes aims to strengthen confidence in the local government’s capacity to improve people’s lives, a comprehensive strategic communication plan for the project as a whole as well as for each grant is essential. This plan should define how to use publicity to get more representative participation on the CDCs. It should also communicate to the general population the message that the local government has brought funds to the community,

has successfully implemented projects, and has done so through a collaborative process of community consultation in which activities were prioritized. Projects similar to TIS should include a strategic communication plan within each grant, to communicate to the community stakeholders before, during, and on completion of activities. This would also build government capacity for constituent communication after the project is complete through the training of government officials.

2. Project Oversight Committees: Project Oversight Committees (POCs) were not included in DAI and IOM activities until 2013, when IOM began using them. POCs are important, especially for infrastructure projects. A committee, whose members live within walking distance of the infrastructure, is needed to monitor the project’s implementation. The committee should include at least one member of civil society, at least one member of TIS, and at least one member of government. At least three members of the committee should be able to walk to the site on a daily basis, eliminating the need for transportation and per diem costs. With proper training, local committee members would provide the best “eyes and ears” possible because they would have been involved in the planning session, and would know the required design of the project. POCs can also provide the latest security information. IPs can call and ask a POC member to check quantities, take photographs and verify if the contractor is on site. To ensure proper conditions for the POC’s success, it is important to provide an orientation on the POC’s role for all stakeholders, including the contractor, government, and community. In an ideal scenario, the POC’s government representative would report to the local administration on a daily basis, the civil society representative would report to the community, and the TIS representative would report to TIS.

3. Better supervision and engineering oversight is required during implementation of infrastructure projects.  The POCs’ role at the community level should be strengthened and institutionalized. POCs should be provided with adequate training to allow adequate monitoring of projects. The contractor and the community must properly understand the POC’s role.  International engineers should be brought on board to provide management/quality control oversight of infrastructure activities. This would help ensure that international engineering standards are adhered to and help streamline the design approval and tendering process. At least one international expert should be hired to play an oversight role with respect to local engineers. International oversight should, however, respect the principles of local ownership and the use of Somali engineers, Somali engineering companies, and Somali/Kenyan engineers for project implementation.

4. A sustainability plan A sustainability plan for each infrastructure project should be formally agreed to by the grantee and socialized within the community. The POCs’ role should be expanded to include monitoring the maintenance of infrastructure and the correct utilization of infrastructure after implementation.

5. Budgeting: Individual settlements should not be assigned fixed, up-front amount funding. Fixed budgets limit flexibility and encourage micro-management (i.e., scrutinizing every small expense) by the POCs. Under TIS, additional community meetings were required to decide how to use any leftover funds, causing delays that thwarted the intention of “quick impact” grants. Budgeting should not be a feature of the community prioritization process. Instead, following the community prioritization meetings, budgets should be prepared and projects should be prioritized based on available funding. The community should get the deliverable (be it infrastructure, goods or services), not the money. The dollar amount should be shared at the community-contracting phase to enable

the POCs to assess value for money of tenders. However, it is not necessary to start a community planning session with the commitment of a certain dollar amount for the whole community.2

6. Develop a Master Development Plan at the Settlement Level. Instead of doing a one-off CPS, a more efficient and sustainable approach would be to assist each beneficiary settlement to develop a multi-year Master Development Plan, using the TIS process and creating a steering committee comprising local government, TIS, other development partners, civil society and the private sector.3

7. Diversify beneficiaries: Under TIS, the Mayor or District Commissioner (DC) was named as the beneficiary and signatory in almost all settlements, placing a great deal of authority and power in one individual’s hands. There were cases in which the Mayor or DC became the gatekeeper, creating an impasse that could have been avoided had there been more than one grantee. For instance, the beneficiary of MCH Centre could be the director of the center, and the school principal could be the beneficiary of a school infrastructure project. This diversification of beneficiaries could be done in a way that preserves the local administration’s role in bringing the project to the community. In this way each project contributes to improving collaboration with and inspires confidence in local government.4

2 TIS+ has already moved away from this approach and is just giving indicative examples of types of projects other communities have prioritized. 3 This idea has been incorporated into TIS+. 4 TIS+ is also planning to adopt this approach to provide greater accountability. The Mayor/DC will still be aware, and in some cases will be a grantee, but not in all cases.

xii

I. INTRODUCTION This document presents the findings and recommendations from the final performance evaluation of the Transition Initiatives for Stabilization (TIS) project, a five-year initiative implemented from 2011 to 2016 in Somalia by two USAID Implementing Partners (IPs): Development Alternatives Incorporated (DAI) and the International Organization for Migration (IOM).

II. EVALUATION PURPOSE The main purpose of this final evaluation is to assess the extent to which USAIDs stabilization activities in Somalia have met the project goal to increase stability in Somalia through targeted interventions that foster good governance and economic recovery and reduce the appeal of extremism in the communities in which the activities were undertaken. Another purpose of this evaluation is to learn lessons about conducting stabilization activities in Somalia to improve future programming. The evaluation is designed to evaluate the performance of the Stabilization Project collectively rather than the activities of TIS-IOM and TIS-DAI individually (see Statement of Objectives in Annex A). The audience for this study is the USAID/KEA/Somalia Office, DAI, IOM, other USAID offices, such as those managing funds for activities that address complex crises and activities that aim to counter violent extremism, and other donors supporting Somalia. The evaluation will also be used by AECOM, the partner implementing the Transition Initiatives for Stabilization Plus activity.

III. DESCRIPTION OF THE STABILIZATION PROJECT, INCLUDING GOAL AND TIS OBJECTIVES USAID’s TIS project was a five year, US$ 102.8 million project. Its goal was to increase stability in Somalia through targeted interventions that foster good governance and economic recovery and reduce the appeal of extremism in the communities in which the activities were undertaken. TIS provided grants to finance the implementation of projects throughout Somalia at the settlement level (see map in Annex B), TIS sought to increase confidence in all levels of government through targeted, strategic interventions that improve service delivery and government responsiveness. TIS aspired to encourage Somali government institutions, the private sector, and civil society to collaborate in the identification and prioritization, design, monitoring and implementation of projects with quick and lasting impacts on the lives of Somalis. Project Goal As shown in Figure 3, TIS activities were designed to contribute to the three TIS objectives, which in turn, contribute toward both objectives and the goal of the USAID/KEA/Somalia Office. In terms of number of grants and amount disbursed, Figure 4 shows the distribution of grants across the three TIS objectives and per IP.

Figure 3: TIS Results Framework

USAID/ Somalia Goal: Somalia's stability increased through targeted interventions that foster good governance and economic recovery and reduce the appeal of extremism

USAID Objective 1: USAID Objective 2: Strengthen local and national capacity Improved social services delivery and to promote good governance economic growth

TIS Objective 1: TIS Objective 3: Increase conﬁdence in all TIS Objective 2: levels of government through Support collaboration between Increase dialogue on peace, the delivery of targeted, government, private sector and recovery, and development in strategic interventions that civil society. Somalia. improve service delivery

Figure 4: Number of Grants and Disbursements by IP

IOM – Number of Grants IOM – Amount Disbursed (US$$Mill iMillions)ons) 5% 9%

22% 15 7% 2.5 2.0 71 84% 324 73% US$27 Grants Million 238 22.5

DAI – Amount Disbursed DAI – Number of Grants DAI - Amount Disbursed (US$Millions) (US $Millions) 3% 14% $0.8 72 7% 35 20% $5.2 504 Grants US$26 79% Million 77% 397 $20

Source: TIS Grant Database as of February 1, 2016. Note: the numbers presented in this graph will not represent the final figures at the end of the project.

THE TIS PROCESS TIS included a participatory process that was innovative for Somalia, through which community stakeholders from civil society, the private sector and local administration came together to collaboratively identify and prioritize projects that would provide the greatest benefit to their communities. Community stakeholders were brought together for three-day work sessions during which the stabilization objectives of TIS were explained, followed by the collaborative identification and prioritization of interventions that would, in the view of participants, provide the greatest benefit to their communities within the context of stabilization. Grants were prioritized based on an established budget allocation, typically between US$150,000 and US$200,000. During the early days of TIS, in 2011 and 2012, consultative meetings were initially held outside of Somalia for security reasons and to enable USAID to participate. Security conditions changed for the positive by the beginning of 2013, at which time this consultative, participatory process transitioned to Somalia. Both IPs refined the consultative process during TIS’s implementation. Over time, it was noted that, although women constitute at least 30% of the community committees, they did not always appear comfortable expressing their own opinions regarding community priorities. To address this issue, in mid- 2015, separate break-out sessions for women were conducted in some settlements. Transparent procurement was another TIS characteristic. TIS helped communities to transparently select local contractors to implement each project through a competitive bidding process. These two processes -- community-based consultation and transparent procurement processes -- were intended to ensure a higher level of Somali ownership, and improve the level of collaboration between citizens and their government. TIS activities aimed to increase public trust, confidence in government, and social cohesion through a variety of interventions. For TIS, the process of determining priorities and jointly moving forward was as important as the final product. In addition to working at the community level, TIS partners also worked at the district, regional, and federal levels with a variety of actors, including the government, civil society and private sector. TIS grants were implemented across Somalia in 17 of Somalia’s 18 regions, in both the relatively stable north and the newly liberated south.

IV. KEY EVALUATION QUESTIONS The evaluation sought to answer the following five key questions: Question 1: To what extent have TIS/DAI and TIS/IOM achieved their objectives? Did the approach used have an independent effect on the achievement of TIS objectives? Question 2: To what extent has the project contributed to the goal and/or to the achievement of USAID objectives? Were any TIS approaches especially effective in contributing to stability or to the achievement of USAID objectives? Question 3: To what extent did the process modeled by TIS become institutionalized, and did physical structures constructed by TIS continue to be used by communities? Question 4: How successful was the project in addressing gender equity and empowerment issues, such as increasing inclusion of women in community-level decision making, promoting representation of women’s issues, enabling women to advocate for themselves and take action, and ensuring both men and women benefit from the activity? Question 5: To what extent were community-planning sessions effective in producing decisions that were

inclusive, consensus-based and reflective of community priorities?

V. METHODOLOGY The evaluation integrated different methods and tools for data collection and analysis into an overall strategy for answering the evaluation questions. The evaluation used a mixed-method approach to data collection and analysis in a two-phase implementation plan. Phase 1 involved a thorough desk review of activity data and documents generated by the TIS project, particularly records in a grants database maintained by both implementing partners. Phase 1 also entailed in-depth qualitative interviews with USAID Officials, and TIS-DAI and TIS-IOM senior management and field implementation staff. An output of Phase 1 was the development of a categorization framework for types of TIS approaches. This categorization, and the other knowledge gained from the desk review and from USAID and IP interviews, informed the design of primary data collection in the field, including the quantitative opinion survey and qualitative in-depth key informant interviews (KII) and focus group discussions (FGDs) with members of the communities in which TIS grants were implemented. Each evaluation question was answered using the combination of quantitative and qualitative methods for collecting a robust set of data from different sources in the field. Data were triangulated during the analysis phase to minimize respondent and analyst bias that might affect answers to the evaluation questions. Annex C contains a data collection matrix and detailed description of methodology that links each of the five key questions to a set of more detailed sub-questions, the type of evidence used to answer the questions, the data sources, data collection tools (see Annex D), data collection method, sampling approach, and data analysis method used to answer each. The Data Collection Matrix thus provides a concise summary of the evaluation methodology. The TIS evaluation team mobilized on October 11, 2015. Data collection took place in Somalia from December 28, 2016 to March 20, 2016. A mixed methods approach included quantitative household surveys (1,719 respondents), focus group discussions (87), and in-depth key informant interviews (221). There were a total of 849 FGD participants, including 306 women, 252 men and 291 youths. The achieved sample size for the household survey exceeded the original planned sample of 1,623 by 96 respondents. Excess interviews were conducted in order to form reserves for potential replacements of interviews in which errors were detected during the analysis stage (the lists of respondents are presented in Annex G). Data collected from the 2016 household survey were compared with data contained in a 2014 Stabilization Report prepared for USAID by IBTCI. That report presented the findings of a 2013/2014 household nationwide survey of over 3,100 respondents from TIS intervention settlements and non-TIS settlements. For this final evaluation of TIS, select questions from the 2014 survey were asked again of respondents in many of the same communities. The comparison of findings from the two years was used to infer the contribution of TIS interventions to changes in resident perceptions on topics relevant to project objectives. The quantitative survey has a 95% confidence level with a margin of error of plus or minus 5% at the national level.5 It should be noted that FGD findings are not representative of the general population. FGD participants were drawn from the select settlements but selection was guided by the various grants in each settlement and targeted possible direct beneficiaries. For example, if a market was built, street lights installed, a training conducted, livelihood grants provided, sports equipment supplied or a road renovated, then focus groups were formulated comprising a mix of traders from the market, people living or running businesses along the road, youth using the sports equipment or stadium, or beneficiaries of the livelihood grants. FGD

5 In order to ensure comparability with the data obtained from the 2016 survey, the only data used from the 2014 Stabilization Survey were those corresponding to the same cohort of settlements where interviews were conducted for the 2016 survey.

participants may not have participated in the project selection process but were aware of TIS by virtue of being beneficiaries of the sub-activities. On the other hand, household survey participants were only required to confirm that they were aware of TIS sub-activities, or 'knew' of TIS, in order to be eligible to participate in the survey; but they were not necessarily direct beneficiaries. Evaluation Limitations Two types of limitations affected the quantitative and qualitative data collection: Quantitative data collection: Security-related issues prevented access to nine settlements (see Annex C for details). In order to compensate for the inability to collect data in those nine settlements, the households that had been targeted in those settlements were distributed equally to the remaining settlements, as explained in Annex C. While ensuring that the planned sample size was respected, this method had the potential to introduce a Type I error, by failing to collect data from inaccessible communities. The likelihood of a Type 1 error is considered small, however, since most of the inaccessible communities were deemed so due to security issues affecting access routes, rather than the conditions in the communities themselves. The omitted communities were not necessarily less stable than the surveyed communities. Qualitative data collection: In comparison to the planned target of 90 Focus Group Discussions (FGDs) the achieved number was 87 FGDs, due to the following circumstances: in one settlement (Shangani in Mogadishu), the men’s focus group refused to be audio recorded. In another settlement (Mataban in South Central), the TIS evaluator was met with hostility from the community members due to the fact that only one grant (to conduct a Community Planning Session) of five approved grants had been implemented. The other grants could not be implemented due to an impasse that arose between critical actors. In Mataban the TIS Evaluator succeeded nonetheless in conducting one mixed focus group with 12 of the participants of the Community Planning Session.

VI. QUESTION 1 FINDINGS: ACHIEVEMENT OF IP OBJECTIVES This section presents findings on the extent to which TIS/DAI and TIS/IOM achieved their objectives:

 Objective 1: Increase confidence in all levels of governance through the delivery of targeted, strategic interventions that improve service delivery;  Objective 2: Support collaboration between government, the private sector, and civil society;  Objective 3: Increase dialogue on peace, recovery and development in Somalia. In addressing the question of the extent to which TIS/DAI and TIS/IOM achieved their objectives, the following sub-questions are also addressed:

 To what extent did TIS serve as a catalyst for encouraging greater engagement between communities and government, while strengthening government capacity with the aim of improving government response to community needs?  To what extent did TIS increase the involvement of community members in determining priority needs at the community/local level and create opportunities for improved government-to- community engagement and collaboration?  To what extent did TIS contribute to strengthening government capacity and processes to carry out critical functions?  How well did the TIS consultative process at the community level work? This section also presents the findings regarding the questions of whether the type of approach had an independent effect on the achievement of TIS objectives.

TIS Objective 1: Increase Confidence in All Levels of Governance Two aspects of confidence in local administration and central government were analyzed:

 Confidence in the ability to improve the lives of all people in the settlement.  Confidence in the ability to resolve conflict and violent disputes at the community level. The level of confidence in each year of analysis (2014 and 2016) was calculated as the sum of the percentage of respondents who indicated they were very confident or somewhat confident. The change in the level of confidence was calculated as the level of confidence in 2016 minus the level of confidence in 2014. The analysis also scrutinized the change in the percentage of respondents who were very confident in each year. Figures 5a to 5d provide a summary of the findings related to Objective 1 for each geographic area. Confidence in the ability of government to improve the lives of people in their settlements In 2016, general confidence in the central government’s ability to improve lives is between 86% and 90% in Somaliland, Puntland, and Mogadishu, compared to 73% in the Emerging Federal States (EFS). Between 2014 and 2016, confidence increased by 3 percentage points in Puntland and 6 percentage points in Mogadishu, but decreased by 1 percentage point in Somaliland and by 20 percentage points in the EFS. The percentage of respondents who are “very confident” in the central government’s ability to improve lives rose by between 1 and 4 percentage points in Somaliland, Puntland and Mogadishu, but fell by 31 percentage points in the EFS. In 2016, confidence in the local government’s ability to improve lives is over 80% in Somaliland, Puntland and Mogadishu, and 79% in the EFS. Between 2014 and 2016, confidence in the local government’s ability to improve lives rose by 10 percentage points in Puntland and by 21 percentage points in Mogadishu, but fell by 1 percentage points in Somaliland and by 3 percentage points in the EFS. The percentage of respondents who are “very confident” rose by 1 percentage point in Somaliland, by 7 percentage points in Puntland and by 18 percentage points in Mogadishu, but fell by 2 percentage points in the EFS.

Figure 5: Confidence in Government

Figure 5a: Somaliland 100% 91% 92% 90% 89% 90% 86% 84% 83% 80%

60%

40%

20% Percentage point change -1% -1% +1% -6% between 2014 and 2016 0%

Central Local Central Local

Government Government Government Government Figure 5b: Puntland Ability to Im prove Lives Ability to Resolve Community-based 100% 94% 93% 93% 93% Conflict and Disputes 87% 90% 90% 84% 80%

60%

40% Figure 5c: Emerging Federal States

20% 100% +3% +10% 0% +3% 100% 95% 93% 0%

80% Central Local Central Local 82% 79% 79% 78% Government Government Government Government 73%

60% Ability to Im prove Lives Ability to Resolve Community-based

40% Conflict and Disputes

20% -20% -3% -21% -17% 0% Figure 5d: Mogadishu

Central Local Central Local 100% Government Government Government Government

90% Ability to Improve Lives Ability to Resolve 86% 87% 80% 82% 86% Community-based 80% 74% Conflict and Disputes 60% 61%

40%

20% +6% +21% +4% +13% 0% Central Local Central Local

Government Government Government Government

Ability to Improve Lives Ability to Resolve Community-based Conflict and Disputes

Confidence in the ability of government to resolve conflict and violent disputes In 2016, confidence in the central government’s ability to resolve conflict and violent disputes at the community level is over 80% in Somaliland, Puntland, and Mogadishu and just under that, at 79%, in the EFS. Between 2014 and 2016, confidence in the central government’s ability increased by 1 percentage point in Somaliland and by 4 percentage points in Mogadishu. It remained the same in Puntland and fell by 21 percentage points in the EFS. The percentage of respondents who were “very confident” in the central government’s ability to resolve conflict and violent disputes at the community level increased by 4 percentage points in Puntland and by 3 percentage points in Mogadishu. There was a decline of 1 percentage point in Somaliland and a decline of 24 percentage points in the EFS. In 2016, confidence in the local government’s ability to resolve conflict and violent disputes at the community level is over 80% in Somaliland, Puntland and Mogadishu, and 78% in the EFS. Between 2014 and 2016, confidence in the local government’s ability to resolve conflict and violent disputes increased by 3 percentage points in Puntland and by 13 percentage points in Mogadishu. In comparison, confidence fell by 6 percentage points in Somaliland and by 17 percentage points in the ESF. The percentage of respondents who are “very confident” in the local government’s ability increased by 18 percentage points in Puntland and by 17 percentage points in Mogadishu, but declined by 12 percentage points in Somaliland and by 7 percentage points in the EFS. Contextual Analysis With regard to Somaliland, the following factors may have contributed to the decline in confidence in local government:

 TIS interventions took place primarily between 2012 and 2013, resulting in the increased confidence noted during the 2014 survey. Subsequently, however, there were not a similar number of grants being implemented, which may have contributed to a decline in confidence in 2016. According to key government informants, TIS increased expectations that local administrations did not have the resources to fulfill.

 In general, expectation, accountability and demand are higher in more stable environments than unstable places, and even more in areas with elected local councils, such as in Somaliland, compared with other parts of the country.

 The level of confidence dropped steeply in contested parts of the Sool and Sanag regions where there are two parallel local administrations – reporting to either the governments of Somaliland or Puntland – in Badhan, Dhahar and Bo’ame. The question of applicable local administration is complex in areas with a presence of dual administrations. With regard to Puntland, major factors that may have affected the positive change in confidence in local government include:

 Service delivery through TIS and local governments, especially in coastal areas that had been isolated and neglected since the start of the civil war, has increased the level of confidence in the ability of local administrations to resolve conflicts.

 Puntland government -- the President in particular -- has introduced administrative and good governance reforms, including an anti-corruption campaign and the institutionalization of procurement and recruitment processes. TIS processes may have contributed to the adoption of these practices.

 One major active and recurrent conflict was resolved in Puntland (in Agaarre area along the Ethiopian border).

 The TIS grants that took place in Puntland were highly visible and stood out because they took place in remote and former piracy-dominated coastal areas such as Eyl and Banderbayla. With regard to the EFS, major factors that may have affected the drop in confidence in the central government include:

 There have been major changes in the EFS in both 2014 and 2016. As the Federal Government of Somalia (FGS) (parliament, president and cabinet) was still relatively new in 2013, people in EFS (what was then referred to as South Central) had high expectations in the newly formed government, the installation of which marked the end of the Transitional Federal Government (TFG). The new government was recognized as permanent.

 In contrast to Puntland and Somaliland, EFS settlements referred to the FGS as the Central Government while those in Somaliland and Puntland referred to their governments.

 The biggest drops in confidence were likely caused by key changes and events in EFS areas in 2016. For example, one of the biggest drops occurred in the Galgaduud region, dominated by Ahlu Sunna Wal Jama’a (ASWJ) which fiercely opposed the FGS in 2015 over the formation of Galmudug administration. The accessible major towns of Galgaduud (Dhusamareb, Abudwak, Balanballe, Herale and Guriel) were ruled by ASWJ at the time of the survey (only Adado was under the new Galmudug administration).

 EFS settlements referred to the FGS as the central government – an entity that is distant from their daily lives. Some EFS settlements were living under newly formed federal member states that had not been in existence long enough to gain people’s confidence. Moreover, some of the administration formation processes also left unresolved grievances within the communities. With regard to Mogadishu, major factors that may have affected the increase in confidence in the Banadir local government include:

 TIS had a significant and highly visible impact in Mogadishu because it built or refurbished district administration offices and furnished them. It also renovated police stations and public markets. Of the 16 Mogadishu districts, only six had functioning administrations prior to TIS. Most of the other districts were not accessible or were under Al Shabaab control in 2011 when TIS started. TIS directly supported the establishment of administrations in those districts during a critical time.

 Most of the city had been deserted and was strewn with rubble. Roads had been blocked for almost two decades. As one of the only stabilization actors present in 2011 and 2012, TIS supported local administrations through people-to-people dialogue and trauma-healing sessions, coupled with basic service delivery grants that increased collaboration between people and the government.

 TIS’s community-based, bottom-up approach (as opposed to the top-down, centralist approach that was the common model in Somalia) was successful in bringing services closer to the people, which contributed to increasing confidence in local administrations. In the FGDs across the four geographical areas, participants had a relatively positive perception of government. Participants were most appreciative of the prevailing peace and security situation, for which local governments were credited. Participants considered that local administrations were working hard to provide basic services and collaborate with the public. The FGDs indicated this was a sign of a functional government that is working to address the needs of citizens. Participants in FGDs saw services such as healthcare, education, security and street lighting, as well as functional government offices, as factors increasing the public’s level of confidence in local government. Freedom of movement, freedom of speech, the ability to own and run businesses without interference, and increased participation of the public in

9 development initiatives were said to be factors boosting confidence of citizens in local government. In Mogadishu and the EFS, FGD participants expressed appreciation for increased law and order and credited local government for improved levels of peace and security. FGD participants expressed the view that improved basic services were made possible by improved peace and security, which in turn, was a precursor to economic development. The TIS-supported renovation/construction and refurbishing of government offices, with public service personnel installed, was also said to be a sign of functional government in Mogadishu and the EFS. Despite their appreciation for the improvements introduced by local governments in Somaliland, Puntland and Mogadishu, FGD participants felt the government could do even more to improve basic services. FGD participants often cited a lack of political will from the government rather than a lack of resources. FGD participants felt the government was capable of improving basic service delivery, including health, education, water and sanitation systems, as well as improving the level of public participation in governance. In the EFS, FGD participants indicated the government should do more to provide such basic services as education, health, water and basic infrastructure with the help of development partners, now that most areas had been liberated. However, some FGD participants doubted the local governments’ capacity to carry out such projects. Conclusions Regarding Objective 1 In terms of increasing confidence in government, TIS’s impact has been greatest at the district level, where it contributed to increasing confidence in the local administration’s ability to improve lives, particularly in Puntland and Mogadishu, where it also resolved local conflicts. This resulted from an improvement in the local administration’s professionalism, satisfaction with basic services, and from increased collaboration. In Somaliland and the EFS there was a perceived increase in collaboration with local government and with the level of professionalism of local government, as well as increased satisfaction with basic services. However, confidence in the local government’s ability to improve lives has not changed significantly. Possible explanations for the lack of change in confidence may be:

 Through its community-based consultative process and transparent procurement process, TIS contributed to increasing transparency in the identification, prioritization and implementation of grants. This may have served to sensitize people to flaws with existing government processes, which they would not otherwise have seen as clearly.  TIS increased expectations for additional grants that local governments did not have the resources to fulfill.  The lack of a Strategic Communication Plan limited local governments’ ability to communicate to the settlement population the way in which the grants were selected and implemented and the role played by local government in the process. There is no evidence that TIS contributed to increasing confidence in the central level of government. The confidence that people have in central government is largely a factor of economic and political factors that are external to TIS. Regarding the sub-question, “To what extent did TIS contribute to strengthening government capacity and processes to carry out critical functions?” The increase in the local government’s perceived level of professionalism, and in the level of satisfaction with basic services, demonstrates that TIS contributed to strengthening the local government’s capacity and processes to carry out critical functions.

Achievement of Objective 2: Support Collaboration between Government, the Private Sector, and Civil Society For the purpose of this evaluation, three aspects of collaboration were analyzed at the local administration level:

 The extent to which people living in the settlement felt they had a say in decisions that improve their lives.  The extent to which local administration listens to people’s views.  The extent to which local administration is responsive to people’s views. The perception of respondents regarding the level of collaboration was calculated for each year of analysis (2014 and 2016) as the percentage of respondents who indicated they strongly agreed or somewhat agreed with the following statements:

 “People living in this settlement have a say in the decisions that improve our lives (compared to two years ago).”  “The local administration listens to people’s views (compared to two years ago).”  “The local administration is responsive to the views of the people (compared to two years ago).” The change in perception was calculated as the percentage who strongly agreed or somewhat agreed with those statements in 2016, minus the percentage of those who strongly agreed or somewhat agreed in 2014. The analysis also scrutinized the change in the percentage of respondents who strongly agreed in each year. The findings are presented in Figures 6a through 6d. Extent to which people living in the settlement felt they had a say in the decisions that improve their lives Between 2014 and 2016 there has been an increase in all four geographic areas in the percentage of respondents who feel that they now have a greater say in decision-making: by 6 percentage points in Somaliland, 13 percentage points in Puntland, 10 percentage points in the EFS and 21 percentage points in Mogadishu. This positive change has been accompanied by an increase in all four geographic areas in the percentage of those who “strongly agree”: by 11 percentage points in Somaliland, 22 percentage points in Puntland, 7 percentage points in the EFS, and 15 percentage points in Mogadishu. Extent to which local administration listens to the views of the people Between 2014 and 2016 there has also been an increase in all four geographic areas in the percentage of respondents who feel that their local administration listens more to people’s views: 9 percentage points in Somaliland, 13 percentage points in Puntland, 8 percentage points in the EFS and 12 percentage points in Mogadishu. This positive change has been accompanied by an increase in three of the four geographic areas in the percentage of those who “strongly agree”: 6 percentage points in Somaliland, 17 percentage points in Puntland, and 5 percentage points in Mogadishu. Only in the EFS was there a decline, by 7 percentage points, of those who strongly agreed that local administration now listens more. The extent to which local administration is responsive to the views of the people Between 2014 and 2016, there was an increase in all four geographic areas in the percentage of respondents who feel that local administration is now more responsive to people’s views by 17 percentage points in Somaliland, 16 percentage points in Puntland, 11 percentage points in the EFS and 19 percentage points in Mogadishu. This positive change has been accompanied by an increase in three of the four geographic areas in the percentage of those who “strongly agree”: by 7 percentage points in Somaliland, 16 percentage points in Puntland, and 9 percentage points in Mogadishu. Only in the EFS was there a decline, by 3 percentage points of those who strongly agreed that local administration now listens more.

Figure 6: Indicators of Collaboration

Figure 6a: Somaliland

100% 95% 89% 84% 84% 80% 75% 67% 60%

40%

20% +6% +9% +17% Percentage point change 0% between 2014 and 2016 People have greater Administra on Administra on more Figure 6b: Puntland say in decisions listens more responsive 100% 92% 90% 89% 79% 80% 77% 73%

60%

40%

20% +13% +13% +16% 0% People have greater Administra on Administra on more Figure 6c: Emerging Federal States say in decisions listens more responsive

100%

80% 78% 68% 67% 59% 61% 60% 50% 40% Figure 6d: Mogadishu

20% +10% +8% +11% 100% 0% People have greater Administra on Administra on 80% 83% say in decisions listens more more responsive 68% 67% 60% 62% 56% 48% 40%

20% +21% +12% +19% 0% People have greater Administra on Administra on say in decisions listens more more responsive

In summary, in Somaliland, Puntland and Mogadishu, respondents unequivocally believe there is now greater collaboration with local government in terms of people’s participation in decisions, the extent to which local administration listens to them, and the responsiveness of local government. In the EFS the percentage of respondents who believe there is now greater collaboration has increased. However, there has been a decline in the percentage of respondents who strongly agree that this is the case. The four figures in Annex H provide further insights into the relationship between collaboration, confidence and four other indicators of good governance: local administration’s level of professionalism, satisfaction with basic services, the absence of nepotism, and the absence of corruption. In these graphs, collaboration is shown as the arithmetic average of the three elements of collaboration presented in Figures 6a to 6d. For example, collaboration in Somaliland in 2016 is calculated as the sum of those who strongly agree or somewhat agree with the three statements regarding collaboration, divided by three; i.e., 95% (people have greater say) plus 84% (administration listens more) plus 84% (administration more responsive) which, divided by three, equals 88%. Similarly, the graphs in Annex H show the level of satisfaction with basic services as a single point that was calculated as the arithmetic mean of the level of satisfaction with health services, drinking water, road/transportation, electricity, and education, presented in Annex I. As shown in Annex H, professionalism, collaboration and satisfaction with basic services have increased in all four geographic areas. Corruption is perceived to have decreased (i.e., the “absence of corruption” has increased) modestly in Somaliland and the EFS, and significantly in Puntland. In Mogadishu it is perceived to have risen considerably. Nepotism is seen to have increased (i.e., the “absence of nepotism” has decreased) everywhere except Puntland. For Somalia as a whole, Figure 7 indicates there is a weak but positive correlation between the absence of corruption and confidence in government, professionalism, collaboration and satisfaction with basic services. The correlation between nepotism and the indicators of good governance is perverse, in the sense that a perceived increase in nepotism does not appear to diminish confidence in government, although professionalism and the absence of nepotism have a weak, positive correlation. The correlation matrices for the four geographic areas may be found in Annex J.

Figure 7: Correlation between Indicators of Good Governance (Country Level)

Conf: Confidence in Local Government

Prof: Professionalism

Coll: Collaboration

BasS: Basic Services

AC: Absence of Corruption

AN: Absence of Nepotism

The information from the household survey was “Two years ago, the public did not trust the corroborated by the views of FGD participants who government at all. But things have changed and stated that collaboration between the public and the now the public and the government have a better government has improved in the last two years. Both relationship.” female and male FGD participants were positive about the collaborative efforts of government in particular. Female FGD participant, Xamar Jabab district Participants cited increased social cohesion, integration and the involvement of the private sector in government grants and planning meetings as indicators that the government was promoting collaboration and cooperation and was responsive to the community’s needs. They said that, in response to government initiatives, they were encouraged to cooperate with and support the government. In Somaliland, participants from all three types of FGDs (women, men and youth) gave the example of citizens who were increasingly and willingly paying taxes to the local government without duress as a sign of increased collaboration with and confidence in local government. In Puntland, FGD participants indicated there was more collaboration with local government. This view was based on FGD perception of increased joint security surveillance between community members and local government as well as by financial and material support to government initiatives such as roads and sanitation improvement grants. FGD participants indicated that, as a demonstration of their good relationship with both the government and TIS, they provided vital security information to avert violent incidents. The private sector was also said to have developed a better working relationship with the government through its involvement in TIS initiatives, which demonstrated the value in working closely with the local government.

TIS was credited by FGD participants “It is apparent there is a working relationship between the as having initiated and enhanced the government and the general public. You can see the women collaboration between the association, the youth, elders and religious leaders all confident in government, the public and the private the government. That shows there is a working relationship.’’ sector by involving all stakeholders in the processes of planning, Female participant, Yaqshid district implementation and monitoring of sub- activities. FGD participants indicated that the TIS-led procedure of granting contracts to local enterprises

through a competitive and transparent process has contributed to increasing trust in the local administration and that this, in turn, has contributed to increased collaboration. TIS was also credited for promoting collaboration by providing convenient venues (community social halls and sports stadiums) where various stakeholders could meet to discuss matters of interest to the citizens.

“Because of the collaboration, people have started linking up in general, especially when they have meeting places such as the district administration block, the community hall, the women’s building, and the sports ground, which is the most important place for youth. Youth have now started to meet and there is a good relationship among them, which has not been there before because the sports ground was constructed for them. They meet for many games. That is how it has led to more collaboration. Currently, the administration, women, youths and all the people have linked up. TIS has linked up the administration and the people nicely.” Male Participant, Dhusamareeb district

Conclusions Regarding Objective 2 The TIS process contributed to an increase of 10 percentage points or more in the perceived level of collaboration between people in settlements and local government. The TIS approach brought innovation to Somalia in terms of the collaborative, community-based process used to identify, prioritize, procure and implement projects. The TIS process brought communities (civil society and business people) local administration and central government together to discuss and identify the community’s priority needs. Through this process, TIS proved to be a strong catalyst in encouraging greater engagement between communities and government. The aspect of collaboration to which TIS most strongly contributed was in providing people in the settlement (women, men and youth) with greater input in decision-making. TIS also contributed to increasing the degree to which local administration listened to and was responsive to stakeholders in the community. Regarding the perceived level of corruption and nepotism, however, TIS does not appear to have strengthened the local government’s capacity to use transparent processes in appointing individuals based on merit. Achievement of Objective 3: Increased Dialogue on Peace, Recovery, and Development in Somalia Under Objective 3, TIS financed 82 grants covering a diverse range of sectors:

 Planning sessions (three-day CPS);  Civic education (e.g., voter education campaigns; peace caravans to engage youth in art and cultural activities through art competitions and travelling shows to schools in the region);  Communication, coordination and media;  Conflict mitigation;  Reconciliation (e.g., training on social healing and reconciliation; activities to enhance the role of arts, culture, and sports in promoting peace and countering violent extremism; training of trainers on trauma healing);  Service delivery (training of Peace Committees);  Infrastructure (e.g., rehabilitation of a national theatre; community peace gardens);  Institutional strengthening (to strengthen communication capacity of the Office of the Somali Prime Minister; training of Peace Committees);  Civil society (sports equipment for youth);  Community security and safety (e.g., bringing peace actors together to discuss social vices affecting stability, their causes and how to overcome them; develop an interactive radio program to trigger open discussion and understanding on how harmful habits are an affront to peace and how to overcome them; engaging local theatre groups in delivery of targeted messages on peace building

and the need to discourage social vices; skill building and presentation of interventions against harmful vices through arts and murals;  Education (e.g., a mobile library with materials, documentation and literature to promote social integration and common understanding of issues relating to citizenship and identity among the citizens of Somaliland). The impact, however, of TIS on increasing dialogue on peace, recovery and development went beyond these grants. When designing a new grant, the grant-tracking software permitted implementing partners to assign the grant to only one of the three objectives, despite the fact that a given grant may have contributed to more than one objective. For example, certain community infrastructure grants implemented under Objective 1 included the construction or rehabilitation of meetings halls where peace dialogues took place. They also included women’s and youth centers that created a space for community dialogue and conflict resolution, thus contributing to Objective 3. In this way, TIS supported dialogue through projects that fell under all three objectives. Dialogue on Peace, Recovery and Development The following are examples mentioned by FGD participants and Key Informants of how TIS facilitated dialogue on peace and recovery:

 Trauma-healing sessions helped in social integration and brought people closer to each other;  Dialogue activities brought diverse sectors of the community together and contributed not only to social integration but to economic recovery;  Peace dialogues facilitated by TIS led directly to conflict resolution between two clans in Bander Bayla district;  TIS empowered the community and sensitized them to understand the benefits of cooperating for the common good rather than for clan and individual priorities;  Peace committees and peace “This country has a long history of violence and instability. We meetings that TIS supported and facilitated have come through long periods of vicious violence and destruction. This is why we need to be mindful of maintaining served as forums for discussing issues and peace and continue to ingrain it in the community. There are for preventing and/or resolving conflicts occasional incidents of insecurity where explosions take place. within the community; There is need to continuously remind the people about the  Community infrastructure, such as importance of maintaining peace. People need to be sensitized, social halls, contributed to peace by we need to emphasize on this, people need to leave and return creating a physical space to discuss to homes peacefully. It is the duty of every citizen to help bring and maintain peace because it benefits everyone; yourself, community issues and resolve conflicts; family, children, neighbors and everybody else. Because FGD participants in all four geographic everyone needs peace, it is therefore not the sole responsibility of the government to talk about it. This is why peace should be areas associated TIS with Peace part of the community’s conversation.’’ Committees. Female FGD participants in all four geographic areas see the existence Female participant, Xamarweyne District of peace and conflict resolution committees, the collaboration between various stakeholders, and the efforts in community integration, as key contributors to peace and development. They also state that community members’ shared aim to uphold peace is an enabler of the relative peace being enjoyed across many areas in the EFS. Many participants credited TIS support of Dispute Resolution Centers as having contributed to peace. Now they have a specific place to meet and contribute to conflict prevention and resolution processes. They said that involvement of youth in sports and education has diverted them from violent activities.

Male focus group participants in all the geographical areas indicated they have been involved in peace and development meetings. They said grants implemented by TIS were started in a consultative manner and this may have encouraged the government to embrace more consultative approaches in dealing with matters of peace, security and development. Peace Committees already existed in the settlements before TIS. TIS took advantage of their existence and built upon their organizational structure. Members of the Peace Committees were community leaders. When TIS created the Community Development Committees as part of its consultative, participatory process, many of the Peace Committee members also participated in the CDCs. Figures 8a through 8d present the findings for the country as a whole from the household survey regarding the impact of the Peace Committees (please see Annex K for the findings disaggregated for each of the four geographic areas). Peace Committees have proved an effective mechanism for resolving local conflicts (Figure 8a) and are still widely used to address and resolve conflicts at the settlement level (Figure 8b). The evidence indicates that Peace Committees are increasingly used for addressing and resolving conflicts (Figure 8c). Seventy-two percent of respondents indicated that community dialogue and planning with local officials (both aspects promoted by TIS) have helped to make settlements a better place to live (Figure 8d). Key informants from all geographic areas commented on the role of Peace Committees in conflict prevention and resolution. Government officials indicated that TIS not only helped establish Peace Committees but trained and encouraged them to raise awareness at the community level. Figure 8: Peace Committees and Community Dialogue

Figure 8a. “Was the Peace Committee ever Figure 8b. “Does this settlement still use successful in finding solutions that ended the Peace Committee to try to address and the conflicts it addressed?” resolve conflicts in this settlement?”

2% 12% 10% Always 37% Some mes Yes Rarely No

50% Never 90%

Figure 8d. Community dialogue and Figure 8c. Extent to which Peace planning with local officials have helped Committees are used for addressing and make this settlement a better place to live. resolving conflicts. Increased in Strongly Agree the past 2 10% 20% years Somewhat 17% 38% Decreased in Agree 11% the past 2 years Somewhat 69% Disagree Remained the 34% same Strongly Disagree

Key informants credit TIS with enhancing peace through projects such as roads, street lighting, Peace Committees, women and youth centers, as well as community halls used by the peace and conflict resolution committees. TIS contributed to peace and development by renovating buildings that make it possible for people to meet and discuss development issues. FGD participants were of the view that TIS activities geared toward improving livelihoods for youth also made a positive contribution to peace by engaging youth in positive activities. In Somaliland, women are part of the peace and development system, working closely with government security Peace Committees according to the female FGD participants. Women were not previously involved in peace efforts, which had traditionally been carried out by elders. Now they are appreciated as propagators of peace and incorporated in relevant peace and development committees. A number of FGD participants indicated that TIS (through its gender inclusive approach to the composition of development committees) has been a catalyst to greater involvement of women. Some female FGD participants felt that security in Somaliland is considered a men’s issue and that women are not invited to participate in security discussions. However, women FGD participants said they had much to contribute to this topic and wanted to be more involved. In Puntland, FGD participants indicated that peace was vital for development and highlighted the contribution made by TIS in establishing infrastructure that promoted security such as street lighting, roads and promoting cooperation between the government and other stakeholders. In the Emerging Federal States, many FGD participants credit TIS with creating the necessary awareness, and providing training for community members to initiate peace and dialogue processes. The majority of female FGD participants said they were involved in peace and dialogue initiatives in their settlements. They said they were actively involved, and perceived peace and dialogue forums as a means of maintaining and promoting peace and security. Other members of the committees, according to the female participants, include elders, youth, community, religious leaders and representatives from government. All female FGD participants indicated awareness of the peace initiatives, either through direct involvement, or by knowing someone involved in peace initiatives. In Mogadishu, FGD participants credited TIS support of Dispute Resolution Centers/Community Halls as having contributed to peace. They noted they now have specific places to meet and contribute to conflict prevention and resolution processes. The involvement of youth in sports and education was said to have contributed to diverting youths from destructive activities in Mogadishu. Meetings supported by TIS helped ensure focus was diverted to positive community initiatives rather than hatred, discrimination and violence. Male FGD participants in Mogadishu also credit the good working relationship with local government administrators with having better enabled communities to take action and report suspicious individuals entering their communities. Community policing in cooperation with security agencies has provided information that has also enabled peace and development. FGD participants said local residents own the process of maintaining peace and will “tell on” anyone who is seen to work against the spirit of peace, including any new people coming into their communities. FGD participants said that TIS strengthened this process through the community social integration and good governance trainings. Female FGD participants in seven out of eleven groups in Mogadishu perceive collaboration and coordination between community members, stakeholders and security agencies as a key factor in enabling peace and development. The involvement of citizens in community policing and surveillance has been successful because people who have suffered violence and uncertainty in the past want to work together to keep the peace. Participants in five out of eleven groups credited TIS initiatives with having helped create the necessary awareness and provide the training for community members to initiate peace and dialogue processes. Similarly, female key informants credited TIS initiatives with promoting peace at the community level by renovating sport stadiums, where youth can spend their time constructively, as well as by providing community halls where Peace Committees can hold meetings to discuss issues. TIS also

18 contributed to community reconciliation, integration and conflict mitigation through targeted training coupled with other sub-activities. Youth FGD participants in Mogadishu credit TIS-supported community infrastructure as a key enabler of peace and development. They said the improvement of security enhanced collaboration brought people together to discuss maintaining peace. Youth also indicate that the renovation of community social halls by TIS has created a conducive environment for consultative meetings between stakeholders without an atmosphere of fear. Youth participants also view sports as an important way of bringing youth together for community integration and peace. They also credit Peace Committees and the availability of meeting spaces as enablers of consultative dialogue.

Conclusions Regarding Objective 3 TIS contributed to dialogue on development in Somalia through the community-based, consultative process through which all projects were identified and prioritized by stakeholders at the settlement level. The TIS process contributed significantly to improving dialogue and peace, through trauma healing, peace committees, and community consultative meetings.

Independent Effect on the Achievement of TIS Objectives To determine whether there was an independent effect on the achievement of TIS objectives, a statistical analysis (t-test) was conducted to compare (1) the average Stability Index for the cohort of settlements where TIS was the sole development actor present at the time TIS grants were implemented, with (2) the average Stability Index for the cohort of settlements in which other Development Partners (DP) were active. TIS-DAI and TIS-IOM indicated in which settlements they acted alone at the time that their grants were implemented and in which settlements other DPs were present. Table 1 presents the average Stability Index for each geographic area. It should be noted that, in the case of Somaliland for “TIS alone” and the EFS for “TIS plus other DPs,” there was only one settlement in each case. One settlement does not provide an adequate basis for conducting a statistical comparison of averages. Therefore, a comparison of means was not conducted for Somaliland and the EFS. In the case of Mogadishu, there were no instances of settlements in which TIS alone implemented projects. For the above reasons, the comparison of means could only be conducted for Puntland and for Somalia as a whole. The results of this analysis are presented in Table 2. A p-value greater than 0.05 indicates there is no difference between two groups that are being compared. In Table 2, the p-value for Somalia as a whole and for Puntland is greater than 0.05, indicating there is no significant difference between the mean Stability Index scores for "TIS alone" and "TIS plus other DPs" in either Puntland or Somalia as a whole. This suggests that TIS results were comparable in communities with and without other development programs, and gives greater confidence that observed results are not attributable to other programs in the TIS communities.

Table 1: Comparison of Average Stability Indexes by Geographic Area

Somalia Emerging Somaliland Puntland Mogadishu (Country level) Federal States TIS alone 65% 56% 68% 63% - TIS plus other DPs 66% 69% 66% 64% 64%

Table 2: Results for Comparison of Means (TIS alone and TIS plus other DPs)

P value Somalia 0.2739

Puntland 0.0812

VII. EVALUATION QUESTION 2: TIS CONTRIBUTION TO USAID GOAL & OBJECTIVES This section presents findings related to the analysis of the extent to which TIS contributed to the goal of increasing stability.

TIS Contribution to Increasing Stability The concept of stability is complex, given the large number of factors that influence the stability of a community or a country. For the purpose of this evaluation, USAID defines a stable community as one that has the following characteristics:6 1. Sense of security among the public; 2. Access to basic services - basic education, health, water and sanitation (The government may not need to provide these services. However, it must ensure availability of basic services at fair rates.); 3. Opportunities for economic growth; 4. Citizens have confidence in government structures / administration at the community / district level; 5. Government effectively partners with civil society and the private sector; 6. No support for violent extremists among majority of community members; 7. Return of Internally Displaced Persons (IDPs) or refugees; 8. Sense of justice among citizens; 9. Community members respect Somali culture and values. To analyze TIS’s contribution to increasing stability in Somalia, the evaluation team designed a Stability Index corresponding to the first six characteristics mentioned above. The seventh characteristic (IDPs or refugees who left the settlement and have returned) was not incorporated into the Stability Index because not all TIS settlements had residents who left the settlement to become IDPs or refugees. The eighth and ninth characteristics were not incorporated into the index because TIS did not directly address them. Time series data provide the best source of information for analyzing the extent to which a change takes place over time. For this reason, a comparative approach was used to compare 2016 data collected as part of this evaluation and the data collected for the 2014 Stabilization Survey. The 2016 household survey included a number of questions from the 2014 Stabilization Survey. The direct comparison of these two years is limited to the following characteristics of stability, for which data were available for 2014:

 Sense of security  Confidence in government  Collaboration  Basic services In addition to these four characteristics, the Stability Index for 2016 also incorporates two additional characteristics:

 Economic growth opportunities  No violent extremism support

6SOL-623-15-000012 – Statement of Objectives for the Final Performance Evaluation of the USAID/SOMALIA Stabilization Project.

To enable a comparison of “like with like,” a partial Stability Index for 2016 was calculated solely comprising the four characteristics common to both years. A summary of the findings for this partial measurement is presented in Figure 9. Detailed findings are presented in Annex L.

Figure 9: Stability Index (Partial Measurement)

100%

80%

69% 60% 65% 66% 67% 2014 51% 52% 40% 48% 46% 2016

20%

0% Somaliland Puntland Emerging Mogadishu Federal States

Figure 9 shows that the Stability Index rose significantly in all four geographic areas: by 14 percentage points in Somaliland, 18 percentage points in Puntland, 15 percentage points in the EFS and 23 percentage points in Mogadishu. Although “return of IDPs or refugees” was not included in the formulation of the Stability Index, information was collected on this issue from FGDs in areas where IDPs or refugees had returned. A summary of findings is presented below:7 Somaliland: According to the FGD participants in Somaliland, there are no returnee IDPs or refugees that have come back. Somaliland has been stable for a long time and therefore FGD participants indicated they host refugees from Yemen and Ethiopia as well as Somali climatic refugees, fleeing drought, from other geographic areas. Puntland: FGD participants indicated that, since their state is stable, they have no local returnee IDP population, but they are host to IDPs from the EFS who have fled violence and extremism. There are returnees from Yemen and other Arab countries who had left looking for better opportunities but have returned back home after those countries became volatile. These were not necessarily refugees. When they returned, they settled back in the homes they had left behind. Puntland is also host to Yemeni refugees. Emerging Federal States: Respondents in 18 FGDs indicated that there are both returnee IDPs and refugees in their settlements. They have returnee refugees from the Diaspora and from refugee camps in Kenya. Mogadishu: Participants in all FGDs that answered this question indicated there have been returnee refugees and IDPs that have come back from various places – Kenya, Ethiopia and Sudan and the general diaspora. People that were internally displaced, to other parts of Somalia that were considered more secure, have also come back to their settlements. The reasons cited by respondents in Mogadishu for coming back include:

7 The information presented in this section reflects the perceptions of FGD participants. According to OCHA statistics, there are still over 360,000 IDPs in Mogadishu. There may also be IDPs in Somaliland and Puntland of whom FGD participants may not have been aware.

 There is peace and security in Somalia. Somalia is much safer than it has been since the collapse of the central government.  There is a functional central government and functional regional and local governments that have ensured peace and security.  Due to the relative peace, there are those who have come back to help rebuild their country.  Those living in IDP camps in other countries were never comfortable. However, these camps promised peace and security they lacked in Somalia. Now that many parts of Somalia are peaceful and the violence by Al Shabaab has stopped, they have decided to come back. Were Any TIS Approaches Especially Effective in Contributing to Stability or to the Achievement of USAID Objectives? Statistical analysis was used to identify the blend of TIS sector and sub-sector grants that resulted in the highest rating on the Stability Index. The team found, however, that virtually every community received a unique mix of grants, a reflection of how flexible, adaptable and responsive TIS was to each community’s needs. Therefore it was not possible to compare grant bundles using that technique. A different avenue was then explored. The evaluation team identified the communities with the “top 20” highest Stability Index score and the blend of grants that had been provided to each of those communities. Next, the most common patterns of grant mixes in those “top 20” grants were identified. Figure 10 presents the findings. Sixty percent of the settlements with the highest Stability Index score received a bundle of grants that included community infrastructure, institutional strengthening and economic development, plus a wide variety of other grant types. Fifty-five percent of communities in the “top 20” received grants in the following sectors: community infrastructure, institutional strengthening, economic development, and health and hygiene.

Figure 10: Grant Bundles with Highest Stability Index

As an additional avenue of exploration, econometric analysis (beta regression analysis) was conducted to determine whether there was any significant statistical evidence that settlements with projects in certain sub-sectors have a higher rating on the Stability Index. The findings, presented in Annex C, suggest that settlements ranked significantly higher on the Stability Indexes when they received grants in one or more of the following sectors:

 Economic Development/Livelihoods  Institutional Strengthening  Roads  Political Process  Service Delivery

Additional detail on the specific activities that were commonly seen in the above sectors is presented in Annex C, Section 8.

Question 2 Conclusions  The Stability Index (SI) in TIS communities rose by 15 percentage points on average.  Of the elements comprising the SI, TIS contributed most significantly to increased collaboration, improved professionalism of local administration, and to improving basic services.  The community-level, demand-driven projects TIS supported were very diverse. Each community received a unique mix of grants (this is seen as a strength).  A mixture of grants that appeared particularly effective included a combination of Institutional Strengthening, Community Infrastructure, Economic Development, and Health & Hygiene.

VIII. EVALUATION QUESTION 3: INSTITUTIONALIZATION OF THE TIS PROCESS AND COMMUNITY USE OF TIS PHYSICAL STRUCTURES This section analyzes the extent to which the processes modeled by TIS have become institutionalized, and whether the physical structures constructed by TIS continue to be used by communities. “TIS created a great unprecedented impact. The Institutionalization of the TIS processes planning conference in Djibouti had a great impact This section presents evidence regarding the extent on me. It was the first time where the communities to which the TIS process of community-based were listened to. The openness, transparency, consultation and transparent procurement processes consultation and participation were all new to the have been institutionalized. Based on key informant people who participated. interviews with Somali government officials, the Based on the experience I gained from TIS and the evaluation team found the following evidence that impact it left with me, I will develop a Master Plan indicates that TIS processes have been or will be for Galmudug State, which follows the same adopted by the following entities and actors: consultative process of TIS. My ministry  will adopt the TIS approach.  TIS processes have been used and We want to develop a Master Plan where all our institutionalized mostly in Somaliland and needs are recorded in detail and with budgets. Puntland. TIS processes have not been Anyone who wants to support us will consult the adopted as much in the Emerging Federal Plan.” States and Mogadishu, perhaps partly because Senior Government Official, Galmudug Interim there has been more turnover of Administration government officials there.8  The National Tender Board (Somaliland) has adopted TIS procurement processes. All government ministries in Somaliland go through the NTB for the procurement of goods and services. TIS-IOM strengthened the NTB by working with them on procurement processes for TIS grants, included embedding procurement experts in the NTB. Thanks to this capacity-building and

8 Some of the EFS only recently emerged as federal member states during latter stages of the TIS implementation period.

the on-the-job practice during the implementation of TIS, the NTB has been entrusted with the management of other large funds, such as the Kuwait Government funds for the Hargeisa and Berbera airports. Prior to TIS, the NTB only conducted procurement for government projects.

 The Banadir Regional Administration has created a procurement committee and is using systems for purchasing, recruitment and accounting.  Some newly graduated students embedded with district and regional government have advanced to higher position levels.  Local communities like the TIS approach and have started challenging other agencies (NGOs and UN) to work with grassroots consultation.  Some districts supported by TIS (e.g., Bandarbeyla) have developed procurement units. Tendering for contracts has become a governance good practice. These districts began using these practices after working with TIS.  In places where nascent TIS procedures were already being used, TIS empowered and strengthened those systems. TIS procedures are now being applied to development projects (previously they were only used for government projects).  The Galmudug Interim Administration intends to adopt the TIS processes in all of its ministries.  TIS processes have been embodied in the Wadajir National Framework for Local Governance launched by President Hassan Sheikh Mohamed in March 2016. Two major challenges that were cited for the continuation of the adoption of TIS processes: the lack of other projects by the local administrations and the high turnover of government officers.

Continued Use of TIS Physical Structures According to FGD participants, most of the physical structures built by TIS continue to be put to good use, are used as intended, and are equitably accessible. TIS infrastructures are perceived to be public facilities under the custody of government, with the government having the primary responsibility for protecting and maintaining them. “No there are no obstacles at all to access to the building. The In general, FGD participants spoke buildings don’t belong to any individual but the whole community shares them and the buildings belong to the community. Other positively of the contribution TIS communities who come to this district also use the buildings, and infrastructure has made to the no one is refused access to the buildings, be it the office or the improvement of living conditions in their other groups such as women’s associations, the hospital, the respective settlements. When asked, MCH, or the ambulances. Everyone (?) is allowed to use the however, whether there were any issues buildings” with TIS infrastructure, FGD participants Government official, Bargaal mentioned the following issues:

 Ownership: some TIS structures were reported as privately owned, rather than government owned (e.g., repairs made to a privately-run school in Mogadishu; a school being used for non-educational purposes);  Incompleteness: at the time of the FGDs, a number of TIS structures had not yet been completed. It is noted that, at the time of the FGDs, DAI work was still ongoing;  Location/inaccessibility: there were “TIS has built markets at the outskirts of the city. The cases where the TIS structure was markets are not currently in use due to security concerns. built in a location that women did The buildings are of high quality but the offices and the other not consider to be safe by some buildings are not used due to fear of security threats. We ask intended users (e.g., the market and TIS to consult with the women of Baidoa and build centers Women’s Social Centre in Baidoa); for them in the city, not at the outskirts of the city.”  Size: some FGD participants Women’s FGD, Baidoa considered the TIS structures to be

too small;  Quality: a number of quality-related issues were reported (examples: Galkayo: asphalt too thin; poor quality substrate; Garowe: roads have developed potholes; Kismayo: issues with cracks in roads; shoulder has started to chip; Mogadishu: cracks in building walls). Figure 11 presents a summary of the issues reported by FGD participants. The bar chart indicates the percentage of FGDs reporting a particular issue. Annex L presents a detailed summary of the FGD findings on this topic, with details on the location and specific issues reported. As part of this evaluation, on-site visits were undertaken by the evaluation team to a sample of TIS physical structures. Using an observation checklist, evaluators assessed whether there were any issues with the location, size or quality of the project (issues of ownership and incompleteness were not included in the evaluation checklist). The percentage of structures in which issues were detected on-site by the TIS evaluators is indicated in Figure 11. Direct observation indicated that location/accessibility was an issue in more cases than the FGDs had reported. In contrast, FGDs noted issues related to size and quality at a rate higher than that found by the evaluators during on-site visits.

Figure 11: Issues Reported on TIS Physical Structures

Ownership FGD = 6%

Incomplete FGD = 13%

Loca on/inaccessibilty FGD =5% [On-site observa on = 9%]

Size FGD = 13% [On-site observa on = 4%]

Quality FGD = 23% [On-site observa on =17%]

0% 20% 40% 60% 80% 100%

Percentage of Focus Groups Repor ng an Issue

Somaliland Puntland Emerging Federal States Mogadishu

 According to some FGD participants, some buildings are underutilized because they were not equipped in the way that the participants of the planning sessions had understood they were to be.  One FGD reported that quality issues identified by Project Oversight Committees were not taken into account by the engineer. According to the key informants from IPs, two factors may have contributed to the quality issues on some TIS projects:

 Lack of international engineering technical oversight: The principle of local ownership that characterized TIS led, during the first years of the project, to the exclusive use of Somali engineers, Somali engineering companies, and/or Somali/Kenyan engineers. There was no international engineering expertise contracted on TIS until mid-2014, when TIS-IOM engaged the services of an engineer who was based at the airport in Somalia. In the opinion of the IPs, international expertise helps to ensure that international engineering standards and quality assurance are adhered to.

 Lack of Project Oversight Committees (POCs): In mid-2013, IOM introduced POCs to provide community-based oversight of TIS infrastructure projects. The POCs comprised members from the settlement who live within walking distance of the project, to monitor its

implementation. The POCs included at least one member of civil society, at least one member of TIS, and at least one member of local government. A POC that has received good orientation and training can provide an IP with reliable information about what is going on at the project site, basic quality control information, as well as up-to-date information about the security situation. DAI also experimented with the POC concept in Baidoa. The experience proved unsuccessful, however, because some of the POC members became “gate keepers” of the project. Following that unsuccessful experience, DAI did not try to establish POCs for other projects. This points to the need to provide POCs with adequate orientation and training on their role, as well as to ensure that the community as a whole is made aware of the POCs’ role. Question 3 Conclusions  TIS processes have been or will be adopted by important entities and actors at the district and central level.  Communities that have experienced TIS processes are applying pressure on local administrations to continue using those processes.  Administrations and local communities alike expect and demand other development actors to follow the community-driven consultative planning processes introduced by TIS.  TIS practices have been used and institutionalized in more stable environments, mostly in Somaliland and Puntland.  TIS practices have not been adopted as much in the south, possibly because there has been more turnover of government officials there.  Most of the physical structures built by TIS continue to be put to good use, are used as intended, and are equitably accessible. TIS infrastructures are perceived to be public facilities under the custody of government, with the government having the primary responsibility for protecting and maintaining them. TIS infrastructure is seen to have contributed to improving the living conditions in settlements.  Notwithstanding the overall positive view of TIS physical infrastructure, there are a number of issues, the most numerous of which are related to quality control.  The use of an International Engineer and community-based POCs to provide technical oversight could help to improve the quality of engineering projects.

IX. EVALUATION QUESTION 4: TIS SUCCESS AT ADDRESSING GENDER ISSUES This section analyzes how successful the project was in addressing gender equity and empowerment issues, such as increasing inclusion of women in community-level decision-making, promoting representation of women’s issues, enabling women to advocate for themselves and take action, and ensuring both men and women benefit from the activity. Increased inclusion of women in community-level decision-making An important contribution of TIS in increasing the inclusion of women in community-level decision-making was the requirement that women should comprise at least 30% of the membership on the Community Development Committees.9 However, this percentage was a target, not an actual requirement to hold a planning session. Figure 12 shows the percentage of focus groups in which participants indicated there was increased inclusion of women in community-level decision-making compared to two years ago. “Inclusion” in this context refers to the presence of women in various decision-making platforms, while “representation” refers to participants’ perception of the extent to which their interests are being well represented and

9 In 2014 both DAI and IOM increased this target to 40%.

promoted by women who are in decision-making positions. FGD participants indicated that, while there are more women in decision-making positions, they are too few in number or not sufficiently influential to adequately and fully represent women’s interests.

Figure 12: Increased Inclusion of Women in Community-Level Decision-Making and in Representation of Women’s Interests

Other findings from the FGDs were:

 There has been an increase in women’s inclusion in decision-making in areas such as peace and conflict resolution committees, women’s participation in local councils and community development committees.  Female participants in the recently liberated areas said they no longer feel confined to their homes due to insecurity and are able to go out and work. Participants said they felt empowered as a result of participating in CPS and TIS skills trainings.  There has been increased participation of women in education in the past two years in the EFS including Mogadishu, particularly in areas supported by TIS. More women are members of school boards or management/committees; others have volunteered as teachers.  Women in Puntland are less involved in the political process than they are in other regions. However, their role in communal work, peace-building and economic activities reflects their involvement in important community issues.

Enabling women to advocate for themselves and take action Across the four geographic areas, the findings from the FGDs indicate respect and support for women’s interests has increased, both at the community and government levels. Across the four geographic areas, female FGD participants indicated they have created a culture of self-mobilization. After receiving training from TIS, they started to undertake more initiatives. Moreover, female FGD participants indicated that TIS-financed community facilities, such as the social halls, have also provided a physical space that gives women the opportunity to meet, mobilize and hold consultative meetings. According to the FGD participants, TIS trainings on governance have contributed to an increase in women’s interest in local politics. In all three regions, women’s organizations were seen to be champions of women’s rights and advocating for women’s issues. Figure 13 presents the findings from the survey on the question “Who advocates for women”? Community women volunteers, women leaders, and women’s organizations are perceived as the principal actors advocating for women. Local government was only mentioned as an advocate for women in 5% of FGDs.

Figure 13: Perception Of FGDs Regarding Women’s Advocacy

Development partners/NGOs Local 8% government 5% Community Women Volunteers 38% Women’s organiza ons/ 29% women’s groups

21% Professional women/women leaders Ensuring both men and women benefit from the activity Across the four geographic areas, female FGD participants said that increased economic activities of women have raised their status, as they are now better able to access and control economic resources. FGD participants indicated that TIS economic/livelihood training, markets and street lighting have increased the economic activity of women, which has contributed to increasing their level of respect in the settlement. The trainings mentioned by the women included: vocational skills training (arts and crafts, tailoring, cookery), computer training, training of trainers (TOT) on trauma healing and counseling, good governance, social integration and conflict resolution. In Somaliland, the garbage trucks were greatly appreciated across all focus groups, especially by the women FGDs. Men indicated they benefited from improved service delivery and infrastructure. Evaluation Question 4 Conclusions Across the three regions, there has been increased respect and support for women’s interests, both at the community and government levels. TIS contributed directly to an increase in women’s participation in decision-making in TIS communities and to an increase in the visibility of their participation. This, in turn, has contributed to a positive change in the way in which women are perceived in TIS communities, opening up opportunities for women to be involved in other community forums.

X. EVALUATION QUESTION 5: EFFECTIVENESS OF COMMUNITY PLANNING SESSIONS This section presents findings regarding the effectiveness of CPS in terms of their ability to produce decisions that are inclusive, consensus-based and reflective of community priorities. Figure 14 presents a summary of the findings by geographic area from FGDs comprising women, men and youth, as well as from KIIs with women, men and government officials. Each square represents the summary of opinions for one geographic area and one type of FGD or KII. Each square comprises two triangles: the bottom-left triangle represents the opinion regarding whether the CPS were inclusive and consensus-based, while the upper-right triangle represents the opinion regarding whether the projects implemented by TIS reflected community priorities. The triangles are shaded dark green, light green or red to indicate whether the

majority opinion was positive, intermediate, or negative. For example, a dark green triangle on the bottom-left side of the square for Somaliland men’s FGDs indicates that the majority of men’s FGDs strongly agreed that the CPSs were inclusive and consensus-based. Majority mixed opinions are represented with mixed-color triangles. For example, the red/green triangle on the top-right side of the square for women’s FGDs in Mogadishu indicates that the majority of the women’s FGDs were of mixed opinion regarding the extent to which projects reflected community priorities. Some women’s FGDs strongly agreed that the projects reflected community priorities, while others strongly disagreed. Minority opinions are indicated in Figure 14 with circles colored either light green or red, indicating that a minority of individuals in the FGDs had an opinion that differed from the majority view. According to the findings from the FGDs and KIIs, most CPS were both representative and effective in “Unlike other development partners, TIS IPs producing decisions that were inclusive and reflective consulted with various stakeholders and left the of community priorities. The projects prioritized by decision of the priority projects to be determined by the communities were the ones implemented under the government and the people”. TIS that contributed to support for and ownership of Government official, Puntland those projects by local government and the community. Multi-stakeholder involvement in CPS was hailed as one of the strengths of the TIS approach. In some areas in Puntland and Somaliland a majority of youth FGD participants felt they were not represented. A minority of participants also said that they considered the choice of representatives in the CPS was not always inclusive.

Figure 14: Perceptions of the Effectiveness of TIS Community Planning

Women’s Men’s Youth Female Male Gov’t FGDs FGDs FGDs KIs KIs KIs Minority intermediate view Minority nega ve view Somaliland Majority posi ve view Majority intermediate view Puntland Majority nega ve view Emerging Majority mixed opinion Federal (intermediate with nega ve) States Majority mixed opinion (posi ve with nega ve)

Mogadishu Reflected Inclusive and Community consensus-based Priorities

Respondents to the household survey were asked how many of the grants selected through CPS’s priorities were for the people in their settlement. Table 3 presents a summary of findings, showing the percentage of respondents who consider that community planning meetings have been effective in “The last meeting in which we selected the tarmac road, the selection of projects that are priorities for the I remember that all the sections of the community people in the settlement. Additional details are participated: women, mothers, youth, police, members of presented in Annex M. the district council, and the business community. The general public, including every section, participated in it. I Only 61% of respondents, on average, were aware that recall that we sat in that meeting for three days.” TIS projects were selected through CPS, reflecting the Female youth focus group participant, Somaliland lack of an adequate strategic communication strategy

to inform the general population in the settlements about the project selection process. On average, 87% of respondents thought that “all,” “most” or “some” TIS grants addressed settlement priorities. Table 3: Knowledge of CPS and Perception of CPS Effectiveness Did projects selected through the Were TIS Projects Selected CPS meet the priorities needs in Geographic Area through CPS? (% “Yes”) your settlement? (%)1 Somaliland 52 86 Puntland 84 92 Emerging Federal States 43 82 Mogadishu 63 89 Average 61 87 1 The percentage shown is the sum of respondents who answered that “all,” “most” or “some” of the projects met the priority needs in the settlement.

XI. RECOMMENDATIONS This section provides actionable statements for ongoing programming and recommended future objectives and types of activities based on lessons learned. The recommendations are intended to be action-oriented, practical and specific, with defined responsibility for the action.

1. Include a strategic communication component and strategic communication activities in each grant. The TIS project, through its grants and processes, aims to strengthen confidence in and increase the local government’s capacity to improve people’s lives. It is therefore essential that a strategic plan is put in place to communicate to the general population the message that the local government has brought funds to the community, has successfully implemented projects, and has done so through a collaborative process of community consultation in which projects were prioritized. Project similar to TIS should therefore include a strategic communication plan in each grant activity which, moreover, is a separate sub-sector of itself. In addition to training government officials and building their capacity for constituent communication, it is essential to build government capacity for constituent communication once the project is complete.

2. Project Oversight Committees: These important oversight committees were missing from both the DAI and IOM activities until 2013, when IOM began using them. POCs are important, especially for infrastructure projects. A committee, with members who live within walking distance of the infrastructure, is needed to monitor the project’s implementation. The committee should include at least one member of civil society, at least one member of TIS, and at least one member of government. At least three members of the committee should be able to walk to the site on a daily basis, eliminating the need for transportation and per diem costs. With proper training, local committee members would provide the best “eyes and ears” possible because they would have been involved in the planning session and would know the required design. The POC can provide the latest security information. IPs can call and ask a POC member to check quantities, take photographs, and verify if the contractor is on site. To ensure proper conditions for the POCs, there should be an orientation on the POCs’ role for all stakeholders, including the contractor, government, and community. In an ideal scenario, the POCs’ government representative would report to the local administration on a daily basis, the civil society representative would report to the community and the TIS representative would report to TIS.

3. Better supervision and engineering oversight is required during implementation of infrastructure projects.

 The POCs should be strengthened and institutionalized. The POCs should be provided with or provide adequate training for adequate monitoring of projects during implementation. This will also ensure that the contractor and the community as a whole properly understand the role of the POC.

 International engineers should provide oversight of infrastructure activities. While respecting the principle of local ownership and the principle of using Somali engineers, Somali engineering companies, and Somali/Kenyan engineers for project implementation, in order to help ensure that international engineering standards are adhered to, and to help streamline the design approval and tendering process international engineering expertise (at least one expert) should be hired to play a management/quality control oversight role with respect to local engineers.

4. A sustainability plan, including ex-post monitoring should be prepared and implemented for infrastructure projects. A sustainability plan should be prepared for each project. The POCs’ role should be expanded to include responsibilities related to monitoring the maintenance of infrastructure and the correct utilization of infrastructure after implementation.

5. Budgeting: Settlement should not be assigned a fixed amount of available funds to individual settlements. The assignation of a fixed budget limits flexibility and encourages micro-management of budgets (i.e., scrutinizing every small expense). Under TIS, additional community meetings were required to decide how to use any leftover funds, causing delays that thwarted the intention effect of “quick impact” grants. Budgeting should not be a feature of the community prioritization process. Instead, grants should be provided with no dollar figure attached. The community should get the deliverable (infrastructure, goods or service), not the money.

6. Develop a Master Development Plan at the Settlement. Rather than doing one-off CPSs, a more effective and sustainable approach would be to help each beneficiary settlement develop a multi- year Master Development Plan using the TIS process and create a steering committee comprising local government, TIS, other Development Partners, civil society and private sector.

7. Diversify beneficiaries: Under TIS, the Mayor or District Commission (DC) was named as the beneficiary and signatory in almost all settlements, placing a great deal of authority and power in one individual’s hands. There were cases in which the Mayor or DC became the gatekeeper. This created an impasse that could have been avoided had there been more than one grantee. For instance, the beneficiary of MCH Centre could be the director of the center, and the school principal could be the beneficiary of a school infrastructure project. This diversification of beneficiaries could be done in such a way that the local administration’s role in bringing the project to the community could still be maintained. In this way each project contributes to improving collaboration with and builds confidence in local government.

Annex A: Statement of Objectives for TIS Final Performance Evaluation

SOL-623-15-000012 - STATEMENT OF OBJECTIVES FOR THE FINAL PERFORMANCE EVALUATION OF THE USAID/SOMALIA STABILIZATION PROJECT

SECTION B – SUPPLIES OR SERVICES AND PRICE/COSTS

B.1 PURPOSE This Task Order must provide all evaluation services as described in detail in Section C.

B.2 CONTRACT TYPE This is a Firm-Fixed-Price Task Order.

B.3 PRICE The total price of this Task Order is TBD.

B.4 APPLICABILITY OF IDIQ All Sections from the IDIQ are hereby incorporated. If there are any discrepancies between IDIQ and the Task Order, then the Task Order shall take precedence.

SECTION C: STATEMENT OF OBJECTIVES

1. PURPOSE The purpose of this Statement of Objectives (SOO) is to provide guidance to the Contractor on developing a statement of work (SOW) that fulfills the requirements of this RFTOP. Relevant Sections of the Contractor’s Approved Final Technical Proposal in response to this SOO will be incorporated as Section C of the contract.

2. BACKGROUND Starting in February 2010, the USAID/Kenya and East Africa (KEA)/Somalia Office awarded two activities10 named Transition Initiatives for Stabilization (TIS). While these activities were implemented in different locations in Somalia, the overarching goal and type of interventions undertaken by the two activities are similar. Therefore, these activities can be combined to form one project – a Stabilization Project. Thus, for this evaluation, USAID would like to evaluate the performance of its’ Stabilization Project collectively rather than the two activities individually.

The first activity was awarded to International Organization for Migration (IOM) and is referred to as TIS-IOM. The second activity was awarded to Development Alternatives Inc. (DAI) and is referred to as TIS-DAI.

The details of each award are as follows: Name: TIS-IOM Award Number: AID-623-G-10-00005 Award dates: February 2010 to January 2016 Funding: $45,340,640

Name: TIS-DAI Award Number: AID-623-C-11-00007 Award dates: April 2011 to March 2016 Funding: $57,435,291

The two TIS activities focus on increasing public trust and confidence in government, and increasing social cohesion through a variety of interventions. TIS partners gather community stakeholders to

10 The terms activity and project have been defined as per current ADS 200 series

A-1

collaboratively identify and prioritize interventions that will provide the greatest benefit to their communities. For TIS, the process of determining priorities and jointly moving forward is as important as the final product. In addition to working at the community level, TIS partners also work at the district, regional, and federal levels with a variety of actors, including the government, civil society and private sector.

TIS activities are implemented across Somalia, in both the relatively stable north and the newly liberated south. As of July 2015, TIS had supported over 800 projects in 17 of Somalia’s 18 regions.

Project Goal The two TIS activities were designed to contribute towards the goal and both objectives of the USAID/KEA/ Somalia Office. Thus the activities adopted the Somalia Office’s goal at the activity level. However, in 2014, the USAID/Somalia Program Evaluation found that "There is evidence that USAID- funded activities have resulted in noticeable change in Somalia over time, and that this change may have contributed to perceived increased stability in key geographical areas," but that "this noticeable change is not supported by the Evaluation Team’s quantitative data analysis of the IR indicator data, and consequently calls into question claims of measurable, increased stability in Somalia." Given no definition of "stability" was articulated for the USAID goal in Somalia, and the challenges involved in trying to measure a change in stability at a national level, for the purpose of this project evaluation, the goal of the project should be taken to be: Somalia's stability increased through targeted interventions that foster good governance and economic recovery and reduce the appeal of extremism in the communities in which the activities were undertaken.

The concept of stability is a complex one given the large number of factors that influence the stability of a community or a country. Based on its experience and other studies, for the purpose of this evaluation, USAID would like to define a stable community as one that has the following characteristics:  Sense of security among the public  Access to basic services - basic education, health, water and sanitation (The government may not need to provide these services. However, it must ensure availability of basic services at fair rates)  Opportunities for economic growth  Citizens have confidence in government structures / administration at the community / district level  Government effectively partners with civil society and the private sector  No support for violent extremists among majority of community members  Sense of justice among citizens  Return of IDPs or refugees  Community members respect Somali culture and values

Theory of Change and Results Framework Even though the two activities adopted the same goal, the wording of their respective Theories of Change and Results Framework differs as detailed in the sections that follow.

IOM’s Theory of Change (TOC) and Results Framework (RF) As stated in its Activity Monitoring and Evaluation Plan dated April 2015, IOM’s TOC is founded on supporting transformative relationships and processes. The multi-part TIS-IOM TOC is: Enabling a more consultative process and supporting Somali communities’ participation in governance, TIS-IOM helps to open up the space for civic engagement. With a more inclusive civic space, communities, particularly marginalized groups’ are empowered to collaborate and partner with government. Increased government presence and better access to information contribute to improving community trust and confidence in governance structures. A more engaged and informed public combined with a more responsive, accountable government lays the foundation for long lasting impact and greater stability in Somalia.

IOM did not develop an RF for the TIS activity. Instead, it adopted USAIDs RF as follows:

A-2

USAID Goal: Somalia stability increased through targeted interventions that foster good governance, economic recovery and reduce appeal to extremism USAID Objective 2: USAID Objective 1: Improve social service delivery and Strengthen local and national government economic growth capacity to provide good governance

USAID IR 1.1: USAID IR 2.1: USAID IR 2.2: Capacity of key USAID IR 1.2: Access to quality Economic government Participatory basic social opportunities institutions to political processes services increased increased perform essential promoted functions improved

FIGURE 1. USAID RESULTS FRAMEWORK ADOPTED by TIS/IOM

IOM did not articulate activity-specific objectives within the results framework above; however, it did develop a set of three objectives at the beginning of its activity implementation, as follows:

IOM’s original objectives: Objective 1: Increasing the visibility of, and confidence in, all levels of government through the delivery of targeted, strategic interventions focusing on improved service delivery

Objective 2: Increasing access to information on peace, recovery, and development issues in Somalia through support to the media and strategic communication activities

Objective 3: Supporting the creation of collaborative and strategic partnerships between government institutions, private sector, and civil society to build or expand the civic space

These objectives were similar but not identical to DAI’s objectives (see Figure 2 below). However, midway through implementation, IOM chose to drop their objectives above and instead adopt DAI’s version of the objectives, below. Given this fact, SPSS should evaluate the entire TIS project against the objectives listed in Figure 2.

DAI’s Theory of Change (TOC) and Results Framework (RF) Per its Activity Monitoring and Evaluation Plan dated February 2015, DAI uses a TOC that states: A combination of improving confidence in licit government and collaboration between government and citizens will work together to contribute to a more stable Somalia. Collaboration between citizens and government leads to increased ownership and improved abilities to mitigate conflict, develop meaningful social cohesion and political connectivity, which will, in turn, improve institutional confidence and trust – elements that are central for political stability.

A-3

This TOC led to development of a RF with three objectives, as shown below:

USAID/Somalia Goal: Somalia's stability increased through targeted interventions that foster good governance and economic recovery and reduce the appeal of extremism

USAID Objective 1: USAID Objective 2: Strengthen local and national capacity to promote Improved social services delivery and good governance economic growth

TIS TIS Objective 1: TIS Objective 3: Increase confidence in all Objective 2: Support levels of government through Increase dialogue collaboration the delivery of targeted, on peace, strategic interventions that between recovery, and government, private improve service delivery development in sector and civil Somalia. society.

FIGURE 2. RESULTS FRAMEWORK USED by TIS/DAI

C.3 EVALUATION PURPOSE The main purpose of this final performance evaluation is to get a better understanding of the extent to which USAID’s stabilization activities in Somalia have met the project goal articulated herein (Section C.2). Another purpose of this evaluation is to learn lessons about conducting stabilization activities in Somalia to improve future programming.

The audience for this study is the USAID/KEA/Somalia Office, DAI, IOM, other USAID offices, such as those managing funds for activities that address complex crises and activities that aim to counter violent extremism, and other donors supporting Somalia. The evaluation will also be used by AECOM, the partner implementing the Transition Initiatives for Stabilization Plus activity.

C.4 DISSEMINATION AND UTILIZATION PLAN The final performance evaluation will help guide USAID in improving the design of its future stabilization interventions in Somalia. The evaluation report will be uploaded to the Development Experience Clearinghouse and shared with USAID staff, implementing partners and other donors supporting Somalia. The report will also be shared with the Federal Government of Somalia’s Ministry of Interior, a key TIS partner. In addition, the USAID/KEA/Somalia Office will develop an action plan to track any changes made to programming based on this evaluation.

C.5 KEY EVALUATION QUESTIONS The following questions must be answered by the evaluation team:

I. To what extent have TIS/DAI and TIS/IOM achieved their objectives? Did the type of approach used have an independent effect on the achievement of TIS objectives?

A-4

II. To what extent has the project contributed to the goal stated herein (Section C.2.) and/or to the achievement of USAID objectives? Were any TIS approaches especially effective in contributing to stability, as defined in Section C.2 or to the achievement of USAID objectives? III. To what extent did the processes modelled by TIS become institutionalized, and did physical structures constructed by TIS continue to be used by communities? IV. How successful was the project in addressing gender equity and empowerment issues, such as increasing inclusion of women in community-level decision making, promoting representation of women’s interests, enabling women to advocate for themselves and take action, and ensuring both men and women benefit from the activity? V. To what extent were community planning sessions effective in producing decisions that were inclusive, consensus-based, and reflective of community priorities?

Gender considerations must be taken into account in answering each question. In addition to gender disaggregation, the Contractor should capture different trends between male and female beneficiaries, to the extent possible

The Contractor must consider the following information when selecting methods for answering key questions.

For Question 1, the evaluation team will need to determine how well each partner (DAI and IOM) met each of the objectives to which they were accountable, as noted in the results frameworks above. The evaluation team should measure progress toward the unified set of three objectives used by both partners in the second half of the program (see Figure 2). The team will need to assess and advise on what, if any, effect the revision of objectives midway through has had on the measurability of IOM’s achievement of objectives and adjust Evaluation analysis plans accordingly.

In looking at the goal for Question 2, the evaluation team must consider the characteristics of a stable community listed earlier in Section C.2. The team must propose measures / indicators for each of these characteristics that will be used to collect data in the field. Based on its review of documents and consultations, the evaluation team may choose to look at only the direction of change (positive, negative, none) in these measures rather than attempt to determine the size of the change. In addition to this evidence from the field, the team must also examine whether the theory of change reflects current knowledge.

For both Questions 1 & 2, approaches are defined as the various interventions (also referred to as sub- activities or grants) that TIS partners undertake. These include, but are not limited to community planning sessions, community contracting, in-kind assistance and technical assistance. Each TIS grant was associated with only one of the three TIS objectives (which may include the IOM version used in the first half of the project). To answer this question, the evaluation team will need to develop a categorization framework for types of approaches that fits within the project theory of change, and to evaluate the ways in which grants have been linked to objectives by DAI and IOM. This will inform the analytic plan for assessing any associations between types of approaches and their effectiveness in achieving objectives. The categorization framework will also inform the analysis of how approaches contributed to contributing to USAID objectives and the project goal.

In-kind assistance takes a variety of forms such as construction of stadiums, schools, government offices and Women’s Centers; sport and cultural events; peace dialogues etc. Similarly, technical assistance is not uniform and varies in its duration (short-term or long-term) and type of expertise provided (local or international). Thus, when answering this question, the evaluation team must look at whether certain different types of in-kind and technical assistance are more effective in achieving objectives and goals, and how they contribute to different aspects of stability.

Question 3 is focused on sustainability of the interventions used by TIS. Since TIS aims to focus on both the process and the result, the evaluation team must look at the processes modeled by TIS and the infrastructure constructed by TIS. To look at the sustainability of TIS processes (which can be

A-5

determined by looking at repeat use of the processes among TIS beneficiaries or formal institutionalization of the processes in targeted communities), the evaluation team should look at the entire TIS cycle from the inclusion of entire community in setting priorities and identifying needs, community contracting and contribution, community advocacy with government to provide services needed for intervention, and response by government. For structures constructed by TIS, the evaluation team should look at equitable community ownership, as evidenced by continued use, equal access, and adequate maintenance of these structures, as an indication of sustainability.

TIS uses a community-based development approach in which community planning sessions are the focal point. In order to answer Question 5, the evaluation team will need to examine whether the interventions resulting from community planning sessions: (i) included representatives from the entire community - elders, women and youth – and not a select few; (ii) were based on consensus so that no new conflict was created or existing conflict exacerbated; and (iii) captured the community’s priority and not that of TIS or other stakeholders outside the community.

C.6 METHODS The Contractor must use a mixed-method approach for this evaluation. The Contractor must clearly demonstrate the method(s) that will be used to address each of the evaluation questions. Methods proposed by the Contractor must ensure that respondent and analyst bias is minimized. Any limitations of the methods proposed must be clearly identified. In its proposal, the Contractor must also provide details on the quality assurance measures that will be taken during data collection and analysis, such as pilot testing of instruments, training of enumerators, call backs, and accompaniments, and use of multiple sources for triangulating findings.

In developing its approach, the Contractor must consider past evaluations and studies conducted by USAID. In addition, the Contractor must clearly indicate how monitoring and other research data generated by TIS partners and monitoring and verification data reported by USAID’s third party contractors will be used.

The Contractor must work with USAID to obtain a list of sub-activities / grants in order to develop a sampling framework for the proposal. A staff member of the USAID/KEA/Somalia Office’s Stabilization Team will provide reports from TIS’s grant database, as requested by the Contractor. The Contractor is also expected to consult closely with USAID, DAI and IOM, both during the development of the SOW and the conduct of the evaluation, for the identification of key informants and for the development of work plans.

C.7 OPERATING CONSTRAINTS The level of security in Somalia changes rapidly. The Contractor must ensure that they put into place mechanisms to mitigate risks and ensure that necessary data is collected. This may include, but not limited to, hiring local survey firms, hiring security guards for the team etc.

C.8 OPERATIONAL CONSIDERATIONS The statement of work proposed by the Contractor will be reviewed by the USAID/KEA/Somalia Office and other USAID staff. The Contractor must take these reviews and consequent revisions of the statement of work into account when developing a schedule.

The Contractor must be responsible for arranging and paying for all logistics including, but not limited to lodging, office space, meeting spaces, internet access, equipment, vehicle rentals, security and drivers.

C.9 KEY PERSONNEL While the Contractor is expected to determine the appropriate number and mix of key personnel, the key personnel team, collectively, must have the following qualifications:  Demonstrated experience in planning and conducting mixed-method evaluations  Skills in both quantitative and qualitative data collection in fragile environments, and full analysis of that data

A-6

 Experience with community-based programming in insecure / fragile environments  Understanding of Somali gender issues  Expert knowledge of the past and current status of governance in Somalia  Knowledge of / experience with programming focused on Countering Violent Extremism  Fluency in the Somali language

All key personnel must have the ability and willingness to travel to Somalia. In addition, to the extent possible, the Contractor is highly encouraged to ensure that at least one key personnel is female.

C.10 TASKS Illustrative Due Dates # Task (Note: all days are calendar days) 1 Literature Review Prior to team arriving in Kenya Initial meeting between the TIS COR, DAI, IOM, No later than two days after team arrives in USAID/KEA/Somalia M&E Specialist, other relevant Kenya. 2 USAID personnel, and Contractor evaluation team key personnel to clarify roles and responsibilities, logistical issues, and timelines. A draft Work Plan, which provides a projected No later than five days after #1. timeline and describes in detail the final data collection 3 methods (including draft interview questions and data collection tools) to be used. Meeting with USAID and partner to refine and Before field work begins 4 finalize the work plan and tools In-brief with Ministry of Interior, in particular the Before field work begins 5 Permanent Secretary or a designated member of his staff Pilot test tools and submit a report on pilot Before data collection begins; USAID will 6 testing, including revised tools provide comments on the report no later than three days after submission by Contractor Data Collection Contractor to propose based on scheduling 7 limitations highlighted in this Statement Of Objectives Updates on progress during to date, in person or Weekly 8 via email as agreed at the initial meeting, including any issues or problems encountered Out-brief with Ministry of Interior, in particular the Before departing the field 9 Permanent Secretary or a designated member of his staff Presentation of preliminary findings and No later than 12 days after completion of 9 conclusions of the evaluation to USAID fieldwork Presentation of preliminary findings and No later than 13 days after completion of 10 conclusions of the evaluation to key stakeholders fieldwork Draft Evaluation Report for USAID review. No later than 12 days after presentation of 11 preliminary findings Draft Infographic/One-pager, if requested by No later than 14 days after presentation of 12 USAID preliminary findings USAID and Implementing Partner Comments No later than 12 days after draft evaluation 13 on Draft Report due to Contractor submission Final Evaluation Report, Draft Synthesis No later than seven days after receiving Documents and, if request by USAID, Final comments from USAID 14 Infographic submitted to USAID along with all raw data, as stated above Submission of approved report to the DEC and No later than five days after receiving USAID 15 raw data approval on the final evaluation report Evaluation Synthesis documents submitted to No later than five days after receiving USAID 16 USAID approval on the final evaluation report and comments on the draft Synthesis Documents

C.11 STANDARDS

A-7

The proposed Statement of Work, including methods, and the final evaluation must be consistent with and meet the standards of USAID’s Evaluation Policy (especially Appendix I) and industry best practices. The statement of work should include, as relevant, details listed in the Checklist for Reviewing Statements of Work published in 2012 by USAID’s Office of Learning Evaluation and Research. The checklist is available on the USAID Learning Lab at http://pdf.usaid.gov/pdf_docs/PNADU534.pdf

SECTION D: PACKAGING AND MARKING

D.1. INCORPOARATION OF IDIQ CLAUSES The clauses included in the IDIQ are hereby incorporated by reference.

SECTION E – INSPECTION AND ACCEPTANCE

E.1. INCORPORATION OF IDIQ CLAUSES The clauses included in the IDIQ are hereby incorporated by reference.

SECTION F – DELIVERIES OR PERFORMANCE

F.1. INCORPORATION OF IDIQ CLAUSES The clauses included in the IDIQ are hereby incorporated by reference.

F.2 PERIOD OF PERFORMANCE The proposed period of performance is 6 months from the effective date of the award.

F.3 PLACE OF PERFORMANCE The data collection will be conducted primarily in Somalia, although some data may be collected in Nairobi.

F.4 DELIVERABLES The final evaluation report must conform to the standards set forth in the 2011 USAID Evaluation Policy. If the report contains any potentially procurement sensitive information, a second version report excluding this information must be submitted (also electronically, in English). Given that one of the purposes of the evaluation is to learn lessons for future programming, the evaluation must include a section on lessons learned.

The evaluation report should meet the standards listed in the Checklist for Assessing USAID Evaluation Reports published in 2012 by USAID’s Office of Learning Evaluation and Research. The checklist is available on the USAID Learning Lab at http://usaidlearninglab.org/library/checklist- assessing-usaid-evaluation-reports-0

Two Evaluation Synthesis Documents must also be produced to facilitate wider dissemination of evaluation findings. Examples are provided as attachments of one three and one six-page summary, but at a minimum, every brief should include information on: description of activity being evaluated; evaluation methods used; evaluation questions; key findings; either the success of the main approaches or the validity of the theory of change, as appropriate; and lessons learned (recommendations). Somewhere in the brief, gender and sustainability should also be addressed (if not in the formal evaluation questions/findings, then as separate observations).

All primary source data, both quantitative and qualitative, generated during the course of evaluation shall be provided to USAID in an electronic file in an easily readable format; organized and fully documented for use by those not fully familiar with the activity or the evaluation. In addition, all background documents collected for this evaluation must be provided to USAID on Compact Discs (CDs), along with the final report. Also, per the requirements of ADS 579, all data collected using USAID funds should be posted on the USAID Open data website, in a machine readable format. The Contractor must post the final approved report on the Development Clearinghouse.

A-8

# Deliverables A draft Work Plan, which provides a projected timeline and describes in detail the final data collection 1 methods (including draft interview questions and data collection tools) to be used. 2 Report on the pilot test, and all revised instruments. Presentations (2) of preliminary findings and conclusions of the evaluation to (1) USAID and (2) key 3 stakeholders. 4 Draft Evaluation Report for USAID review and feedback. Draft Infographic / One-page summary, if requested by USAID for USAID review and feedback (the 5 format and content of this deliverable to be finalized in consultation with USAID). Final Evaluation Report & Infographic / One-pager submitted to USAID along with all source data, as 6 well as Draft Summary Documents. Syntheses of final evaluation report, maximum of three different versions (identification of audience and 7 page length for each version to be finalized in consultation with USAID based on evaluation type and scope). Submission of final report on the DEC and data on Open Data – the final report must be posted on 8 the DEC and the raw quantitative data and codebook must be submitted on the USAID Open Data website.

SECTION G – TASK ORDER ADMINISTRATION DATA

G.1. INCORPOARATION OF IDIQ CLAUSES The clauses included in the IDIQ are hereby incorporated by reference

G.2 PAYMENT SCHEDULE Deliverable Payment

Upon successful completion and acceptance by COR of deliverable 1-2 30%

Upon successful completion and acceptance by COR of deliverable 3-6 50%

Upon successful completion and acceptance by COR of deliverable 7-8 20%

SECTION H: SPECIAL TASK ORDER REQUIREMENTS

H.1. INCORPORATION OF IDIQ CLAUSES The clauses included in the basic IDIQ are hereby incorporated by reference

SECTION I: CONTRACT CLAUSES

I.1 INCORPORATION OF IDIQ CLAUSES The clauses included in the basic IDIQ are hereby incorporated by reference

SECTION J: ATTACHMENTS Documents provided below contain background information and data necessary to develop a SOW. These documents must also be provided to the evaluation team by the Contractor for review of relevant documents prior to the evaluation team’s in-brief with USAID.

A-9

Office PMP with Results Framework USAID Somalia PMP USAID-SOMALIA FINAL 20141013.docx IPTT_10142014 .xlsx

2012 Mid-term Review (Complex Crises Funds) Previous Evaluations of this Activity (indicate if none done) CCF Mid-term TIS Review_FINAL.docx

Activity Description from Agreement with Partner and ALL Modifications (For ILs must DAI Revised Section IOM Program include project proposals) C - Mod 7.pdf Description.pdf

Activity M&E Plan with Results Framework IOM M&E Plan _ DAI M&E Plan _ APr 2015.doc May 2015.pdf Latest work plan shared with Contractor Activity Annual Work Plans Available in Contractor Clearinghouse database Activity Performance Reports Available in Contractor Clearinghouse database Site Visit & Other Monitoring Reports DAI’s Report on public confidence and social cohesion measures Partner-initiated Assessment

DAITIS Qual report November 2014 final.docx

Stabilization Survey Other relevant M&E documents http://pdf.usaid.gov/pdf_docs/PA00K6SF.pdf Grant Database Given the size of the database, a USAID staff member will be available to provide reports from the database as needed by the Contractor

Sample executive summaries

3 Page Sample Eval 6-Pg Eval Synthesis Synthesis.pdf Example.docx

A-10

Annex B: Map of Somalia Showing TIS Settlements Surveyed during the Final Evaluation

Map of TIS Settlements

A-11

Annex C: Evaluation Design and Methodology Annex C is presented in the following sub-sections:

1. Data Collection Matrix; 2. Universe Definition, Planned and Actual Sample; 3. Quality Control Methods for Quantitative Survey; 4. Development of Survey Weights for 2016 dataset; 5. Weights for overlapping TIS 2013 and 2015 Survey datasets; 6. Development of Stability Index; 7. Regression Analysis. 8. Activities Conducted in Sectors with High Stability Index

A-12

1. Data Collection Matrix EVALUATION SUB – QUESTION TYPE OF DATA COLLECTION SAMPLING DATA ANALYSIS QUESTION EVIDENCE SOURCE11 TOOL METHOD APPROACH METHOD 1. To what extent 1.1 To what extent Descriptive, Community Guide FGD Purposive Content and have TIS/DAI and did various sub- Analytical and population of 30 qualitative analysis TIS/IOM achieved activities result in Comparative sampled disaggregated by sub their objectives? increased 12 communities. 3 per category, gender, Did the type of confidence in all settlement – Youth, geographic location, approach used levels of Women and Men group characteristics have an government in the per settlement and the stability independent effect communities categories. on the where the sub- achievement of activities took Religious and clan Guide KII Purposive Content and TIS objectives? place? leaders, CBO, CSO qualitative analysis stakeholders disaggregated by sub category, gender, Government Guide KII Purposive geographic location, officials/ relevant group characteristics ministries/actors and the stability categories. Household level Survey Field Census of all The analysis will Sample size of 1500 Question Interviews sub–activities broadly be stratified from all the naire (Systematic for into four geographic settlements household regions and linked to selection the stability Purposive at the categories. (The household) analysis will be disaggregated by gender and age category where possible). 1.2 To what extent Descriptive, Community Guide FGD Purposive Content and did various sub- Analytical and population of 30 qualitative analysis activities result in Comparative sampled disaggregated by sub

11 We will be looking at 30% of the settlements and select purposively 10 individuals per FGD with 3 FGDs per settlement. The KII will have a sample size of 230 distributed through the different categories. We will have a sample size of 1500 for the household surveys. 12 The descriptive analytical and comparative will be done based on the findings of the synthesis report.

A-13

dialogues in the communities. 3 per category, gender, communities settlement – Youth, geographic location, where the sub- Women and Men group characteristics activities took and the stability place? categories. Religious and clan Guide KII Purposive Content and leaders, CBO, IPs, qualitative analysis CSO stakeholders disaggregated by sub category, gender, Government Guide KII Purposive geographic location, officials relevant group characteristics ministries/actors and the stability categories. Household level Survey Field Census of all The analysis will Sample size of 1500 Question Interview sub – activities broadly be stratified from all the naire (Systematic for into four geographic settlements household regions and linked to selection the specific Purposive at the characteristic being household) addressed. (The analysis will be disaggregated by gender and age category where possible). 1.3 To what extent Descriptive, Private sector Guide KII Purposive Content and did various sub- Analytical and representatives, IPs, qualitative analysis activities result in Comparative CSO stakeholders disaggregated by sub increased Government Guide KII Purposive category, gender, collaboration officials relevant geographic location, between ministries/actors group characteristics government, the and the stability private sector, and categories. civil society in the Household level Survey Field Census of all The analysis will communities Sample size of 1500 question Interview sub – activities broadly be stratified where the sub- from all the naire (Systematic for into four geographic activities took settlements household regions and linked to place? selection the specific

A-14

Purposive at the characteristic being household) addressed. (The analysis will be disaggregated by gender and age category where possible).

1.4 To what extent did Descriptive, Religious and clan KII Content and various sub-activities Analytical and leaders, CBO, IPs, Guide Purposive qualitative analysis result in unintended Comparative CSO stakeholders disaggregated by sub results in the category, gender, communities where geographic location, the sub-activities took Government Guide KII Purposive group characteristics place? officials/ relevant and the stability ministries/actors categories.

2. To what extent 2.1 To what extent Descriptive, Community Guide FGD Purposive Content and has the project did various sub- Analytical and population of 30 qualitative analysis contributed to the activities result in Comparative sampled disaggregated by sub goal stated herein improved communities. 3 per category, gender, (Section C.2.) governance in the settlement – Youth, geographic location, and/or to the communities Women and Men group characteristics achievement of where the sub- per settlement and the stability USAID objectives? activities took categories. Were any TIS place? TIS staff, Religious Guide KII Purposive Content and approaches and clan leaders, qualitative analysis especially effective CBO, IPs, CSO disaggregated by sub in contributing to stakeholders category, gender, stability, as defined geographic location, in Section C.2 or Government Guide KII Purposive group characteristics to the officials relevant and the stability achievement of ministries/actors categories. USAID objectives? Household level Survey Field Census of all The analysis will Sample size of 1500 Question Interview sub – activities broadly be stratified from all the naire (Systematic for into four geographic settlements household regions and linked to selection the stability

A-15

Purposive at the categories. (The household) analysis will be disaggregated by gender and age category where possible). 2.2 To what extent Descriptive, Community Guide FGD Purposive Content and did various sub- Analytical and population of 30 qualitative analysis activities result in Comparative sampled disaggregated by sub economic communities. 3 per category, gender, recovery in the settlement – Youth, geographic location, communities Women and Men group characteristics where the sub- and the stability activities took categories. place? TIS staff, Religious Guide KII Purposive Content and and clan leaders, qualitative analysis CBO, IPs, CSO disaggregated by sub stakeholders category, gender, geographic location, Government Guide KII Purposive group characteristics officials relevant and the stability ministries/actors categories. Household level Survey Field Census of all The analysis will Sample size of 1500 Question Interview sub – activities broadly be stratified from all the naire (Systematic for into four geographic settlements household regions and linked to selection the stability Purposive at the categories. (The household) analysis will be disaggregated by gender and age category where possible) 2.3 To what extent Descriptive, Community Guide FGD Purposive Content and did various sub- Analytical and population of 30 qualitative analysis activities result in Comparative sampled disaggregated by sub a reduction in the communities. 3 per category, gender, appeal of settlement – Youth, geographic location,

A-16

extremism in the Women and Men group characteristics communities in and the stability which the sub- categories. activities took Religious and clan Guide KII Purposive Content and place? leaders, CBO, IPs, qualitative analysis CSO stakeholders disaggregated by sub category, gender, Government Guide KII Purposive geographic location, officials/ relevant group characteristics ministries/actors and the stability categories. Household level Survey Field Census of all The analysis will Sample size of 1500 Question Interview sub – activities broadly be stratified from all the naire (Systematic for into four geographic settlements household regions and linked to selection the specific Purposive at the characteristic being household) addressed. (The analysis will be disaggregated by gender and age category where possible). 2.4 To what extent Descriptive, Community Guide FGD Purposive Content and did various sub- Analytical and population of 30 qualitative analysis activities result in Comparative sampled disaggregated by sub improvement in communities. 3 per category, gender, the factors of settlement – Youth, geographic location, stability in the Women and Men group characteristics communities in per settlement and the stability which the sub- categories. activities took place? Religious and clan Guide KII Purposive Content and leaders, CBO, IPs, qualitative analysis CSO stakeholders disaggregated by sub District level Guide KII Purposive category, gender, relevant ministries geographic location, group characteristics

A-17

and the stability categories. Household level Survey Field Census of all The analysis will Sample size of 1500 Question Interview sub – activities broadly be stratified from all the naire (Systematic for into four geographic settlements household regions and linked to selection and the specific random within characteristic being household) addressed. (The analysis will be disaggregated by gender and age category where possible). 3. To what extent 3.1 In the Descriptive, Community Guide FGD Purposive Content and did the processes communities Analytical and population of 30 qualitative analysis modelled by TIS where sub- Comparative sampled disaggregated by sub become activities were communities. 3 per category, gender, institutionalized, implemented, to settlement – Youth, geographic location, and did physical what extent does Women and Men group characteristics structures governance have and the stability constructed by TIS the features of the categories. continue to be TIS cycle, from the TIS staff, Religious Guide KII Purposive used by inclusion of entire and clan leaders, communities? community in CBO, IPs, CSO setting priorities stakeholders and identifying Content and needs, community qualitative analysis contracting and disaggregated by sub contribution, category, gender, community geographic location, advocacy with Government Guide KII Purposive group characteristics government to officials relevant and the stability provide services ministries/actors categories. needed for intervention, and response by government?

A-18

3.2 To what extent Descriptive, Direct observation Observat Site visits Purposive Content analysis of are the physical Analytical and ion data from the direct structures Comparative checklist/ observations and constructed by TIS photos photos/videos data in continued use from the checklist. under conditions Community Guide FGD Purposive Content and of equitable population of 30 qualitative analysis community sampled disaggregated by sub ownership, equal communities – category, gender, access, and Youth, Women and geographic location, adequate Men group characteristics maintenance? and the stability categories. TIS staff, Religious Guide KII Purposive Content and and clan leaders, qualitative analysis CBO, IPs, CSO disaggregated by sub stakeholders category, gender, Government Guide KII Purposive geographic location, officials relevant group characteristics ministries/actors and the stability categories. 4. How successful 4.1 To what extent Descriptive, Community Guide FGD Purposive Content and was the project in did various sub- Analytical and population of 30 qualitative analysis addressing gender activities result in Comparative sampled disaggregated by sub equity and increased inclusion communities – category, gender, empowerment of women in Youth, Women and geographic location, issues, such as community-level Men group characteristics increasing decision making in and the stability inclusion of the communities categories. women in where the sub- TIS staff, Religious Guide KII Purposive Content and community-level activities took and clan leaders, qualitative analysis decision making, place? CBO, IPs, CSO disaggregated by sub promoting stakeholders category, gender, representation of Government Guide KII Purposive geographic location, women’s interests, officials/ relevant group characteristics enabling women ministries/actors and the stability to advocate for categories.

A-19

themselves and 4.2 To what extent Descriptive, Community Guide FGD Purposive Content and take action, and did various sub- Analytical and population of 30 qualitative analysis ensuring both men activities result in Comparative sampled disaggregated by sub and women increased respect communities. 3 per category, gender, benefit from the for women’s settlement – Youth, geographic location, activity? interests in the Women and Men group characteristics communities and the stability where the sub- categories. activities took TIS staff, Religious Guide KII Purposive Content and place? and clan leaders, qualitative analysis CBO, IPs, CSO disaggregated by sub stakeholders category, gender, Government Guide KII Purposive geographic location, officials/ relevant group characteristics ministries/actors and the stability categories. 4.3 To what extent Descriptive, Community Guide FGD Purposive Content and did various sub- Analytical and population of 30 qualitative analysis activities result in Comparative sampled disaggregated by sub an increased ability communities – category, gender, of women to Youth, Women and geographic location, advocate for or Men group characteristics otherwise take and the stability action in pursuit of categories. their interests in IPs, TIS staff, CSO, Guide KII Purposive Content and the communities Women and Youth qualitative analysis where the sub- Stakeholders disaggregated by sub activities took Government Guide KII Purposive category, gender, place? officials/ relevant geographic location, ministries/actors group characteristics and the stability categories. Household level Survey Field Census of all The analysis will Sample size of 1500 Question Interview sub – activities broadly be stratified from all the naire (Systematic for into four geographic settlements household regions and linked to selection the specific Purposive at the characteristic being

A-20

household) addressed. (The analysis will be disaggregated by gender and age category where possible). Descriptive, Community Guide FGD Purposive Content and 4.4 To what extent Analytical and population of 30 qualitative analysis did various sub- Comparative sampled disaggregated by sub activities result in communities. 3 per category, gender, benefits to both settlement – Youth, geographic location, men and women Women and Men group characteristics in the and the stability communities categories. where the sub- IPs, TIS staff, CSO, Guide KII Purposive Content and activities took Women and Youth qualitative analysis place? To what Stakeholders disaggregated by sub extent did various Government Guide KII Purposive category, gender, sub-activities officials/ relevant geographic location, result in benefits ministries/actors group characteristics for both men and and the stability women in the categories. communities Household level Survey Field Census of all The analysis will where the sub- Sample size of 1500 Question Interview sub – activities broadly be stratified activities took from all the naire (Systematic for into four geographic place? settlements household regions and linked to selection the specific Purposive at the characteristic being household) addressed. (The analysis will be disaggregated by gender and age category where possible). 5. To what extent 5.1 To what extent Descriptive, Community Guide FGD Purposive Content and were CPSs did CPSs include Analytical and population of 30 qualitative analysis effective in representatives Comparative sampled disaggregated by sub producing from the target communities. 3 per category, gender,

A-21

decisions that are community - settlement – Youth, geographic location, inclusive, elders, women and Women and Men group characteristics consensus-based, youth – and not a and the stability and reflective of select few? categories. community IPs, TIS staff, CSO, Guide KII Purposive Content and priorities? Women and Youth qualitative analysis Stakeholders disaggregated by sub Government Guide KII Purposive category, gender, officials/ relevant geographic location, ministries/actors group characteristics and the stability categories. Household level Survey Field Census of all The analysis will Sample size of 1500 Question Interview sub – activities broadly be stratified from all the naire (Systematic for into four geographic settlements household regions and linked to selection the specific Purposive at the characteristic being household) addressed. (The analysis will be disaggregated by gender and age category where possible). 5.2 To what extent Descriptive, Community Guide FGD and KII Purposive Content and did CPSs mitigate Analytical and population of 30 qualitative analysis or manage new Comparative sampled disaggregated by sub and existing communities. 3 per category, gender, conflict by the settlement – Youth, geographic location, implementation of Women and Men group characteristics sub-activities? and the stability categories. IPs, TIS staff, CSO, Guide KII Purposive Content and Women and Youth qualitative analysis Stakeholders disaggregated by sub Government Guide KII Purposive category, gender, officials/ relevant geographic location, ministries/actors group characteristics

A-22

and the stability categories. Household level Survey Field Census of all The analysis will Sample size of 1500 Question Interview sub – activities broadly be stratified from all the naire (Systematic for into four geographic settlements household regions and linked to selection the specific Purposive at the characteristic being household) addressed. (The analysis will be disaggregated by gender and age category where possible). 5.3 To what extent Descriptive, Community Guide FGD and KII Purposive Content and did CPSs result in sub- Analytical and population of 30 qualitative analysis activities that reflected Comparative sampled disaggregated by sub the community’s communities. 3 per category, gender, priority and not that of settlement – Youth, geographic location, TIS or other Women and Men group characteristics stakeholders outside and the stability the community? categories. IPs, TIS staff, CSO, Guide KII Purposive Content and Women and Youth qualitative analysis Stakeholders disaggregated by sub Government Guide KII Purposive category, gender, officials/ relevant geographic location, ministries/actors group characteristics and the stability categories. Household level Survey Field Census of all The analysis will Sample size of 1500 Question Interview sub – activities broadly be stratified from all the naire (Systematic for into four geographic settlements household regions and linked to selection the specific Purposive at the characteristic being household) addressed. (The analysis will be

A-23

disaggregated by gender and age category where possible).

A-24

2. Universe Definition, Planned and Achieved Sample Size Allocation

2.1 Universe Definition In Somalia the last population census was conducted in 1987 – and this is due to the collapse of the Federal Government of Somalia in 1989 and the subsequent conflicts that followed. To obtain current population data, the United Nations Population Fund (UNFPA)13 have given best population estimates based on a set of methodologies. The UNFPA have documented data on enumeration areas – and each enumeration area is approximated to have between 50 and 150 households. In order to estimate population data, UNFPA have multiplied the number of households by the average number of occupants per household (which varied per region). This has enabled population estimation at each of the 18 regional levels of Somalia (see Table 1). Table 1: Somalia Regional Population Estimates Geographic Area Region Population size Somaliland Awdal 673,263 Woqooyi Galbeed 1,242,003 Several districts in these Togdheer 721,363 regions are contested by Sool 327,428 Somaliland and Puntland Sanaag 544,123 Puntland Bari 719,512 Nugaal 392,698 Emerging Federal States Galguduud 569,434 Hiraan 520,685 Middle Shabelle 516,036 Banadir 1,650,227 Lower Shabelle 1,2002,219 Bay 792,182 Bakool 367.226 Gedo 508,405 Middle Juba 362,921 Lower Juba 489,307 Total 18 12,316,895

2.2 Sample Size Allocation The original calculated sample size was N = 1,476. Therefore, an additional 10% oversampling (147 respondents) were to be interviewed to ensure that the poll reached its target sample size (thus total sample size will be N = 1623), given the 10% rate of non-response typically observed in household surveys in Somalia. The margin of error associated with an N = 1,476 simple random sample of populations larger than 1,000,000 equals +/- 2.5% at the 95% confidence interval. The “design effect” created by the stratification of individual responses in communities may add an additional 2.5 percentage points to the margin of error depending on the survey question. The total margin of error associated with the survey is thus approximately +/- 5%. Allocation of the 1623 sample size proceeded as follows; first, allocation probabilities were calculated at the regional level (Table 2). Allocation probability defines the proportion of individuals to be interviewed at the household level in a given region. Next, settlements were allocated similar sample sizes within every region (Table 3). Table 2: Allocation Probabilities by Region

Geographic Area Region Population Allocation Sample Total size Probability Size Sample Size Somaliland Awdal 673,263 0.06 97 486 Woqooyi 1,242,003 0.10 162 Galbeed Several districts in these Togdheer 721,363 0.06 97

13 http://countryoffice.unfpa.org/somalia/drive/Population-Estimation-Survey-of-Somalia-PESS-2013-2014.pdf

A - 25

regions are contested by Sool 327,428 0.03 49 Somaliland and Puntland Sanaag 544,123 0.05 81 Puntland Bari 719,512 0.06 97 243 Nugaal 392,698 0.03 49 Mudug 717,863 0.06 97 South Central Galguduud 569,434 0.05 81 894 Hiraan 520,685 0.04 65 Middle 516,036 0.04 65 Shabelle Banadir 1,650,227 0.14 228 Lower 1,202,219 0.10 162 Shabelle Bay 792,182 0.07 114 Bakool 367,226 0.03 49 Gedo 508,405 0.04 65 Lower Juba 489,307 0.04 65 Total 17 11,953,974 1.00 1623 1623

The following points summarize the key characteristics of (other) aspects of the survey:  Sample Universe: The adult population of settlements where TIS activities were implemented.  Sampling Frame: The 102 settlements identified from the TIS sub-activity database.  Target Population: Adult members of TIS target settlements between 18 years and 60 years.  Stratification: o 50/50 Male/Female: The plan was to conduct 812 male and 811 female interviews.  Unit of Observation: One individual selected randomly from the members of each household that was systematically selected at each sampling point.  Household Selection Method: Random walk systematic selection. The sampling plan specified a systematically selected starting location, walking direction, and the first household to be contacted for an interview in each settlement. After an interview was obtained from the first household, the male interviewer selected every fifth household on the right hand side and the female interviewer selected every third household on the left hand side along the walk in rural areas, and every tenth household in urban areas.  Respondent Selection Method: Random selection using the two-step Kish Grid procedure: First all eligible members of the household (between 18 years and 60 years old and either male or female depending on the sex of the interviewer) were listed. Then the random numbers in the grid cells were used to randomly select the respondent from the list.  Data Collection Methods: Both Mobile Data Collection (MDC) and Paper & Pencil Interview (PAPI) methods were used14.

2.3 Sample Size Accomplished A total of 1623 interviews were planned but the overall exercise achieved 1719 – which was in excess by 96. These excess interviews were conducted to account for any that would be replaced as a result of not meeting data quality thresholds. Table 3 below shows the distribution of sample size planned versus the sample size accomplished per settlement. Table 3: Data Collection Summary

14 Settlements where PAPI was used are highlighted in green in table 3

A - 26

Household Survey Geographic Planned Actual interviews Region District Settlement Area Sample accomplished by Size 31/01/2016 Awdal Baki Baki 17 16 Awdal Borama Borama 17 18 Awdal Zeylac Zeylac 17 17 Woqooyi Hargeisa Hargeisa 17 Galbeed 26 Woqooyi Hargeisa Axmed Dhagax 17 Galbeed 17 Woqooyi Ibraahim Hargeisa 17 Galbeed Koodbuur 17 Woqooyi Hargeisa Gacan Libah 16 Galbeed 16 Woqooyi Hargeisa Koodbuur 16 Galbeed 16 Woqooyi Koodbuur/Guul Hargeisa 16 Galbeed Ala 16 Woqooyi Hargeisa Koodbuur/Lixle 16 Galbeed 16 Somaliland Maxamud Woqooyi Hargeisa Haybe/Qudhac 16 Galbeed Dheer 13 Woqooyi Hargeisa Balli-Gubadle 16 Galbeed 16 Woqooyi Berbera Berbera 16 Galbeed 15 Woqooyi Gebiley Gebiley 16 Galbeed 16 Woqooyi Gebiley Kala Baydh 16 Galbeed 16 Togdheer Burco Burco 16 13 Togdheer Burco Beer One 16 16 Togdheer Owdweyne Odweyne 16 19 Owdweyne/Barw Togdheer Owdweyne 16 aaqo 14 Togdheer Sheikh Sheikh 16 16 Contested by Sool Laas Caanood Laas Caanood 16 17 Somaliland Sool Laas Caanood Boocame 16 14 and Puntland Sool Caynabo Caynabo 16 16 Sanaag Ceerigaabo Ceerigaabo 16 17 Sanaag Ceerigaabo Ardaa 16 16 Sanaag Ceerigaabo Maydh 16 16 Sanaag Badhan Badhan 16 16 Sanaag Ceel Afweyn Ceel Afweyn 16 15 Sanaag Ceel Afweyn Gar Adag 16 16 Sanaag Laasqoray Dhahar 16 16 Puntland Bari Bossaso Bossaso 12 28 Bari Iskushuban Baargaal 11 14 Bari Iskushuban Masuadde 11 Bari Iskushuban Muudiye 11 15 Bari Iskushuban Qorraxaad 11 13 Bari Iskushuban Tager 11 13 Bari Iskushuban Xaafuun 11 13 Bari Bandarbayla Bandarbayla 11 11 Bari Bandarbayla Bixin 11 12 Bari Bandarbayla Kulule 11 10

A - 27

Bari Bandarbayla Dhuur 11 11 Bari Bandarbayla Qoton 11 Bari Qardho Qardho 11 9 Nugaal Garowe Garowe 11 15 Nugaal Garowe Garowe/Waberi 11 11 Nugaal Garowe Garowe/Wadajir 11 19 Nugaal Burtinle Burtinle 11 11 Nugaal Eyl Eyl 11 13 Mudug Gaalkacyo Gaalkacyo 11 11 Mudug Galdogob Galdogob 11 11 Mudug Jariban Garcad 11 11

Mudug Jariban Jariiban 11 11 Galgaduud Cabudwaaq Cabudwaaq 19 22 Galgaduud Cabudwaaq Balanbal 19 23 Galgaduud Cabudwaaq Xerale 19 27 Galgaduud Cadaado Cadaado 19 24 Dhuusamarre Galgaduud Dhuusamarreeb 19 eb 29 Dhuusamarre Galgaduud Guri-Ceel 19 eb 18 Hiraan Belet Weyne Belet Weyne 19 24 Hiraan Belet Weyne Mataban 19 18 Hiraan Bulo Burto Maxaas 19 30 Middle Shabelle Balcad Balcad 19 24 Middle Shabelle Cadale Cadale 19 18 Middle Shabelle Jowhar Jowhar 19 24 Middle Shabelle Balcad Warshiikh 19 18 Banadir Boondheere Boondheere 19 19 Banadir Cabdlcasiis Cabdulcasiis 19 21 Banadir Daynile Daynile 19 22 Banadir Dharkenley Dharkenley 19 20 Banadir Hawl Wadaag Hawl Wadaag 19 20 Banadir Heliwa Heliwa 19 20 Banadir Hodan Hodan 19 21 Emerging Banadir Karaan Karaan 19 21 Federal States Banadir Shangaani Shangaani 19 21 Banadir Shibis Shibis 19 30 Banadir Waaberi Waaberi 19 31 Banadir Wadajir Wadajir 19 21 Wardhiigleey Banadir Wardhiigleey 19 (Warta Nabada) 31 Banadir Xamar JaabJab Xamar Jaabjab 19 21 Xamar Banadir Xamar Weyne 19 Weyne 33 Banadir Yaaqshiid Yaaqshiid 19 21 Banadir Kaxda Kaxda 19 21 Lower Shabelle Afgooye Afgooye 18 34 Lower Shabelle Barawe Barawe 18 24 Lower Shabelle Qoryooley Qoryooley 24 Bay Baidoa Baidoa 18 32 Bay Baidoa Berdale 18 (PAPI) 12 Bakool Xudur Xudur 18 (PAPI) 18 Gedo Belet Xaawo Belet Xaawo 18 24 Gedo Ceel Waaq Ceel Waaq 18 Gedo Doolow Doolow 18 24 Gedo Baardheere Faafax Dhuun 18 (PAPI) 18 Gedo Garbahaarey Garbahaarey 18

A - 28

Gedo Luuq Luuq 18 24 Lower Juba Kismayo Kismayo 18 24 Afmadow (only planning Lower Juba Afmadow 18 processes grant was implemented) Lower Juba Afmadow Dhobley 18 Lower Juba Afmadow Dif 18 (PAPI) 18 Lower Juba Badhaadhe Badhaadhe 18 All 18 collected Lower Juba Badhaadhe Kolbio 18 through PAPI but of poor quality Lower Juba Badhaadhe Raskaamboni 18 Total 1623 1719

2.4 Classification of settlements based on accessibility The settlements were classified based on ease of accessibility as follows:

 Settlements where interviewers used MDC were considered very accessible;  Settlements where interviewers used PAPI were considered moderately accessible. PAPI method was used in these settlements since the use of MDC would draw attention – thus risking the lives of interviewers;  Settlements where no data were collected were considered completely inaccessible due to insecurity. Key Accessibility No data collected Completely inaccessible (PAPI) Data collected with paper and pencil Moderately accessible Data collected using PAPI but of poor quality Moderately accessible Achieved Sample less than expected Very Accessible Achieved (or exceeded) expected sample size Very Accessible

 No data were collected in Masuadde settlement as there were only three households and all indicated they did not see any TIS – DAI/ TIS – IOM activities.  No data were collected in Qoton since there was a major fight between clans thus was unsafe for the interviewers.  There was a heavy Al Shabaab presence in the routes leading to Ceel Waaq, Garbahaarey, Dhobley, Badhaadhe and Raskaamboni settlements. Travelling to these places was totally unsafe so therefore no data were collected. See table 4 for the distribution of grants in the settlements where data were not obtainable. Table 4: Grant information for settlements where data were not collected

Specific Project Sector Number of Total Region District Settlement (Sub-Category) Grants Grants Community infrastructure 2 Education 2 Economic 1 development/Livelihoods Lower Afmadow Dhobley Planning processes 2 12 Juba Capacity Building 2 Communication and 2 coordination Institution Strengthening 1 Badhaadhe Badhaadhe Planning processes 1 1

A - 29

Planning Processes 2 Education 1 Raskaamboni 5 Economic 2 development/Livelihoods Planning processes 1 Community infrastructure 1 Economic Kolbio 1 4 development/Livelihoods Communication and 1 coordination Community infrastructure 1 10 Health and Hygiene 1 Economic 2 Development/Livelihood Ceel Waaq Ceel Waaq Community Security and safety 1 Planning Processes 3 Gedo Capacity Building 1 Peace Process Support 1 Planning Processes 2 6 Community infrastructure 1 Garbahaarey Garbahaarey Institution Strengthening 1 Education 1 Health and Hygiene 1 Conflict Mitigation 2 Institution Strengthening 2 Hiraan Jalalaqsi Jalalaqsi 8 Economic Development 2 Infrastructure 2 Bari Iskushuban Masuadde Infrastructure 1 1 Economic Bari Bandarbayla Qoton 1 1 Development/Livelihood Total 9 48 48

Note: Data that were not obtainable from these settlements were redistributed to other settlements within the same Geographic area with similar nature of grants. While this redistribution ensured the overall sample size, it is noted that this measure had the potential to introduce Type I error, by failing to collect data from inaccessible communities. The likelihood of a Type 1 error is small, however, since most of the inaccessible communities were inaccessible because of security issues affecting access routes rather than the conditions in the communities themselves. Thus, these omitted communities may not have been unstable compared with the surveyed communities.

A - 30

3. Quality Control Methods for Quantitative Survey

3.1 Mobile Data Collection Quantitative data were collected using tablets15 based on the Dooblo16 platform. The collected data were hosted on a cloud server17 – and data access rights granted (by the data collection contractor) to the relevant TIS evaluation team members. This document describes two levels of data quality strategies that were used on these data. Section I describes level 1 strategies – that were enforced at the questionnaire transcription18 and during the actual field execution of the quantitative survey. Section II describes level II strategies that were used at the database level – where data from the field were consolidated. Level I strategies included:

 Programming the scripted questionnaire (on the Dooblo platform) to incorporate preliminary data cleaning procedures. These included rules to disallow entry of out-of-range values (outliers) as well as checking completeness of all relevant sections of the questionnaire and adherence to skip patterns where applicable.  Physical supervision of field interviewers: The data collection contractor had four interviewers managed by one supervisor. Data were collected by the interviewers and checked by the field supervisors – who in turn submitted (synchronised) these data to the central database (figure 1).

15 Tablet specification: Network (GSM/HSPA/LTE), 2G bands (GSM 850/900/1800/1900-SM-T211,SM-T215), 3G (HSDPA 850/900/1800/1900/2100-SM-T211,SM-T215), 4 GB bands (LTE band 3(1800),7(2600)-SM-T215, Speed (HSPA 21.1/5.76 Mbps, LTE Cat3 100/50 Mbps (SM-T215), GPRS (Yes), EDGE (Yes), Body (Dimensions: 188x111.1x9.9 mm (7.40x4.37x0.39 in), weight: 306 g ), Size (7 inches), platform (OS: Android OS, v4.1.2), chipset (marvell PXA986), CPU (Dual-core 1.2 GHz Cortex-A9),GPU (PowerVR SGX540), card slot (microSD, up to 64 GB), Internal (8/16 GB, 1GB RAM),Battery (up to 8 h – talk time, up to 108 music play). 16 Dooblo is a provider of survey data collection software for android mobile phones, tablets and PDAs. 17 Cloud server is a logical server that is built, hosted and delivered through a cloud-computing platform over the Internet. 18 Converting the paper questionnaire into mobile platform.

A - 31

Figure 1: Level 1 data quality checks

Level II strategy Aimed to track errors19 beyond level I. These involved the use of syntaxes written in R20 that automated generation of error reports by the database manager (contracted by the data collection contractor). This process allowed for real time data interrogation for quick quality checks by the database manager – and any detectable discrepancies were addressed where possible. For instance interviews that were less than 30 minutes were repeated. In particular, the automated set of syntaxes were developed for this progress report – which contained:

 Number of records (questionnaires) collected by respondents’ gender, age and geographic area.  Total number of records per interviewer, mean, median, minimum and maximum interview duration.  Overall coverage (distribution of respondents): GPS coordinates obtained from the interviews were useful in mapping the respondent locations.  Measure of data reliability and the following questions were used to check whether data collected were reliable.

Table 1: Questions used to gauge awareness of respondents Question Q1 Have you or anyone else in this household received vocational training supported by the TIS/DAI/IOM project? TIS project is a development program implemented by two international agencies known as by DAI & IOM Q2 Have you or anyone else in this household ever received any support or skills training from TIS/DAI/IOM to help start a business? Q3 Have there been any improvements made in this settlement by TIS/DAI/IOM activities over the past two years that have helped you to earn income to support your family?

3.2 PAPI The data that were collected using paper and pencil were later entered into mobile platforms. DARS instructed the interviewers to record names and phone numbers of respondents – then supervisors would confirm validity of any inconsistent data by calling back the respondents. 3.3 Data Quality Results Table 2: Summary of Data by Sex, Age Category and Geographic Area

Sex Age Category (Years) Geographic Area Total South21 Male Female 18 – 29 30 - 45 45+ Somaliland Puntland (by date) Central Dec-24 31 (48) 33 (52) 33 (52) 22 (34) 9 (14) 0 (0) 16 (25) 48 (75) 64 Dec-26 1 (100) 0 (0) 1 (100) 0 (0) 0 (0) 0 (0) 0 (0) 1 (100) 1 Dec-30 56 (51) 54 (49) 47 (43) 44 (40) 19 (17) 15 (14) 28 (25) 67 (61) 110 Dec-31 71 (53) 64 (47) 71 (53) 48 (36) 16 (12) 38 (28) 23 (17) 74 (55) 135 Feb-01 51 (59) 36 (41) 46 (53) 31 (36) 10 (11) 14 (16) 11 (13) 62 (71) 87 Feb-02 22 (44) 28 (56) 24 (48) 15 (30) 11 (22) 31 (62) 19 (38) 0 (0) 50 Feb-03 64 (45) 79 (55) 83 (58) 47 (33) 13 (9) 46 (32) 17 (12) 80 (56) 143 Feb-04 60 (51) 57 (49) 51 (44) 39 (33) 27 (23) 44 (38) 11 (9) 62 (53) 117 Feb-05 44 (50) 44 (50) 44 (50) 30 (34) 14 (16) 37 (42) 11 (12) 40 (45) 88 Feb-06 42 (52) 39 (48) 30 (37) 34 (42) 17 (21) 39 (48) 24 (30) 18 (22) 81 Feb-07 49 (58) 36 (42) 43 (51) 31 (36) 11 (13) 48 (56) 13 (15) 24 (28) 85 Jan-01 13 (54) 11 (46) 11 (46) 11 (46) 2 (8) 0 (0) 24 (100) 0 (0) 24

19 Incompleteness of data, out of range values and inconsistencies in skip patterns. 20 Open source statistical programming language. 21 South Central here refers to emerging federal states

A - 32

Jan-02 26 (46) 30 (54) 20 (36) 26 (46) 10 (18) 32 (57) 24 (43) 0 (0) 56 Jan-03 34 (49) 35 (51) 34 (49) 27 (39) 8 (12) 16 (23) 15 (22) 38 (55) 69 Jan-04 34 (58) 25 (42) 24 (41) 25 (42) 10 (17) 16 (27) 12 (20) 31 (53) 59 Jan-05 44 (58) 32 (42) 29 (38) 33 (43) 14 (18) 20 (26) 0 (0) 56 (74) 76 Jan-06 19 (48) 21 (52) 20 (50) 15 (38) 5 (12) 12 (30) 0 (0) 28 (70) 40 Jan-07 21 (44) 27 (56) 21 (44) 18 (38) 9 (19) 0 (0) 0 (0) 48 (100) 48 Jan-08 13 (50) 13 (50) 13 (50) 10 (38) 3 (12) 0 (0) 0 (0) 26 (100) 26 Jan-09 25 (62) 15 (38) 11 (28) 17 (42) 12 (30) 0 (0) 0 (0) 40 (100) 40 Jan-10 20 (69) 9 (31) 17 (59) 9 (31) 3 (10) 0 (0) 0 (0) 29 (100) 29 Jan-11 26 (44) 33 (56) 23 (39) 22 (37) 14 (24) 0 (0) 0 (0) 59 (100) 59 Jan-12 11 (41) 16 (59) 12 (44) 11 (41) 4 (15) 0 (0) 0 (0) 27 (100) 27 Jan-13 3 (75) 1 (25) 3 (75) 1 (25) 0 (0) 0 (0) 0 (0) 4 (100) 4 Jan-14 3 (60) 2 (40) 1 (20) 3 (60) 1 (20) 0 (0) 0 (0) 5 (100) 5 Jan-15 16 (48) 17 (52) 12 (36) 14 (42) 7 (21) 33 (100) 0 (0) 0 (0) 33 Jan-16 9 (53) 8 (47) 8 (47) 6 (35) 3 (18) 17 (100) 0 (0) 0 (0) 17 Jan-17 8 (47) 9 (53) 8 (47) 6 (35) 3 (18) 17 (100) 0 (0) 0 (0) 17 Jan-18 0 (0) 7 (100) 2 (29) 5 (71) 0 (0) 0 (0) 4 (57) 3 (43) 7 Jan-19 18 (64) 10 (36) 8 (29) 11 (39) 9 (32) 0 (0) 10 (36) 18 (64) 28 Jan-20 4 (67) 2 (33) 2 (33) 2 (33) 2 (33) 0 (0) 0 (0) 6 (100) 6 Jan-21 8 (53) 7 (47) 3 (20) 4 (27) 8 (53) 0 (0) 0 (0) 15 (100) 15 Jan-25 11 (69) 5 (31) 6 (38) 8 (50) 2 (12) 13 (81) 0 (0) 3 (19) 16 Jan-26 1 (50) 1 (50) 1 (50) 1 (50) 0 (0) 0 (0) 0 (0) 2 (100) 2 Jan-27 1 (50) 1 (50) 1 (50) 1 (50) 0 (0) 0 (0) 0 (0) 2 (100) 2 Jan-29 6 (55) 5 (45) 6 (55) 5 (45) 0 (0) 0 (0) 0 (0) 11 (100) 11 Jan-30 13 (54) 11 (46) 15 (62) 8 (33) 1 (4) 0 (0) 0 (0) 24 (100) 24 Jan-31 11 (61) 7 (39) 10 (56) 5 (28) 3 (17) 0 (0) 0 (0) 18 (100) 18 Total 889 (52) 830 (48) 794 (46) 645 (38) 280 (16) 488 (28) 262 (15) 969 (56) 1719

Table 3: Summary per interviewer

Total Mean Median Minimum Maximum Interviewer code GPS (%) records (min) (min) (min) (min) nova1 64 59 51 44 251 100 nova10 56 429 286 54 1431 100 nova11 24 52 49 32 123 100 nova12 29 55 51 33 76 100 nova13 32 68 51 34 556 100 nova14 28 182 110 43 564 100 nova15 74 129 59 34 1009 100 nova16 75 97 54 36 486 100 nova2 63 77 58 31 394 100 nova3 63 58 54 32 306 100 nova4 61 53 50 31 132 100 nova5 33 111 53 38 600 100 nova501 39 37 35 12 103 100 nova502 35 88 65 38 315 100 nova503 48 96 71 46 335 100 nova504 56 91 69 37 330 100 nova505 45 67 57 40 235 100 nova506 10 52 52 37 75 100 nova507 43 56 54 39 79 100 nova508 68 56 55 31 102 100 nova509 40 58 53 37 132 100 nova510 53 67 58 35 358 100 nova512 56 61 58 32 100 100 nova513 46 62 60 36 95 100 nova514 26 62 60 35 170 100 nova515 45 62 63 39 92 100 nova517 64 61 58 33 113 100 nova518 21 62 59 43 85 100 nova519 29 58 56 35 83 100

A - 33

nova520 39 69 62 38 336 100 nova521 36 58 57 43 96 100 nova522 55 60 60 36 151 100 nova523 44 63 58 43 122 100 nova524 29 58 56 47 88 100 nova525 42 61 58 48 106 100 nova6 31 117 52 36 771 100 nova7 32 154 58 39 1380 100 nova8 29 87 58 46 762 100 nova9 56 362 214 39 1385 100 Average 44 91 68 37 357 100 Note: Average time interval was also tracked for each interviewer. Table 4: Summary by interview time intervals

Between 30 and At least 120 Interviewer code < 30 minutes Total 120 minutes minutes nova1 0(0) 61(95) 3(5) 64 nova10 0(0) 20(36) 36(64) 56 nova11 0(0) 23(96) 1(4) 24 nova12 0(0) 29(100) 0(0) 29 nova13 0(0) 31(97) 1(3) 32 nova14 0(0) 16(57) 12(43) 28 nova15 0(0) 55(74) 19(26) 74 nova16 0(0) 60(80) 15(20) 75 nova2 0(0) 58(92) 5(8) 63 nova3 0(0) 62(98) 1(2) 63 nova4 0(0) 60(98) 1(2) 61 nova5 0(0) 29(88) 4(12) 33 nova501 16(41) 23(59) 0(0) 39 nova502 0(0) 29(83) 6(17) 35 nova503 0(0) 37(77) 11(23) 48 nova504 0(0) 49(88) 7(12) 56 nova505 0(0) 43(96) 2(4) 45 nova506 0(0) 10(100) 0(0) 10 nova507 0(0) 43(100) 0(0) 43 nova508 0(0) 68(100) 0(0) 68 nova509 0(0) 39(98) 1(2) 40 nova510 0(0) 51(96) 2(4) 53 nova512 0(0) 56(100) 0(0) 56 nova513 0(0) 46(100) 0(0) 46 nova514 0(0) 25(96) 1(4) 26 nova515 0(0) 45(100) 0(0) 45 nova517 0(0) 64(100) 0(0) 64 nova518 0(0) 21(100) 0(0) 21 nova519 0(0) 29(100) 0(0) 29 nova520 0(0) 38(97) 1(3) 39 nova521 0(0) 36(100) 0(0) 36 nova522 0(0) 54(98) 1(2) 55 nova523 0(0) 43(98) 1(2) 44 nova524 0(0) 29(100) 0(0) 29 nova525 0(0) 42(100) 0(0) 42 nova6 0(0) 28(90) 3(10) 31 nova7 0(0) 25(78) 7(22) 32 nova8 0(0) 28(97) 1(3) 29 nova9 0(0) 23(41) 33(59) 56 Total 16 1528 175 1719

A - 34

The three reliability questions were used to track if data were being obtained from the right respondents. If the response to all the three variables was 0% then reliability of the data would not be guaranteed. About 45%, 56% and 53% of the respondents in Somaliland, Puntland and South Central answered “Yes” to at least one of the three questions. Figure 3: Responses per reliability questions Q1, Q2 and Q3

Table 5: Percentage of Respondents indicating “Yes” by project sector

Code Definition of Sub - category Q1 Q2 Q3 01 Community Security and Safety 25.7 60.2 54.3 02 Peace and Security 0 0 14.3 03 Peace Process Support 24 78.6 59.4 04 Reconciliation 21.8 74.9 49.7 05 Strategic Response Fund 25.3 75 54.7 06 Education 20.8 69.9 37.2 07 Health and Hygiene 26.3 66.9 50.8 08 Water & Sanitation 0 0 0 09 Community infrastructure 24.7 70.5 46.5 10 Economic Development 31.8 76.8 56.8 11 Livelihoods 29.6 76.7 43.4 12 Economic Development/Livelihoods 19.9 74 49.4 13 Roads 30.1 53.2 48 14 Civic Education 19.7 69.4 43.4 15 Institution Strengthening 24.4 69.9 47.7 16 Service Delivery 23.5 66.7 42.9 17 Community Planning and Contracting 21.5 69.7 49.8 18 Capacity Building 21.4 53.6 52.7 19 Civil Society 34.8 71.9 70.7 20 Political Process 50 61.5 30.8 21 Training 26.9 55.2 21.3

A - 35

Code Definition of Sub - category Q1 Q2 Q3 22 Conflict Mitigation 25.1 74.6 52.7 23 Media 23.7 71.1 46.4 24 Communication and Coordination 50 61.5 30.8 25 Support Service 50 61.5 30.8 For data that were collected through PAPI, the field interviewers submitted all the paper interviews to the supervisors who in turn checked completeness and consistencies in following logical skip patterns. Results for other aspects of data quality are summarised in Table 6 below: Table 6: Data Quality Parameters

Parameter N (%) Completeness 1,719 (100%) Recording of GPS coordinates 1,719 (100%) Adherence to skip patterns 653 (100%) Interview Duration <30 minutes22 16 (1%) Between 30 and 120 minutes 1,528 (88%) At least 120 minutes 175 (11%) Total 1,719

Appendix The following question was introduced midway in the process of data collection to ensure more reliable data were being captured from the relevant respondents – who were potentially aware of TIS activities: 1. Please tell me if any of the following local TIS/DAI/IOM projects or activities have been going on in this settlement over the past two years.

READ OUT YES

Building new roads ☐1

Repairing / Improving roads ☐2

Building bridges ☐3

Repairing bridges ☐4

Building schools ☐5

Improving school buildings ☐6

Building health centres ☐7

Improving health centre ☐8

Building new market facility ☐9

Improving market facility ☐10

Providing sanitation services ☐11

22 All these were data obtained through PAPI in Xudur settlement. These interviews were considered valid as the reflected duration is for the entry of data into mobile platform by supervisors.

A - 36

Drinking Water ☐12

Water supply (for uses other than drinking) ☐13

Irrigation ☐14

Improving irrigation system ☐15

Electricity ☐16

Street lights ☐17

Building Meeting house ☐18

Building Sports Facility/Field ☐19

Planning a Cultural Event ☐20

Planning a Sports Event ☐21

Planning a Peace Event ☐22

Other: Specify ☐23

4. Development of Survey Weights for 2016 dataset The final survey results were disaggregated by geographic area (Somaliland, Puntland, South Central23 and Mogadishu), age category and gender (male and female). These results were to be weighted in order to be representative of the whole Somalia population as well as the geographic area population. These weights were developed based on the UNFPA population data. No weights were developed for age category as there were no data clearly disaggregated by age reflecting TIS population coverage (18 – 60 years). Weights calculation proceeded as follows; first population and sample size proportions were estimated for each region, then weights calculated by obtaining the quotient between population proportion and sample proportion (see table 1). Table 1: Weights developed at the regional level for weighting total results at the country (Somalia) level

Population Population Sample Region Sample Size Weight (%) Size Proportion (%) Proportion (%) Awdal 673,263 6 51 3 202 Woqooyi Galbeed 1,242,003 10 200 12 86 Togdheer 721,363 6 78 5 132 Sool 327,428 3 47 3 110 Sanaag 544,123 5 112 7 77 Bari 719,512 6 149 9 69 Nugaal 392,698 3 69 4 75 Mudug 717,863 6 44 3 234 Galguduud 569,434 5 143 8 60 Hiraan 520,685 4 72 4 96 Middle Shabelle 516,036 4 84 5 82 Banadir 1,650,227 14 394 23 61 Lower Shabelle 1,202,219 10 82 5 210 Bay 792,182 7 44 3 273 Bakool 367,226 3 18 1 287 Gedo 508,405 4 90 % 76

23 South Central here refers to emerging federal states

A - 37

Lower Juba 489,307 4 42 2 164 Total 11,953,974 100 1,719 100 2294 Note: Several districts in the regions of Sool, Sanaag and Togdheer are contested by Somaliland and Puntland.

Table 2: Somaliland Weights

Population Sample Sample Weight Region Population Size Proportion (%) Size Proportion (%) (%) Awdal 673,263 19 51 10 184 Woqooyi Galbeed 1,242,003 35 200 41 86 Togdheer 721,363 21 78 16 129 Sool 327,428 9 47 10 97 Sanaag 544,123 16 112 23 68 Total 3,508,180 100 488 100 563

Table 3: Puntland Weights

Population Sample Sample Weight Region Population Size Proportion (%) Size Proportion (%) (%) Bari 719,512 39 149 57 69 Nugaal 392,698 21 69 26 81 Mudug 717,863 39 44 17 234 Total 1,830,073 100 262 100 384

Table 4: South Central Weights

Population Sample Region Population Size Sample Size Weight (%) Proportion (%) Proportion (%) Galguduud 569,434 11 143 25 46 Hiraan 520,685 10 72 13 84 Middle 516,036 10 84 15 71 Shabelle Lower 1,202,219 24 82 14 170 Shabelle Bay 792,182 16 44 8 208 Bakool 367,226 7 18 3 236 Gedo 508,405 10 90 16 65 Lower Juba 489,307 10 42 7 135 Total 4,965,494 100 575 100 100

Application of weights Here we analyse responses from data sub-samples using the question below: Q1. Thinking about your economic and financial status now compared with two years ago, would you say you are better off, worse off or about the same. Results Table 5: Illustrating the calculation process of weighting results at the country level (Somalia):

Unweighted (%) Weighted (%) Weight Region Better Worse About the Better Worse About the (%) Off off same off off Same Awdal 202 82 14 4 7 1 0

A - 38

Woqooyi Galbeed 86 71 15 13 3 1 0 Togdheer 132 56 18 26 3 1 1 Sool 110 55 17 28 3 1 1 Sanaag 77 49 28 23 2 1 1 Bari 69 58 9 33 2 0 1 Nugaal 75 58 1 23 2 1 1 Mudug 234 61 20 18 6 2 2 Galguduud 60 62 17 20 2 0 1 Hiraan 96 66 11 23 3 0 1 Middle Shabelle 82 68 12 20 2 0 1 Banadir 61 52 27 21 1 1 1 Lower Shabelle 210 72 12 16 7 1 1 Bay 273 69 15 17 8 2 2 Bakool 287 75 25 0 9 3 0 Gedo 76 88 6 7 3 0 0 Lower Juba 164 62 25 12 4 2 1 Overall 2294 63 18 19 67 18 15

The overall weighted % is the sum of weighted %s across all the regions. See Table 7 for better presentation of results for this question (for all respondents all over Somalia). Table 6: Comparison of weighted vs. unweighted %s

Unweighted (%) Weighted (%) Better off 63 67 Worse off 18 18 About the same 19 15

Note: - To obtain weighted analysis for each of the geographic areas, the same process (in tables 6 and 7) should be repeated using specific weights derived in tables 3, 4 and 5. - The weighting process explained above was disaggregated by gender and used for the analysis of the 2015/2016 TIS dataset (see tables 8, 9 and 10).

Table 7: Weighting by Gender

a) Overall Somalia Analysis Population Sample Weight Region Male (n) Proportion Female (n) Proportion Male (n) Proportion Female (n) Proportion Male Female Awdal 348,479 0.058 324,784 0.055 25 0.028 26 0.031 2.047 1.758 Woqooyi 618,827 623,176 101 99 Galbeed 0.102 0.106 0.114 0.119 0.900 0.886 Togdheer 361,315 0.060 360,048 0.061 38 0.043 40 0.048 1.396 1.266 Sool 173,026 0.029 154,402 0.026 23 0.026 24 0.029 1.105 0.905 Sanaag 283,035 0.047 261,088 0.044 56 0.063 56 0.067 0.742 0.656 Bari 363,698 0.060 355,814 0.060 76 0.085 73 0.088 0.703 0.686 Nugaal 199,671 0.033 193,027 0.033 34 0.038 35 0.042 0.862 0.776 Mudug 363,737 0.060 354,127 0.060 22 0.025 22 0.027 2.428 2.265 Galgaduud 284,255 0.047 285,179 0.048 73 0.082 70 0.084 0.572 0.573 Hiraan 276,315 0.046 244,370 0.041 37 0.042 35 0.042 1.096 0.982 Middle 270,386 245,650 51 33 Shabelle 0.045 0.042 0.057 0.040 0.778 1.047 Banadir 813,399 0.134 836,828 0.142 200 0.225 194 0.234 0.597 0.607 Lower 604,835 597,384 35 47 Shabelle 0.100 0.101 0.039 0.057 2.537 1.788 Bay 402,182 0.066 389,999 0.066 29 0.033 15 0.018 2.036 3.658 Bakool 194,261 0.032 172,965 0.029 10 0.011 8 0.010 2.852 3.042 Gedo 249,900 0.041 258,505 0.044 56 0.063 34 0.041 0.655 1.070 Lower Juba 247,492 0.041 241,815 0.041 23 0.026 19 0.023 1.580 1.791 All Regions 6,054,813 1.000 5,899,161 1.000 889 1.000 830 1.000 b) Somaliland Analysis1 Region Population Sample Weight

A - 39

Male (n) Proportion Female (n) Proportion Male (n) Proportion Female (n) Proportion Male Female Awdal 348,479 0.195 324,784 0.188 25 0.103 26 0.106 1.898 1.776 Woqooyi 618,827 623,176 101 99 Galbeed 0.347 0.362 0.416 0.404 0.834 0.895 Togdheer 361,315 0.202 360,048 0.209 38 0.156 40 0.163 1.295 1.280 Sool 173,026 0.097 154,402 0.090 23 0.095 24 0.098 1.024 0.915 Sanaag 283,035 0.159 261,088 0.151 56 0.230 56 0.229 0.688 0.663 Total 1,784,682 1.000 1,723,498 1.000 243 1.000 245 1.000 Note: Several districts in the regions of Sool, Sanaag and Togdheer are contested by Somaliland and Puntland. c) Puntland Analysis Population Sample Weight Region Male (n) Proportion Female (n) Proportion Male (n) Proportion Female (n) Proportion Male Female Bari 363,698 0.392 355,814 0.394 76 0.576 73 0.562 0.681 0.702 Nugaal 199,671 0.215 193,027 0.214 34 0.258 35 0.269 0.836 0.794 Mudug 363,737 0.392 354,127 0.392 22 0.167 22 0.169 2.354 2.317 Total 927,106 1.000 902,968 1.000 132 1.000 130 1.000 d) South Central Analysis Population Sample Weight Region Male (n) Proportion Female (n) Proportion Male (n) Proportion Female(n) Proportion Male Female Galgaduud 284,255 0.112 285,179 0.117 73 0.232 70 0.268 0.483 0.437 Hiraan 276,315 0.109 244,370 0.100 37 0.118 35 0.134 0.927 0.748 Middle Shabelle 270,386 0.107 245,650 0.101 51 0.162 33 0.126 0.658 0.798 Lower Shabelle 604,835 0.239 597,384 0.245 35 0.111 47 0.180 2.145 1.362 Bay 402,182 0.159 389,999 0.160 29 0.092 15 0.057 1.721 2.786 Bakool 194,261 0.077 172,965 0.071 10 0.032 8 0.031 2.411 2.317 Gedo 249,900 0.099 258,505 0.106 56 0.178 34 0.130 0.554 0.815 Lower Juba 247,492 0.098 241,815 0.099 23 0.073 19 0.073 1.336 1.364 Total 2,529,626 1.000 2,435,86 1.000 314 1.000 261 1.000 7 Note: - Weighting was conducted at the regional level within each geographic area. However, Mogadishu results were unweighted – since Mogadishu is not sub-divided into administrative regions. - The weights developed in this document are usable for weighting the 1719 household interviews conducted. See a separate annex describing the weighting process for subsets of data with similar settlements between 2013/14 and 2015/16 surveys.

5. Weights for Overlapping TIS 2013 and 2015 Survey Datasets There was no baseline survey conducted for TIS projects and therefore there was a need to find potential data sources that would be useful for inferring direction of change. However, an evaluation conducted in 201324 formed a data source for conducting comparisons with results in 2015/16. This document is therefore sub-divided into the following sections: Section II documents sample sizes for 2013 and 2015/16 that would be used for comparisons; Section III shows derived weights for each of the two survey datasets.

5.1 Sample Sizes used for 2013 and 2015/16 Survey Analyses The 2013 survey was conducted in about 67 settlements where TIS had implemented some activities, while the 2015/16 survey was conducted in 90 settlements. Between the two surveys, 55 settlements were similar (in 14 regions) with the following total sample sizes: a) 1,186 for the 2013 Survey b) 1,101 for the 2015 Survey

5.2 Weights Derived for the Analysis of Each Dataset The process of deriving weights for analysis is the same as described in ANNEX 3. This document still uses the universe data defined by UNFPA.

a) Weights for Analysis of 2013 Dataset (Overall Somalia) Population Sample Weight Region Male (n) Proportion Female (n) Proportion Male (n) Proportion Female (n) Proportion Male Female Awdal 348,479 0.067 324,784 0.064 10 0.0167 17 0.029 4.035 2.199

24 http://pdf.usaid.gov/pdf_docs/PA00K6SF.pdf

A - 40

Banadir 813,399 0.157 836,828 0.164 146 0.2433 148 0.253 0.645 0.651 Bari 363,698 0.070 355,814 0.070 54 0.0900 46 0.078 0.780 0.890 Galgaduud 284,255 0.055 285,179 0.056 50 0.0833 65 0.111 0.658 0.505 Lower Juba 247,492 0.048 241,815 0.047 48 0.0800 31 0.053 0.597 0.898 Lower Shabelle 604,835 0.117 597,384 0.117 10 0.0167 9 0.015 7.003 7.639 Middle Shabelle 270,386 0.052 245,650 0.048 3 0.0050 7 0.012 10.435 4.039 Mudug 363,737 0.070 354,127 0.070 28 0.0467 41 0.070 1.504 0.994 Nugaal 199,671 0.039 193,027 0.038 52 0.0867 33 0.056 0.445 0.673 Sanaag 283,035 0.055 261,088 0.051 24 0.0400 30 0.051 1.365 1.002 Sool 173,026 0.033 154,402 0.030 9 0.0150 9 0.015 2.226 1.974 Togdheer 361,315 0.070 360,048 0.071 31 0.0517 24 0.041 1.350 1.727 W/Galbeed 618,827 0.119 623,176 0.122 100 0.1667 108 0.184 0.717 0.664 Gedo 249,900 0.048 258,505 0.051 35 0.0583 18 0.031 0.827 1.653 All Regions 5,182,055 1 5,091,827 1 600 1 586 1

b) Weights for Analysis of 2013 Dataset (Somaliland) Population Sample Weight Region Male (n) Proportion Female (n) Proportion Male (n) Proportion Female (n) Proportion Male Female Awdal 348,479 0.195 324,784 0.188 10 0.057 17 0.090 3.398 2.084 Woqooyi Galbeed 618,827 0.347 623,176 0.362 100 0.575 108 0.574 0.603 0.629 Togdheer 361,315 0.202 360,048 0.209 31 0.178 24 0.128 1.136 1.636 Sool 173,026 0.097 154,402 0.090 9 0.052 9 0.048 1.874 1.871 Sanaag 283,035 0.159 261,088 0.151 24 0.138 30 0.160 1.150 0.949 Total 1,784,682 1 1,723,498 1 174 1 188 1 Note: Several districts in the regions of Sool, Sanaag and Togdheer are contested by Somaliland and Puntland

c) Weights for Analysis of 2013 Dataset (Puntland) Population Sample Weight Region Male (n) Proportion Female (n) Proportion Male (n) Proportion Female (n) Proportion Male Female Bari 363,698 0.392 355,814 0.394 54 0.403 46 0.383 0.973 1.028 Nugaal 199,671 0.215 193,027 0.214 52 0.388 33 0.275 0.555 0.777 Mudug 363,737 0.392 354,127 0.392 28 0.209 41 0.342 1.878 1.148 Total 927,106 1 902,968 1 134 1 120 1

d) Weights for Analysis of 2013 Dataset (South Central) Population Sample Weight Region Male (n) Proportion Female (n) Proportion Male (n) Proportion Female (n) Proportion Male Female Galgaduud 284,255 0.172 285,179 0.208 50 0.342 65 0.500 0.501 0.416 Middle Shabelle 270,386 0.163 245,650 0.179 3 0.021 7 0.054 7.942 3.330 Lower Shabelle 604,835 0.365 597,384 0.436 10 0.068 9 0.069 5.330 6.298 Lower Juba 247,492 0.149 241,815 0.177 48 0.329 31 0.238 0.454 0.740 Gedo 249,900 0.151 258,505 0.189 35 0.240 18 0.138 0.629 1.363 Total 1,656,868 1 1,370,028 1 146 1 130 1

e) Weights for Analysis of 2015/16 Dataset (Overall Somalia) Population Sample Weight Region Male (n) Proportion Female (n) Proportion Male (n) Proportion Female (n) Proportion Male Female Awdal 348,479 0.067 324,784 0.06378536 25 0.044 26 0.049 1.536 1.300 Woqooyi Galbeed 618,827 0.119 623,176 0.1223875 87 0.152 81 0.153 0.784 0.801 Togdheer 361,315 0.070 360,048 0.07071096 22 0.039 26 0.049 1.810 1.441 Sool 173,026 0.033 154,402 0.0303235 16 0.028 17 0.032 1.192 0.945 Sanaag 283,035 0.055 261,088 0.0512759 41 0.072 40 0.075 0.761 0.679 Bari 363,698 0.070 355,814 0.06987944 23 0.040 25 0.047 1.742 1.481 Nugaal 199,671 0.039 193,027 0.03790918 21 0.037 18 0.034 1.048 1.116 Mudug 363,737 0.070 354,127 0.06954812 16 0.028 17 0.032 2.505 2.168 Galgaduud 284,255 0.055 285,179 0.05600721 63 0.110 57 0.108 0.497 0.521 Middle Shabelle 270,386 0.052 245,650 0.04824398 31 0.054 17 0.032 0.961 1.504 Banadir 813,399 0.157 836,828 0.1643473 163 0.285 169 0.319 0.550 0.515 Lower Shabelle 604,835 0.117 597,384 0.11732213 18 0.032 16 0.030 3.703 3.886 Gedo 249,900 0.048 258,505 0.05076861 34 0.060 14 0.026 0.810 1.922 Lower Juba 247,492 0.048 241,815 0.04749081 11 0.019 7 0.013 2.479 3.596 All Regions 5,182,055 1 5,091,827 1 571 1 530 1

f) Weights for Analysis of 2015/16 Dataset (Somaliland) Population Sample Weight Region Male (n) Proportion Female (n) Proportion Male (n) Proportion Female (n) Proportion Male Female Awdal 348,479 0.195 324,784 0.188 25 0.131 26 0.137 1.492 1.377 Woqooyi Galbeed 618,827 0.347 623,176 0.362 87 0.455 81 0.426 0.761 0.848

A - 41

Togdheer 361,315 0.202 360,048 0.209 22 0.115 26 0.137 1.758 1.527 Sool 173,026 0.097 154,402 0.090 16 0.084 17 0.089 1.157 1.001 Sanaag 283,035 0.159 261,088 0.151 41 0.215 40 0.211 0.739 0.720 Total 1,784,682 1 1,723,498 1 191 1 190 1 Note: Several districts in the regions of Sool, Sanaag and Togdheer are contested by Somaliland and Puntland

g) Weights for Analysis of 2015/16 Dataset (Puntland) Population Sample Weight Region Male (n) Proportion Female (n) Proportion Male (n) Proportion Female (n) Proportion Male Female Bari 363,698 0.392 355,814 0.394 23 0.383 25 0.417 1.023 0.946 Nugaal 199,671 0.215 193,027 0.214 21 0.350 18 0.300 0.615 0.713 Mudug 363,737 0.392 354,127 0.392 16 0.267 17 0.283 1.471 1.384 Total 927,106 1 902,968 1 60 1 60 1

h) Weights for Analysis of 2015/16 Dataset (South Central) Population Sample Weight Region Male (n) Proportion Female (n) Proportion Male (n) Proportion Female (n) Proportion Male Female Galgaduud 284,255 0.172 285,179 0.175 63 0.401 57 0.256 0.428 0.685 Middle Shabelle 270,386 0.163 245,650 0.151 31 0.197 17 0.076 0.826 1.979 Lower Shabelle 604,835 0.365 597,384 0.367 18 0.115 16 0.072 3.184 5.113 Gedo 249,900 0.151 258,505 0.159 34 0.217 14 0.063 0.696 2.528 Lower Juba 247,492 0.149 241,815 0.148 11 0.070 7 0.031 2.132 4.730 Total 1,656,868 1 1,628,533 1 157 1 111 0.498

A - 42

6. Development of Stability Index 6.1 Definition of Stability Since stability is not directly measurable, USAID described a set of characteristics that would be used to define (and potentially measure) stability. The 2015 – 2016 evaluation of Transition Initiative for Stabilisation (TIS) project defined the following seven characteristics (see also the report on Categorization of TIS Grants – dated: October 19, 2015):

 Sense of security among the public  Access to basic services  Opportunities for economic growth  Confidence in Government  Partnership between government and civil society & the private sector  No support for violent extremism  Return of IDPs or refugees

6.2 Linking the stability characteristics to USAID’s intermediate results Furthermore, linkages have been formulated between the four USAID intermediate results and the stability characteristics as follows: USAID IR 1.1: Capacity of key government institutions to perform essential functions improved.  Sense of security among the public  Confidence in Government  Return of IDPs or refugees USAID IR 1.2: Participatory political processes promoted  Partnership between government and civil society & the private sector  No support for violent extremism USAID IR 2.1: Access to quality basic social services increased  Access to basic services USAID IR 2.2: Economic opportunities increased  Opportunities for economic growth Survey questions were formulated to address each of the seven stability characteristics (categories) as shown in table 1 (also see deliverable #2 dated November 16, 2015 submitted to USAID). Table 1: Linking stability characteristics to survey questions

Characteristic Specific Questions Per Characteristic Sense of security 24,26 Confidence in government 11,23,28 Return of IDPs/Refugees 34,35 Government partnership 21 with civil society No appeal for violent 36,37 extremism Access to basic needs f,13 Economic growth 1,2

6.3 Formulation of Indices A set of models were formulated with model one being the primary and the remaining two used for sensitivity analyses to see how the index scores would vary under different considerations.

1) Model 1:  Questions were grouped by the 7 stability characteristics (see sub – titles a – g).

A - 43

 Index score was derived for each of the seven characteristics, applying the weighting scheme shown for each characteristic in Tables a) through g), below. Each characteristic was weighted equally. Below are the specific questions used for model 1:

a) Sense of Security Sense of Security 100%

100.0% Weighted Unweighted Question Response Weighted Response Value Weight Value Response Question Score 50.0% 24. Do you personally feel more safe, less safe or about the same living in this settlement compared to two years ago? 50.0% 100% 50.0% 1 More safe 0% 0.0% Less safe 0% 0.0% About the same as two years ago Don't know (Code as 99 SYSMIS)

50.0% 26. Thinking back over the past two years, do you think that the level of conflict and 50.0% violent disputes between different groups in this settlement has increased, remained about the same or decreased? 0% 0.0% Increased in the past two years 0% 0.0% Remained about the same 100% 50.0% 1 Decreased in the past two years There are no conflicts/ tensions in this settlement (Code as 99 SYSMIS for this index)

b) Confidence in Government

Confidence in Government 100% 100.0% Sub - Unweighted Weighted Weighted Question question Response Response Score for sub Weighted Weight weight Value Value Response Question questions Score 33.3% 11. Taking everything into consideration, how confident are you that the following institutions or organisations are able to improve the lives of all the people in this settlement? Would you say you are Very Confident, Fairly Confident, Not very confident or Not at all confident? 33.3% 16.7% 100% 16.67% 1 Central government - very confident 67% 11.10% Central government - fairly confident 16.7% 33% 5.55% Central government - not very confident 0% 0.00% Central government - not at all confident

16.7% 100% 16.67% 1 Local administration - very confident 16.7% 67% 11.10% Local administration - fairly confident 33% 5.55% Local administration - not very confident 0% 0.00% Local administration - not at all confident

Don't know (Code as 99 SYSMIS)

33.3% 28. How much trust/ confidence do you personally have in each of the following to resolve conflict and violent disputes between different groups in this settlement? Do you; Trust them a lot, trust them a little, do not trust them very much or do you not trust them at all? 33.3% 16.7% 100% 16.67% 1 Central government - Trust a lot 16.7% 67% 11.10% Central government - Trust a little 33% 5.55% Central government - Do not trust them very much 0% 0.00% Central government - Do not trust them at all

16.7% 100% 16.67% 1 Local administration - Trust a lot 16.7% 67% 11.10% Local administration - Trust a little 33% 5.55% Local administration - Do not trust them very much 0% 0.00% Local administration - Do not trust them at all

Don't know (Code as 99 SYSMIS) 33.3% 23. To what extent do you agree with the following statements about some of the institutions in the country? For each one I read out I would like to know if you Strongly agree, somewhat agree, somewhat disagree or completely disagree 33.3% 1 The local administration here is very professional - Strongly 8.333% 100% 8.333% agree 8.333% The local administration here is very professional - 67% 5.550% Somewhat agree The local administration here is very professional - 33% 2.775% Somewhat disagree The local administration here is very professional - 0% 0.000% Completely disagree

1 People are selected to work in government offices/ positions 8.333% 100% 8.333% because of who they know - Strongly agree 8.333% People are selected to work in government offices/ positions 67% 5.550% because of who they know- Somewhat agree People are selected to work in government offices/ positions 33% 2.775% because of who they know - Somewhat disagree

A - 44

People are selected to work in government offices/ positions 0% 0.000% because of who they know- Completely disagree

8.333% 100% 8.333% 1 Our local politicians are corrupt - Strongly agree 8.333% 67% 5.550% Our local politicians are corrupt - Somewhat agree 33% 2.775% Our local politicians are corrupt - Somewhat disagree 0% 0.000% Our local politicians are corrupt - Completely disagree

1 Those serving government at the national level are corrupt - 8.333% 100% 8.333% Strongly agree 8.333% Those serving government at the national level are corrupt - 67% 5.550% Somewhat agree Those serving government at the national level are corrupt - 33% 2.775% Somewhat disagree Those serving government at the national level are corrupt - 0% 0.000% Completely disagree

c) Return of IDPs and Refugees Return of IDPs and Refugees 100% 100.0% Weighted Unweighted Question Response Weighted Response Value Weight Value Response Question Score 33.3% 35. a) To your knowledge, have any of the people in your settlement that left for safety reasons returned to this settlement in the last two years? 100% 33.3% 1 Yes 33.3% 0% 0.0% No Don't know (Code as 99 SYSMIS) 33.3% 35 b) To your knowledge, have any of the people who fled to your 33.3% neighbourhood or settlement for safety reasons returned to their settlement of origin in the last two years? 100% 33.3% 1 Yes 0% 0.0% No Don't Know (Code as 99 SYSMIS) 33.3% 34. Have you personally, welcomed anyone into your household who has fled from upheaval or violent clashes from other parts of the country in the past two years? 100% 33.3% 1 Yes 33.3% 0% 0.0% No Don't know (Code as 99 SYSMIS)

d) Government Partnership with civil society and the private sector Government Partnership with civil society and the private sector 100% 100.0% Sub - Unweighted Weighted Question question Response Weighted Score for sub Weighted Weight weight Value Response Value Response Question questions Score 100.0% 21. Here are some comments about the ways in which decisions are made in this settlement. Please tell me how much you personally agree or disagree with each of them compared to two years ago 100.0% 1 People living in this settlement have a say in the decisions that improve our lives (compared to two years ago) - 25.00% 100% 25.00% strongly agree 25.0% People living in this settlement have a say in the decisions that improve our lives (compared to two years ago) - agree a 67% 16.7% little People living in this settlement have a say in the decisions that improve our lives (compared to two years ago) - 33% 8.3% disagree a little People living in this settlement have a say in the decisions that improve our lives (compared to two years ago) - 0% 0.0% strongly disagree 1 The local administration is responsive to the needs of the people who live here (compared to two years ago) - strongly 25.00% 100% 25.00% agree 25.0% The local administration is responsive to the needs of the people who live here (compared to two years ago) - agree a 67% 16.7% little The local administration is responsive to the needs of the people who live here (compared to two years ago) - disagree 33% 8.3% a little The local administration is responsive to the needs of the people who live here (compared to two years ago) - strongly 0% 0.0% disagree (The extent to which people of this settlement have a say in the decisions that improve our lives) - increased in the past 2 25.00% 100% 25.00% 1 years 25.0% (The extent to which people of this settlement have a say in the decisions that improve our lives) - decreased in the past 0% 0.00% 2 years (The extent to which people of this settlement have a say in 0% 0.00% the decisions that improve our lives) - remained the same (The extent to which the local administration is responsive to the needs of the people who live here) - increased in the past 25.00% 100% 25.00% 1 2 years 25.0% (The extent to which the local administration is responsive to the needs of the people who live here) - decreased in the 0% 0.00% past 2 years (The extent to which the local administration is responsive to 0% 0.00% the needs of the people who live here) - remained the same

A - 45

e) No Support for Violent extremism No Support for violent extremism 0% 100.0% Unweighted Weighted Question Response Response Weighted Weight Value Value Response Question Score

50.0% 36. Do you believe that young people (below 35 years of age) in your settlement are 0.0%

attracted to become members of violent extremist groups compared to two years ago? 100% 0.0% 1 Yes 0% 50.0% No Don't Know (Code as 99 SYSMIS)

37. In your opinion, are extremists more or less welcome or not welcome at all in this 50.0% settlement compared to two years ago?

50% 0.00% 1 Less welcome 0.0% 0% 0.00% More welcome 0% 0.00% No change – some welcome 100% 50.00% Not welcome at all

f) Access to and satisfaction with basic needs Access to and satisfaction with basic services 100% 100.0% Weight for Unweighted Weighted Question sub Response Response Weighted Weight questions Value Value Response Question Score 50% f) Which of the following facilities and services, if any, do you have within 2 km of your house? 5.00% 100.00% 5.00% 1 Health Clinic - Yes 5.0% 0% 0% 0.0% Health Clinic - No

5.00% 100.00% 5.00% 1 Police Post - Yes 5.0% 0% 0% 0.0% Police Post - No

5.00% 100.00% 5.00% 1 Primary School - Yes 5.0% 0% 0% 0.0% Primary School - No

5.00% 100.00% 5.00% 1 Secondary School - Yes 5.0% 0% 0% 0.0% Secondary School - No

5.00% 100.00% 5.00% 1 Government or local administration office - Yes 5.0% 0% 0% 0.0% Government or local administration office - No

5.00% 100.00% 5.00% 1 Local court - Yes 5.0% 0% 0% 0.0% Local court - No

5.00% 100.00% 5.00% 1 Market - Yes 5.0% 0% 0% 0.0% Market - No

5.00% 100.00% 5.00% 1 Access to drinking water - Yes 5.0% 0% 0% 0.0% Access to drinking water - No

5.00% 100.00% 5.00% 1 Settlement centre - Yes 5.0% 0% 0% 0.0% Settlement centre - No

5.00% 100.00% 5.00% 1 Sports hall- Yes 5.0% 0% 0% 0.0% Sports hall - No 13. Taking everything into consideration, how satisfied are you with the provision of each of the following services in your settlement? Would you say you are Very satisfied, Fairly satisfied, Not very satisfied or Not at all satisfied? 50.0% 10.00% 100% 10.00% 1 Health Services - very satisfied 10.0% 67% 6.7% Health Services - fairly satisfied 33% 3.3% Health Services - not very satisfied 0% 0.0% Health Services - not at all satisfied

10.00% 100% 10.00% 1 Education- very satisfied 10.0% 67% 6.7% Education - fairly satisfied 33% 3.3% Education - not very satisfied 0% 0.0% Education - not at all satisfied

10.00% 100% 10.00% 1 Provision of Drinking Water - very satisfied 10.0% 67% 6.7% Provision of Drinking Water - fairly satisfied 33% 3.3% Provision of Drinking Water - not very satisfied 0% 0.0% Provision of Drinking Water - not at all satisfied

10.00% 100% 10.00% 1 Roads/transport - very satisfied 10.0% 67% 6.7% Roads/transport - fairly satisfied 33% 3.3% Roads/transport - not very satisfied 0% 0.0% Roads/transport - not at all satisfied

10.00% 100% 10.00% 1 Electricity - very satisfied 10.0% 67% 6.7% Electricity - fairly satisfied 33% 3.3% Electricity - not very satisfied 0% 0.0% Electricity - not at all satisfied

A - 46

Don't know (Code as 99 SYSMIS) g) Opportunities for Economic Growth Opportunities for Economic Growth 100% 50.0% Unweighted Weighted Question Response Response Weight Value Value Response Question Weighted Score 2. Do you believe there are opportunities in this settlement for people to make enough money to support the family compared to two years ago? 50.00% 100.00% 50.00% 1 Yes 50.00% 0% 0.0% No Thinking about your economic and financial status now compared with two years ago, would you say you are better off, worse off or about 50.00% the same? 100% 50.0% 50.0% Better off now compared with two years ago 50.0% 0% 0.00% 0.00% Worse off now than two years ago 0% 0.00% 0.00% About the same as two years ago

6.4 Validation Process Three models of the Stability Index were developed: For Model 1, each of the stability characteristics added up to 100 percent and a single score was derived by averaging scores across the seven stability characteristics. For Model 2, each of the seven stability characteristics added up to 14.28 percent (one hundred percent divided by seven). To derive a single stability score, the scores corresponding to the seven stability characteristcs were calculated. For Model 3, all the questions across the stability characteristics were pooled and each was assigned a score of 100% divided by the total number of questions. To derive a single stability score, scores from each of the questions were calculated. 6.4.1: Model 1 was validated through sensitivity analyses. This sensitivity analysis process involved the formulation of other two models to check how estimates under model one would vary. These two models were defined as follows: 6.4.2: Model 2:  Questions were grouped by the 7 stability characteristics.  Each of the 7 characteristics was assumed to contribute equally to the overall Stability Index – so assigned equal weights (with questions weighted equally within each characteristic). That is to say, each stability characteristic contributed 14.28% to the overall stability score and this 14.28% was divided equally to the number of questions under each characteristic. 6.4.3: Model 3:  All the questions in the 7 characteristics have been assumed to contribute equally to the overall Stability Index – so each is assigned equal weight (regardless of the stability characteristic). The following are the summary results of these models – when applied on the TIS evaluation dataset. 6.5 Validation Results Model 1 Model 2 Model 3 Sense of security 65% 9% 3% Confidence in Government 67% 10% 14% Return of IDPs/Refugees 31% 4% 2% Government Partnership with civil society 64% 10% 7% No appeal for violent extremism 66% 9% 3% Access to basic needs 61% 9% 22% Economic Growth 65% 9% 7%

A - 47

Total25 60% 61% 58%

The average Stability Index scores do not vary substantial from one model to the other. Also see figure 1 for the distribution curves for Stability Index scores for each of the models: Figure 1: Distribution curves for indices under models 1 – 3

6.6 Using the Index Scores for Analysis The development of the index model utilised all the seven stability characteristics. However, the analysis of 2015/16 dataset alone used six of the seven characteristics. One of the stability characteristics (Return of IDPs) was not included in the analysis as answers to survey questions under this characteristic varied from one geographic area to the other. For comparison of results between 2013 and 2015/16 survey results, Stability Index was derived using four stability characteristics that had questions common to both of the surveys. These four characteristics included:

 Sense of Security  Confidence in Government  Government Partnership with civil society

25 Totals for models 1 is obtained by averaging the percentage scores across the stability characteristics while totals for models 2 and 3 are obtained by summing up the percentage scores across the characteristics. We should note that the scores per characteristic for model 3 may not interpretable as these depend on the number of questions considered under a characteristic.

A - 48

 Access to basic Services 7. Regression Analysis A report on categorisation of TIS grants (dated: October 19, 2015) linked grants in the TIS database to 25 sub categories (also referred to as project subsectors). These include the following:

Table 1: Grant sub categories

Definition of Sub-category Code of Sub-category Community Security and Safety 01 Peace and Security 02 Peace Process Support 03 Reconciliation 04 Strategic Response Fund 05 Education 06 Health and Hygiene 07 Water & Sanitation 08 Community infrastructure 09 Economic Development 10 Livelihoods 11 Economic Development/Livelihoods 12 Roads 13 Civic Education 14 Institution Strengthening 15 Service Delivery 16 Community Planning and Contracting 17 Capacity Building 18 Civil Society 19 Political Process 20 Training 21 Conflict Mitigation 22 Media 23 Communication and Coordination 24 Support Service 25

There was a need to investigate which mixes of grant sub-categories would yield the highest stability (measured using Stability Index). However, an exploratory analysis of the grants per settlement showed that almost each settlement received a unique set of grants – and therefore relating a cluster of settlements to a given mix was not feasible. Therefore, a beta regression26 analysis was fitted for each of the grant sub categories (with the outcome being Stability Index) and results presented in terms of log odds. See table 2 for full results. It is important to note that these effects may not be interpretable individually as interactions would be more meaningful – though this was not achievable through the model. Table 2: Beta regression model estimates

Estimate (log odds) Std. Error Z value Pr(>|z|) (Intercept) 0.7330 0.0541 13.5412 0.0000 sector1 -0.1512 0.0696 -2.1742 0.0297 sector2 -0.2858 0.0839 -3.4059 0.0007 sector3 -0.0545 0.0841 -0.6486 0.5166 sector4 -0.1317 0.0557 -2.3661 0.0180 sector5 0.0350 0.0555 0.6311 0.5280

26 Beta regression is suitable for modeling data within the range of 0 and 1. The traditional linear regression if used may lead to predictions outside the range of the data.

A - 49

sector6 -0.1352 0.0680 -1.9875 0.0469 sector7 0.0886 0.0437 2.0257 0.0428 sector8 -0.2884 0.0555 -5.1947 0.0000 sector9 -0.0387 0.0550 -0.7025 0.4824 sector10 0.1436 0.0550 2.6123 0.0090 sector12 0.1116 0.0673 1.6585 0.0972 sector13 0.4735 0.1603 2.9530 0.0031 sector14 -0.1536 0.0701 -2.1906 0.0285 sector15 0.3392 0.0943 3.5959 0.0003 sector16 0.2785 0.0785 3.5483 0.0004 sector17 -0.2883 0.1042 -2.7661 0.0057 sector18 -0.4673 0.2116 -2.2086 0.0272 Puntland -0.1624 0.0869 -1.8691 0.0616 South Central -0.0109 0.0549 -0.1985 0.8426 Mogadishu -0.0115 0.0993 -0.1156 0.9080 Note: Reference Levels for sectors 1 - 18 is non- implementation Reference for Geographic Area is Somaliland

Key27 sector1 Civic Education sector2 Civil Society sector3 Communication, Coordination and Media sector4 Community Infrastructure sector5 Community Security and Safety sector6 Conflict Mitigation sector7 Economic Development/Livelihoods sector8 Education/Training sector9 Health and Hygiene sector10 Institution Strengthening sector12 Planning Processes sector13 Political Process sector14 Reconciliation sector15 Roads sector16 Service Delivery sector17 Strategic Response Fund sector18 Water/Sanitation

27 Some of the grant sub categories were merged in the regression analysis – hence resulting in 18 sectors

A - 50

Interpretation of coefficients A regression analysis was attempted but was unsuccessful due to the complexity involved in the interpretation of parameter (intervention) specific effects; therefore a valid interpretation of individual coefficients is not possible. The model is nonsensical in that many of the sectors with negative coefficients were implemented in settlements with a high Stability Index. The model suggests that settlements where project sectors 1, 2, 4, 6, 8, 14, 17, and 18 were implemented had reduced odds of being stable by 0.86, 0.76, 0.87, 0.87, 0.74, 0.85, 0.75, 0.62 times, respectively. On the other hand, settlements where project sectors 7, 10, 13, 16 were implemented had increased odds of being stable by 1.09, 1.15, 1.61 and 1.32 times. As many settlements with a high Stability Index had projects sectors implemented in sectors with both positive and negative effects, this “finding” appears to be contradictory.

8. Activities by Sector, undertaken in Settlements with a High Stability Index Service Delivery Improving sanitation and health in Jowhar District: Provision of Garbage truck Political Process Promoting Peace in Kismayu through Art and Culture Economic Development Economic empowerment and peace-building in Bargaal and Livelihoods Increasing Livelihood Opportunities through Provision of a Market in Burtinle Improving Access to Alternative Livelihoods and Awareness Raising on Sea Piracy Improving Youth’s Livelihood through Skills Training and Job Creation In-kind support to women businesses in Baki; and Engaging youth through basic Information Technology training & business support Vocational skills training and business support for Zeylac youth Improving livelihoods in Karaan District: Rehabilitation of Meat Market Improving the Infrastructural capacity of Berbera Marine College; and Strengthening the Economic Growth of Fishing Cooperatives in Saahil Region Enhancing economic development: Constructing a slaughter house in Jowhar District Institutional Strengthening Supporting Gar’adag Municipality in Service Delivery- Building Mayor’s Office Building the Capacity of the Kismayo Municipality Providing Infrastructural Support to the Local Government Office in El- Afweyn Strengthening Bandarbeyla community's confidence in their local government Strengthening the Capacity of the Local Administration in Galdogob Building Government Capacity in Baki: IT and Furniture for Ward Offices Improving service delivery through support to the Mayd regional administration Enhanced Community confidence through Administration block and Hall provision Improving Governance in Zeylac: Capacity Building Training for Local Government Improving Service Delivery of Karaan District Administration Increasing the Capacity of Go’da Municipality in Service Delivery; and Supporting Mandera Local Authority in Service Delivery

A - 51

Building Capacity in Jowhar: Hiring Two Staff for the District Administration Roads Rehabilitation and Construction of 15.9 KM Main Road in Kismayo Municipality Increasing Public Transport in Baargaal through Construction of Gravel Road Increasing Access to Jariban through Construction of Gravel Road Construction of 1.1km Tarmac Road in Gardo District and Improving Public Transport in Gardo through Construction of a Tarmac Road

A - 52

Annex D: Evaluation Tools Copies of all tools such as survey instruments, questionnaires, discussions guides, checklists

REVISED QUANTITATIVE SURVEY QUESTIONNAIRE

INTERVIEW INFORMATION

FOR INTERVIEWER COMPLETION ONLY --DO NOT ASK RESPONDENT!

[Local Greeting] My name is ______, I work for xxxxxxx, a local research company. We are conducting a study in this settlement, and your household has been randomly selected to participate. The study is all about how people feel about living in this settlement. All data will be confidential; no one will know what you have told me. Today, we will sit down with a member of your household, aged between 18 and 60, selected at random by lottery, for an interview lasting about 45 minutes.

SELECTION OF RESPONDENT FOR INTERVIEWING

INTERVIEWER READ OUT: Including yourself, how many people who are residents of this country are aged between 18 and 60, currently live in this household? When I say “household” I mean members of the family who live together and share resources, i.e. people who eat together, and who pool their resources to buy food and other necessities. Visiting relatives and domestic workers are not included.

01 ______Total number of adults in household

INTERVIEWER  List all of the male/female adults between the ages of 18 and 60 years living in the household (together with their ages) who are present at the time of household visit. Start with the oldest and work down to the youngest.  Write all the names on small pieces of paper, fold, mix them and drop in a container or on a flat surface.  Invite someone (usually the head of the household if he/she is available) to pick one paper out of the lot.  The person selected will be interviewed- ask to speak to the person whose name appears on the paper picked out

A-53

Names Gender Age Person Selected for Interview 1 2 3 4 5 6 7 8 9 10

1. Supervisor code 2. Interviewer code 3. Serial Number of Device

4. Interviewer Name 8. Date ____/_____/2015 (dd / mm)

5. Sex of interviewer Male O1 Female O2

9. Interview start time: ____/____/ (hh/mm)

6. Sex of respondent 10. Interview end time: ____/____/ Male O1 Female O2 (hh/ mm)

7. Age of the respondent (in years) ______

11. Geographic 12. Region 13. District 14. Settlement 15.GIS Location area Coordinates Somaliland Puntland

A-54

South Central Mogadishu

16. Urban/Rural: [DO NOT READ – CODE TO FIT]

SINGLE CODE

Urban 1

Rural 2

a) RECORD the main source of lighting for the house [DO NOT READ – CODE TO FIT]

SINGLE CODE

Electricity 1

Lantern (kerosene) 2

Paraffin Lamp 3

Fuel Wood 4

Solar 5 Other:

 ______6

b) How long have you personally lived in this settlement? [READ OUT – CODE ONE ONLY] SINGLE CODE INSTRUCTION

Less than one year 1 CLOSE

1 to 2 years 2

3 to 5 years 3

More than 5 years 4

A-55

c) What is your current employment situation? [READ OUT – SINGLE CODE] SINGLE INSTRUCTIONS CODE

Formal employment earning a regular wage 1  2 Informal employment earning money from time to time 3 Business Owner 4 Sell items caught or produced at a market? (fish,  agricultural products, clothes, etc.) 5

Housewife 6 Student 7 Retired 8

Unemployed 9 Refused to answer  10 Other (write in) 11

d) If employed, what is your current occupation? [DO NOT READ – CODE APPROPRIATELY] IF BUSINESS OWNER, PROBE IF SMALL BUSINESS OR MIDDLE-LARGE BUSINESS OWNER SINGLE CODE Manual labourer 1 Farmer 2 Small business owner (micro-enterprise) 3 Middle-large business owner 4 Security worker 5 Domestic worker 6 Driver 7 Teacher 8 Fisherman 9 Tradesman/woman (electrician, mechanic, tailor, auto-  body, welder) 10 Health worker 11 Office / Admin worker 12 NGO/INGO 13 UN 14 Government worker 15 Pastoralist 16 Other: 17

Refused to answer 18

A-56

e) Are there any other members of this household that earn money through: NOTE: Read out and ask how many people in the household for any to which they say yes and place in column “number of household members.” If no, write in “0”

READ OUT NUMBER OF MULTIPLE HOUSEHOLD CODES MEMBERS (0, 1, 2, etc.) Formal employment earning a regular ☐ wage 1

☐2

Informal employment earning money ☐3 from time to time

Their own business ☐4

Selling items at market (fish, ☐5 agricultural products, dairy, etc.)

No other members of this family earn ☐6 money

Other: (write in) ☐7

Refused to answer ☐8

f) Which of the following facilities and services, if any, do you have within 2 km of your house?

READ OUT EACH IN TURN Within 2km

Health clinic ☐1

Police post ☐2

Primary school ☐3

Secondary school ☐4

Government or local administration ☐5 office

Local court ☐6

Market ☐7

Access to drinking water/ ☐8

Access to water point (for other uses ☐9 such as farming, for animals, etc.) sources (for farming, animals, etc.)

Settlement center ☐10

Sports Hall ☐11

Bank/financial institution ☐12

A-57

MAIN SURVEY QUESTIONS

I now would like to ask you a number of questions about how your life is today, living here in this settlement compared to two years ago.

1. Thinking about your economic and financial status now compared with two years ago, would you say you are better off, worse off or about the same?

SINGLE CODE Better off now compared with two  years ago 1

Worse off now than two years ago 2

About the same as two years ago 3

Refused to answer 4

2. Do you believe there are opportunities in this settlement for people to make enough money to support the family compared to two years ago? Yes No Do not know

3. a) Have you or anyone else in this household received vocational training supported by the TIS program? (If no, skip to question 8) Yes No