2. Methodology the Evaluation Was Conducted As a Combined Internal and External Evaluation by a Team of Seven People, Together with Part of the LEADS Staff

2. Methodology The evaluation was conducted as a combined internal and external evaluation by a team of seven people, together with part of the LEADS staff. I as the team leader focused on institutional aspects, on some of the statistical analysis, on the overall co-ordination of the evaluation and on writing the report. Mr. Lakshman Perera, National Programme Manager, was my sparring partner during the fieldwork. Mrs. L.L.P. Priyangika and Ms. Madushani Jayathilaka together with the staff at the Hambantota office focused on the social aspects of the projects implemented in the Sinhala-speaking project areas in Hambantota District. Mrs. Nazeera Parwin and Mrs. Indrani Rajendran together with the staff at the Kalmunai and Trincomalee offices focused on the social aspects of the projects implemented in the Tamil-speaking project areas in Ampara and Trincomalee Districts. Dr. Lesly Ekanayake, Mr. R.U. Halwathura and Mr. David Edirisinghe focused on the technical aspects of the housing and infrastructure project in Matara, Hambantota, Ampara, and Trincomalee Districts.

In order for the team to prepare, LEADS provided relevant documents to read through. After two joint preparation meetings at the LEADS office in Colombo, a checklist with key questions, organised by project result and by aspect of the evaluation, was developed, and a selection was made as to which question could best be asked to whom and with which method (annex 4). After this, I had a long interview with the Executive Director, Mr. Roshan Mendis, in which many aspects were touched upon. This was followed by the actual field work (see table below for the time schedule of the social evaluation team). The social evaluation team and the technical evaluation team travelled separately because the technical evaluation team did not have as much time available as the others.

Preparatory meetings March 26th, May 24th, June 27th 2007 Interviews in Colombo June 27th, July 24th 2007 Travel to Hambantota July 8th 2007 Fieldwork in Hambantota July 9th – 11th 2007 Travel to Kalmunai July 11th 2007 Fieldwork in Kalmunai July 12th – 14th 2007 Travel to Trincomalee July 15th 2007 Fieldwork in Trincomalee July 16th – 18th 2007 Travel to Colombo July 19th 2007 Debriefing with PALTRA July 24th 2007 Debriefing with the LEADS staff July 24th 2007 Debriefing with the Board of LEADS July 26th 2007 Writing of the report First draft ready August 3rd 2007, final draft ready September 11th 2007, final version ready September 19th 2007

The methods used for the evaluation were fairly straightforward: • Document review: before the evaluation started, proposals, project progress reports and financial reports were scrutinized. Based on this, a memo was submitted to LEADS with several dozen further questions, which were all answered. During the time in the districts, beneficiary files for the livelihoods project and whatever other documents were available in the field office (including documentation related to the housing project) were reviewed. In Hambantota, a sample of the beneficiary files was inspected; in Kalmunai and Trincomalee all files were reviewed, and some of the available data was copied into an Excel file for further analysis. In certain cases, specific beneficiaries were selected for family visits based on this analysis, some because they either did particularly well, others because they did particularly poorly, and others because they seemed exemplary for a large group of others. After family visits, the feedback of the families was compared with the information in the beneficiary files. • Semi-structured interviews: in the Colombo office and in the field offices, key staff were interviewed individually to get an understanding of the work that was done, and the processes in the organisation. Also, key government officials (five Divisional Secretaries, an Additional Government Agent and two Livelihood Officers) were interviewed about the work of LEADS.

• Focused Group Discussions: in each district, focused group discussions were held with beneficiaries: in Hambantota four for the temporary shelter and livelihood project (in Hambantota and Ambalantota) and one for the both projects combined (in Ambalantota); in Kalmunai three for both projects combined in Kalmunaikudy, Karaithivu and Kalmunai, and two with children in Kalmunaikudy and Karaithivu, and in Trincomalee two for temporary shelter and livelihood in Kinniya and Eechchanthivu, two for both projects together in Muthur, and one for the housing project in Kuchchaveli. These meetings lasted about one to two hours each. Occasionally, a few PRA tools were used to visualise aspects of the discussions. • Family visits: in order to get more detailed information on the projects and their impact on the lives of individual beneficiaries, family visits were made. For the temporary shelter and livelihood project, about fifteen to twenty beneficiaries were visited in each district; for the housing project, about 5 families were visited in Hambantota, about 30 in Kalmunai, and about 30 in Trincomalee. Most of these visits were made by one or more evaluators together with LEADS staff. However, to get an idea whether or not people might be reluctant to criticise LEADS in the presence of staff, some families were visited only by evaluators. In Muthur it was decided to split up the team so that the female evaluators could meet separately and speak with women, because in the presence of male evaluators and LEADS staff the men did most of the talking and it was felt that women didn’t have the opportunity to speak freely. • Physical verification and inspection was done in all districts. The technical team inspected about five houses in Hambantota, 48 in Kalmunai, 19 in Trincomalee and 17 in Matara. During the family visits, the social evaluators inspected houses, livelihood items, and non-food items. • Towards the end of the visit to each district, a focused group discussion was held with the district staff of LEADS, to get their opinions on the work they had done. In Kalmunai and Trincomalee, a strengths, weaknesses, opportunities and threats exercise was done as part of this.

Not strictly within the ToR, but considered relevant nevertheless, was the issue of the relations between LEADS and its funding partners. In order to gain insight into this, contact was sought by e- mail with the relevant people in Tearfund UK and Tear Netherlands (who are personally known to me). An accidental encounter in church brought me in touch with the Country Representative of another funding partner, Samaritan’s Purse. Their feedback was discussed with the ED.

After the fieldwork was over, the initial findings were discussed with the ED in Colombo on July 24th, and subsequently with the District Officers (DOs) and District Engineers (DEs). The idea was to present the findings and, more importantly, get feedback from the LEADS leadership. Following this meeting, further interviews were held with Ms. Nirasha Jayatilleke (on reporting), Ms. Chrishanthi Durairajh (on the functioning of the Programs Section), and with two people from the Finance section (on financial aspects of the projects).

Finally, a presentation was made to the Board of LEADS on July 26th, again to present findings and get feedback. All the feedback received was incorporated in the draft version of this report. The draft version of the report was sent out to LEADS and to the co-evaluators, and upon receiving feedback, the report was finalised.