Report of the joint evaluation of the Indonesian ECB consortium’s responses to the West Java and West Sumatra earthquakes
April 2010
Independent Evaluation by Pauline Wilson with the support of Budi Setiawan, Maria Josephine Wijiastuti, LeAnn Hager and Yacobus Runtuwene.
1. The joint evaluation team and process
The evaluation took place from 16 February to 7 March, 2010.1 Evaluation methods used included document review, key informant reviews, semi-structured interviews, focus group discussions, observation and interactive workshops.
The team was led by an independent evaluator. CARE, CRS and WVI provided staff to be a part of the evaluation team; two were from CRS and one was from each of the other agencies. Each person could dedicate only a limited number of days for the evaluation. Two staff members joined the team to do evaluative work in West Sumatra. One of them was involved in the West Sumatra response and the ECB consortium and the other was not. Both participated in an interagency workshop in Jakarta on 2 March and one continued to provide support in report writing after the workshop. One team member was a West Sumatra response manager and could only provide a few days to the team. The fourth member provided support during the first week of the evaluation in Jakarta and in design of the interagency workshop on 2 March.
Due to the more limited size of the team than planned originally and the level of knowledge2 of team members about the response, the evaluation focused its attention on assessing the effectiveness of the consortium’s actions to improve the speed, quality and effectiveness of the West Sumatra response. The West Java response was used for comparing the effectiveness of the consortium’s actions together only. No documents on West Java were reviewed and a visit was not made to response programs underway by the ECB agencies there. Views described on West Java are based on interviews with respective ECB member staff, including the Directors of member agencies.
A field visit was made to West Sumatra. Due to time and staff constraints only a snapshot is provided of the response by these agencies in West Sumatra. While seven ECB member agencies responded in West Sumatra, this snapshot primarily focuses on the work of five ECB member agencies that participated in an Office of Foreign Disaster Assistance (OFDA) funded emergency response program. These agencies are CARE, CRS, Mercy Corps, Save the Children and WVI.
Each of these five agencies selected a village for the evaluation team to visit. These five different villages were in two districts that were severely affected by the earthquake. Criteria for village selection were severity of earthquake effects; inclusion of the village in the OFDA funded program and the distance of the village to within a two and half hour radius of Padang City.3 Two members of the evaluation team worked together and visited two agency programs and the villages where they worked. To reduce bias these staff did not visit their agency’s program. They also conducted interviews with external agencies. The lead evaluator worked with a Minang–English translator and visited the programs of three member agencies, spending a day in a village served by a specific agency.
1 Please see Annex 2 for a detailed schedule of the joint evaluation. 2 There was only one staff person on the evaluation team for two weeks with in-depth knowledge about the emergency response programs by the agencies in West Sumatra and the ECB. This person was the CRS West Sumatra Program M&E officer. 3 The city of Padang is the capital of West Sumatra. Prior to visiting each village, the agency implementing staff provided a program brief to the team. At village level focus groups were conducted with men and women separately that had benefited under the OFDA funded program. A key informant interview was conducted with the village leader and random interviews were done with village residents. In addition team members walked around the village to observe housing conditions, the sector most severely affected by the earthquake.4
The rest of the evaluation process concentrated on hearing the views of staff on the response and the ECB consortium. This process included in-depth discussions with each director of the eight ECB member agencies. Individual discussions were conducted with three ECB project management team (PMT) members and the ECB Indonesia field facilitator (FF). An interagency workshop with emergency response staff of six ECB member agencies was held in Padang, West Sumatra on the 20th of February. The workshop focused on what was done by the consortium and how it affected the speed, quality and effectiveness of the response by each agency. Emergency response staff from the seventh ECB member agency that was responding in West Sumatra was interviewed separately in Jakarta. In addition, two members of the ECB Global Project team based in the United Kingdom were interviewed.
Key informant interviews were conducted with the following external agencies: OFDA representatives in Jakarta, a local NGO partner in West Sumatra, government officials of Padang and Padang Pariaman districts, two staff of the shelter cluster and an INGO partner that had provided technical assistance on transitional shelter in West Sumatra to five of the ECB member agencies.5
Information from all sources was summarized and the findings presented and reviewed during an interactive workshop on 2 March. The workshop was attended by seven ECB member agencies. Workshop participants drew conclusions and prioritized recommendations. A draft report was then written that captured the findings and conclusions from all these processes. The draft report was reviewed by consortium members in Indonesia. This final report takes into account the feedback received from agencies in Indonesia. The report is primarily for learning purposes and secondarily for accountability purposes.
The main criteria6 the evaluation team used to focus the evaluation were: Coverage by all ECB agencies in West Sumatra and the types of people served. The appropriateness of the activities carried out. The quality of the response in terms of how agencies involved and informed beneficiaries about their programs. Effectiveness in terms of the speed of the response, the needs assessment process and the use of standards. The immediate impacts of the programs. The effectiveness of coordination between member agencies and with other partners and the GOI.
4 Please Annex 3 for a summary of those we met at village level. 5 Please see Annex 4 for the names of all those who participated in the evaluation. 6 Definitions of most criteria, except that for impact, are from the ALNAP Guide on Evaluating Humanitarian Action Using the OECD – DAC criteria. The definition of impact used is from the ECB Good Enough Guide.