Beneficiary satisfaction survey in and – Final report Floods operation (MDRBA003)

Background

The survey was conducted between 31 March and 3 April 2009 among the people assisted through the DREF operation MDRBA003 after the floods in January‐February 2009. Simultaneously another survey was conducted among the people assisted through another DREF operation (Cold wave ‐ MDRBA002) in . The two survey teams worked closely together exchanging ideas and experiences before, during and after the surveys. The areas covered included the following locations: Glumina, Zalenikovac, Medugorje, D. Zalenikovac and Ravno in the Federation of Bosnia and Herzegovina and and in the Republika Srpska.

Team members: The two teams consisted of regional disaster response team (RDRT) members as well as two people from the planning, monitoring, evaluation and reporting (PMER) team in the Europe zone office in Budapest.

SFloods operation survey report urvey team members for the floods operation were: Emir Krehic ‐ Operations manager in Federation office in Sarajevo and RDRT member Alen Kajtaz ‐ Mostar branch secretary and RDRT member Diana Szasz ‐ PMER officer in Europe zone office

This report has been produced by joint efforts of the team members.

The main purpose of the survey was to find out what the people reached with Red Cross assistance thought about the goods and services they received.

Preparation phase ‐ Organization of the mission: The reference paper for the mission and the plan of action for the surveys (both attached in annexes) were prepared jointly by the two teams with the former one initiated by the office in Budapest, the latter one drafted in the office in Sarajevo and coordinated by the head of the Federation office in Sarajevo, Samra Campara who also played a vital role regarding the practical arrangements and the overall coordination of the implementation of the two surveys.

Most preparations have been done prior to the arrival of the two PMER officers (one in each survey team) from Budapest to Sarajevo. Each member had the possibility to provide input during the preparation phase.

Purpose of the survey

1. To find out whether beneficiaries were satisfied with the goods and services received during the operation and whether they have any recommendations for improving the quality and appropriateness in the future 2. To expand the knowledge base, identify best practices and provide concrete recommendations for the further development of the beneficiary satisfaction tool 3. To enhance the visibility of the International Red Cross and Red Crescent Movement, especially the National Society 4. To contribute to the final report

Main activities carried out: Survey team meetings, meetings with stakeholders, field visits and data gathering (interviews, observation), data analysis, survey report writing. For more details please see the plan of action in annex 2.

Page 1 of 8

Scope of the survey

The survey questions touched upon issues of relevance/ appropriateness of goods and services, timeliness, information dissemination and visibility. Most of the questions in the questionnaire (see attached in annex 3) can be classified in one of these categories. Therefore this report will present the findings and conclusions based on these four main criteria.

The survey was conducted between 31 March and 3 April 2009 among the people assisted through the DREF operation MDRBA003 after the floods in January‐February 2009. Geographic scope: The areas covered included the locations of Glumina, Zalenikovac, Medugorje, D. Zalenikovac and Ravno in the Federation of Bosnia and Herzegovina and Trebinje and Popovo Polje in the Republika Srpska.

Context

Status of the operation at the time of conducting the survey: The operation was postponed due to some misunderstanding and lack of defined roles, therefore the operating timeframe was extended until 17 April, and the distributions took place only 1‐2 weeks before the mission. The number of families reached with assistance was around 765 families.

Methodology

Methods: the methods used for data collection included mainly individual and household interviews and to some extent observation. However as a result of lack of clearly defined roles for the team members during the interview situation, observation was rather conducted as an ad hoc element. If resources allow, it is advisable to designate one team member whose main task would be observation, using a detailed observation form with previously defined criteria.

The data collection was based on a questionnaire. Data analysis was carried out by team members continously, and on the last day a final analysis was done together with the other survey team during a joint meeting.

Selection of beneficiary sample: Identification of beneficiaries and locations: A prescreening has been done by the local branches to assess the people’s ability and willingness to participate in the survey. This prescreening was rather ad hoc, and people who were considered to be interesting in regard to the survey were identified as potential interviewees. When it comes to the sample size the team did not aim to reach a scientifically representative sample, but rather focused on getting a good enough impression from the group of people interviewed. Selection of beneficiaries representing all groups of the assisted population: this was not really taken into account, other considerations prevailed, like selecting people who are on the way with little detours, thus ensuring time efficiency.

Survey team dynamics: The roles and responsibilities had not been clearly defined among team members prior to conducting the interviews. It would have been advisable to agree on who will be the focal point during the interviews, and what each person will focus on in order to have the best outcome and to obtain a sufficient level of information from the people interviewed.

Page 2 of 8

Validity and reliability of the data collected: It is crucial to pay attention to the way questions are asked in order not to influence the answer that people give, but to let them speak for themselves. This has to be clear for other stakeholders accompanying the survey team (local branch secretaries or volunteers) who might possess the information to answer some of the questions. It is important to remain open to get a different point of view from the people assisted, and not to look for only a confirmation of the information that is already available.

Meeting with stakeholders and their involvement in the implementation of the survey: The survey team had the opportunity to meet secretary generals of local branches, which was a good opportunity to listen to their points of view on the operation and added value to the the overall picture. When arranging such a meeting, and if the branch secretary or volunteers are joining the team for the interviews, it is important to have the possibility of sitting down quietly and talking through a few issues before actually going to the field to do the interviews.

Findings, conclusions and recommendations

1) General conclusions

The biggest value of conducting the survey was that it shifted the emphasis of the analysis from output or outcome level indicators to the voices of the people assisted and managed to emphasize the very essence of the Red Cross work which is improving the lives of individuals. It is important to have the voices of the people heard not only during the initial assessment but also in the following phases of the project cycle.

Apart from obtaining information on the way people felt about the goods and services in terms of appropriateness and timeliness, anecdotal information and individual stories can add value to the analysis and can put the people reached in the focus of the reports and in the focus of our thinking again. After all, the operation was done to address their needs.

The conclusion of the two survey teams during the joint meeting on the last day was that it is highly recommended to make this kind of survey a routine after emergency operations and therefore to include the corresponding costs in the budget.

Although surveys might be laborious and costly, they provide valuable information on the overall success of an operation. It is important to not only provide assistance to the people but to also get their feedback about it.

It also has a value in itself to visit some of the families in difficult conditions and talk to them a little bit, it makes them feel that the Red Cross/ Red Crescent cares about them.

An inhabitant of the village Glumina is not only taking care

of his disabled mother and his sick cousin but also helping

out his neighbours in need. Photo: International Federation

2) Findings, conclusions and recommendations regarding the DREF operation MDRBA003

Page 3 of 8

During two days of field work, some 18 families were interviewed. This number could be increased if there was at least 3‐4 days available for field work. When making an estimation of the number of people the team would like to reach, it is important to take into consideration the time necessary to reach some of the remote villages in the mountains.

Table 1: Number of families visited per locations Number of families During data collection it was noticed that there was substantial Location interviewed difference in the way people were affected by the floods in the Glumina 5 remote villages in the mountains and in the towns of Trebinje Zalenikovac 2 and Popovo Polje. In the former area people’s lands were Medugorje 1 damaged but the flooding did not reach their houses, whereas D. Zalenikovac 1 in the latter one mainly people’s houses were flooded. Realising this difference in their situation led to the decision of Ravno 3 analysing the data seperately and presenting the findings and Trebinje 4 conclusions separately in this report for the two areas. Popovo Polje 2

a) Findings and conclusions regarding the remote villages in Herzegovina:

Out of the 12 families visited (middle‐aged/ elderly people working on their lands), all families mentioned that their land was damaged and that they lost all crops. They mentioned that they had no fodder to feed their animals anymore (mainly cows, some goats and sheep), so some of them had to sell their livestock. They had no seeds to be replanted. Four families mentioned that they lost their wine stocks, and one family mentioned olive trees damaged by the floods.

When asked about their needs, five families mentioned that they needed seeds, seven mentioned fertilizers to prepare the land for the autumn seeding season, and two families mentioned that they would need clover to feed their animals. Four families also mentioned the need for wheat. When it comes to replanting, one of the people interviewed was on the opinion that it was already too late to replant the seeds and that they would rather need to focus on preparing the land for the autumn planting season.

They all knew that they received the food and hygiene parcels from the Red Cross, and they said that only the Red Cross asked them about their needs.

All families interviewed confirmed that the assistance was provided a week before the survey, which was around 40‐45 days after the floods.

They all received the assistance at home, so the questions around the details of the distribution were not relevant. All of them also confirmed that the local branch informed them about the planned distribution.

All of the families except one confirmed that the volunteers were wearing an emblem when they distributed the food and hygiene parcels to them, and they also confirmed that the behaviour of the volunteers was good. They also know how to make a complaint if they are not satisfied with the assistance provided. When asked about any changes they would recommend in regard to the operation, one family said that they would like to see seeds and fertilizer or money being provided, but the most important would be the fertilizer. Another person said that the Red Cross was a lot more active right after the war, then it is now, whereas the needs have not decreased since then.

Observation: one of the first people interviewed seemed to be very frustrated, and even said that he would have refused the Red Cross assistance. This reaction however cannot be unequivocally attributed to his

Page 4 of 8 attitude in regard to the Red Cross assistance, but is probably the result of several factors, including the fact that the state did not help him, and that being a hard working person he is now in a position to receive external help.

Conclusions:

Appropriateness/ relevance: People mainly mentioned other needs than those addressed with the food and hygiene parcels. Although the Red Cross assistance has temporarily improved their situation, it did not reduce the damage which the floods had on their lands. These families could not replant their seeds, and will not have anything to harvest. Timeliness: Although the people interviewed were not asked to judge whether the assistance came in time or too late, it is quite clear that providing assistance to the people 40‐45 days after the floods cannot qualify as immediate disaster response. At the same time the fact that the items distributed did not address the primary needs of the people downplays the importance of the timeliness of the assistance. Information dissemination: There seems to be a very close relationship between the Red Cross and the local communities. The local branch secretary accompanied the survey team to the villages, and it was observed that they knew each other well, were in regular contact with each other. People also seemed to feel comfortable about complaining to the local Red Cross if they had any problems. Visibility: Volunteers were wearing an emblem during the distribution of assistance.

The two problems identified in regard to the operation are its relevance to the needs of the people and the delay in providing assistance to the people. The Red Cross Society of Bosnia and Herzegovina and the International Federation will have to work jointly on finding ways to improve decision‐making in the future in order to provide timely and relevant assistance to the people in need.

b) Findings and conclusions regarding the towns of Trebinje and Popovo Polje

During the six interviews conducted very different problems and needs were mentioned by the people assisted through the DREF operation, therefore it is difficult to draw clear conclusion on the appropriateness of the Red Cross assistance.

Most houses were located on the bank of the river or close to it, where it has been prohibited to build houses, and the houses and gardens were flooded. Four families mentioned that they had lost their crops, two of them managed to replant some of the lost seeds. The water stayed for around 15 days in the houses, damaged the furniture, food reserves, in one case even the windows. One family mentioned that they had no electricity for 20 days, and that they were cold.

When asked about their needs, two families answered that they needed food, another said that they needed everything including food, and another family although not saying explicitly that they needed food but mentioned that it was important to receive flour. Transport was mentioned by five families as a significant problem, they had no means of transport for weeks, children could not go to school, one family said that they could not reach the town for two weeks. One person mentioned that he would have needed seeds and fertilizers but also said that it was already too late for seeds now. One of the people suggested an emergency escape road to be built which could also allow of neighbours helping each other in emergency situations.

Besides the Red Cross asking them about their needs, three families mentioned that the fire brigade came to help them and to ask them about their needs, and four families mentioned that civil defense visited them. One person said that he knew that Red Cross would come and help.

Page 5 of 8

All families confirmed that they received the Red Cross assistance two weeks before the interviews, which means some 30‐35 days after the floods, and one family said that it could have come earlier and that she would recommend to provide assistance earlier in the future.

When asked whether they knew who funded the Red Cross response, one family said that the Red Cross of Bosnia and Herzegovina and the International Red Cross, another family said that they knew funding came from the Red Cross family, and that is all they needed to know.

It became apparent that volunteers were not wearing a visible emblem when distributing the food and hygiene parcels, and this was confirmed by a volunteer who was accompanying the team during the interviews.

All families received the assistance at home, so the questions around the details of the distribution were not relevant.

Conclusions:

Appropriateness/ relevance: It is difficult to draw a clear conclusion on whether the assistance met the needs of the people reached. Four families out of the six mentioned the need for food, so this can be considered an appropriate assistance. Hygiene items were not mentioned as something they needed or as something that turned out to be useful for the families. Also there was one family who said that this assistance was just a nice gesture from the Red Cross and could happen more often. But this very same family did not answer the question about whether they needed anything after the floods, they said they were born there and learnt how to handle these kind of situations. Another interesting finding which is in line with the fact that the Red Cross used to be very active after the war, is that one of the people interviewed said that he actually expected the Red Cross to provide assistance after the floods. It was also confirmed by a volunteer that some people do appreciate the assistance, others not so much, and that actually people are expecting quite a lot from the Red Cross. Most probably in the case of these two locations (Trebinje and Popovo Polje) the number of families interviewed do not allow of getting a good enough impression on the relevance of the operation. Not everyone was at home when the visit took place, some of the people were probably working on their lands. Besides that it has to be mentioned that the follow‐up questions to the answers of the people were also limited as a result of lack of clearly defined roles and responsibilities between the team members. Timeliness: The assistance by no doubts arrived very late. Information dissemination: There were no issues identified in regard to information dissemination. Visibility: The volunteer accompanying the team said that sometimes they do wear emblems and sometimes they do not. It is rather on an ad hoc basis. However the emblem was on the car that the volunteers were provided with for the distribution.

The two main issues identified in regard to the operation in these locations were around timeliness and visibility. It is recommended that, in order to ensure a more effective response next time, the Red Cross Society of Bosnia and Herzegovina and the International Federation work jointly on finding ways to improve decision‐making. Regarding visibility, it is recommended to make sure that whenever staff or volunteers are going to the field, they are provided with some kind of visible emblem or a uniform, and they hand it back after the field work to the local branch.

3) Recommendations to improve the tool

Getting familiar with tool materials and previous survey reports: It is essential that all team members are familiar with the tool, previous survey reports and the questionnaire itself, and that they have a good

Page 6 of 8 understanding on the information we are seeking with each question. It is recommended to go through the questionnaire together with the whole team and to discuss in regard to each question what is the thinking behind its formulation. This meeting can be combined with the definition of the roles and responsibilities of the team members.

Defining the roles and responsibilities of the team members during the interviews: It is essential to agree on who will be the focal point during the interviews and what each person will focus on in order to have the best outcome and to obtain a sufficient level of information from the people interviewed. If resources allow, it is advisable to designate one team member whose main task would be observation, using a detailed observation form with previously defined criteria. The same person can be responsible for taking photos.

It is also important to make sure that the person actually doing the interview and ensuring that the answers are noted can engage in a more informal conversation with the people being interviewed. Some people have the necessary skills for that by nature, others might need to acquire these skills. Showing interest in the life and problems of the people assisted is necessary if we want to get their honest feedback on the work of the Red Cross Red Crescent. This person conducting the interview needs to speak the local language, in order to be able to have some informal chatting with the people, and because continous translation makes the interviews longer and less comfortable for the respondents.

Validity and reliability of the data collected: It is also crucial to pay attention to the way questions are asked in order not to influence the answer that people give, but to let them speak for themselves. This has to be clear for other stakeholders accompanying the survey team (local branch secretaries or volunteers) who might possess the information to answer some of the questions. It is important to remain opened to get a different point of view from the people assisted, and not to look for only a confirmation of the information that is already available.

Other stakeholders involved in interviewing: If the branch secretary or volunteers are joining the team for the interviews, it is important to have the possibility of a short briefing with them and to talk through a few issues before actually going to the field to do the interviews. If there is a possibility to involve volunteers in conducting the interviews, they also need to be briefed. It is recommended to translate the questionnaire, which is actually quite short. Involving volunteers could strengthen the local ownership of the process and could also enhance visibility. The people assisted by the Red Cross Red Crescent also feel more comfortable if people from the National Society are conducting the interview rather than Federation staff. Although Federation representatives can take part and be a member of the survey team, it is recommended that the focal point for the interviews is someone from the National Society.

Visibility issues: If there is time to arrange a meeting with the authorities, it can strengthen the visibility of the Red Cross consulting the people it assists. It is also important that survey team members as well as others accompanying them (local branch secretaries, volunteers) are wearing visible emblems when going to the field.

Availability of beneficiaries: The team has to make sure that people are at home when the team is planning to visit them.

Recommendations to amend the questionnaire:

The questions around the distribution are not relevant when distribution takes place at home, therefore it is recommended to have these questions as sub‐questions which are only asked if the distribution takes place at a distribution point.

Page 7 of 8

The team has to be careful during each interview not to ask too many questions from the people as this can lead to ‘survey fatigue’ among respondents. It does require the interviewer to pay attention and to monitor the willingness of the person to talk and to answer to the questions.

The question inquiring about whether the people assisted know who funded the operation does not really provide us with valuable information. People usually know that the assistance came from the Red Cross or Red Crescent and they are not so much interested which donor National Society or government replenished the DREF. What is important for them is that the Red Cross Red Crescent Movement provided them with help. The inclusion of this question in the questionnaire has to be revisited.

Another question which needs some clarification regarding the kind of information it seeks to obtain is the following: ‘Have you received any information about the work of the RCRC since the disaster happened?’ The question was not relevant in the current case, since the people received assistance 1‐2 weeks before the survey, which meant that they did not receive any information about RCRC work since then. However even if the question is relevant, and the answer is yes, it is quite uclear what kind of information we are talking about here. Is that information on other programmes of the RCRC? If yes, how valuable it is if some of the elderly people in the remote villages know about the work of the RCRC in other parts of the country? Shouldn’t we aim to be effective in our work and target the right people with the right information and messages instead of providing information to everyone on everyhting? What does it tell us if the answer to the question is yes and if the answer is no? It seems to be difficult to analyse tha data we can collect with this question.

It is recommended to include the above mentioned issues as practical suggestions in the survey tool. This would help ensure that they are considered when preparing the surveys and that effective data collection can be done.

Use of this report

The report is going to be shared with the Red Cross Society of Bosnia and Herzegovina and will be disseminated within the International Federation to advance organisation‐wide learning, and to contribute to the improvement of the quality of services provided by the Red Cross Red Crescent. It will enrich the collection of good practices by putting the people assisted in the focus of analysis.

The report will also contribute to the final report on the DREF operation MDRBA003.

Page 8 of 8