Report of the External Evaluation of the Echo Funded Save the Children Dprk Programme

REPORT OF THE EXTERNAL EVALUATION OF THE ECHO FUNDED SAVE THE CHILDREN DPRK PROGRAMME.

SEPTEMBER 14TH - OCTOBER 3RD 2005

ALAN ETHERINGTON (WATERAID ADVISORY SERVICES) AND EMMA ROBERTS (INDEPENDENT CONSULTANT).

Table of Contents

Page 2 Table of Contents

Page 3 Introduction

Page 4 Objectives of the evaluation

Page 4 Methodology

Page 6 Findings of the Evaluation Team

Page 6 Water and Sanitation

Page 13 Health

Page 18 Procurement, Warehousing and Logistics

Page 19 Recommendations

Page 21 Conclusions

Page 22 Appendix A – Terms of Reference for the External Evaluation

Page 26 Appendix B – Timetable for Evaluation

Page 27 Appendix C - List of key people met during the field trip.

Page 28 Appendix D - Project Spreadsheet

Page 29 Appendix E – Details of Ri Kits and Midwifery Kits.

A. INTRODUCTION

Save the Children UK (SCUK) has been operational in the Democratic Peoples Republic of Korea, (DPRK) since 2003, but had been engaged in DPRK for a number of years previously through the secondment of technical staff to UNICEF. Since arriving in DPRK, SCUK has implemented two programmes supported by ECHO focusing on the needs of children and their families through support to institutions, winterization projects and the provision of winter clothing and a small pilot water and sanitation project and through these various initiatives has begun to develop good experience of working in DPRK.

In October 2003, SCUK was invited to undertake a needs assessment in Tanchon County, South Hamgyong, with a view to working with the City Management Committee (CMC) in the implementation of the project addressing the basic health needs of the communities of Tanchon County. Following the assessment a proposal was submitted to and approved by ECHO for a 12 month project aimed at “Improving primary health care access and quality in South Hamgyong and North Pyongan, Provinces, DPRK.”

Within the project designed was provision for an external evaluation to be carried out at the end of the project to ensure that the project objectives were met and to assist in the learning process for the project staff for the design of future work in South Hamgyong.

The evaluation was planned for the last two weeks of September 2005 and two consultants, a water and sanitation specialist and a health professional with previous experience of working within DPRK were employed by SCUK to undertake the external evaluation.

The following report includes a brief look at the key objectives of the terms of reference and then the methodology used for the evaluation. There will then be a consideration of the water and sanitation work and whether or not those objectives have been met and then the report with continue with a consideration of the health impacts.

There will then be a summary of the main findings and a list of recommendations for future work. While not for the scope of the evaluation, it cannot be ignored that there are currently discussions taking place concerning the future of all humanitarian assistance to DPRK and the future is very unclear with the possibility that SCUK and all other NGOs will be leaving DPRK at the end of 2005. However the evaluators have made recommendations with the hope that there will be an opportunity in 2006 for future work.

B. OBJECTIVES OF THE EVALUATION.

The purpose of the external evaluation, as stated in the Terms of Reference, agreed by SCUK and ECHO was to “determine the effectiveness of the project in achieving its broader objective of improving the health of children and their mothers in targeted communities”

a) to determine whether the project has been successful in achieving its objectives and outputs established in the project log frame b) determine how far the project has been able to address the issues of sustainability of the installed facilities through local capacity building initiatives c) to determine the effectiveness of the project in addressing the issues outlined under risk and assumption in the project document d) determine the capacity of the SCUK project team in planning, implementing and monitoring the project progress and activities e) access to quality health services being one of the key issues in rural DPRK; recommend the effective ways to replicate the successful approaches in other Counties and Provinces f) to advise on possible changes to project approaches for development of future interventions (see Appendix A for full Terms of Reference)

C. METHODOLOGY

The evaluation was carried out using a variety of methodologies. Before arrival in country the consultants were provided access to a number of project documents and received a briefing in China from the SCUK Programme Director. On arrival in Pyongyang, each consultant received a briefing from the relevant team members before departure for the field.

During the field visit to Tanchon County, South Hamgyong, the team were briefed by the City Management Committee as the local counterparts, visited the warehouse and met the warehouse manager and were able to make visits to four rural communities (ri’s) and one urban community (dong) to view the work that had been done and to talk to the village leaders and also members of the communities.

During the site visits the team were able to view much of the water infrastructure that has been built and talk to those trained to maintain it. Also visits were made to houses to view the water provision and the latrines. Discussions were held with representatives of the mothers groups and also the staff in the health centres and the institutions.

Throughout the trip good access was given to the team and all those met were extremely helpful in answering the many questions. Before leaving Tanchon, a debriefing was held with the counterparts including the health counterpart which was helpful to discuss what had been seen and the areas that the team were considering for recommendations for future work.

On the return journey, the team met with the Head of the Provincial FDRC in Hamhung to hear his opinion of the work that SCUK has been carrying out. (for full details of the visits undertaken and the key people met, please see appendix B and C)

While the project includes work in two provinces, Tanchon County in South Hamgyong Province and Yomju and Unjon County in North Pyongyan Province, time constraints only allowed for the visit to one county due to the distance from Pyongyang to Tanchon County. However the majority of the project work is undertaken in Tanchon County and therefore this was a right prioritisation for the team and allowed for the best understanding to be gathered.

In Pyongyang, the team met with UNICEF, the Federation of the Red Cross and ECHO to gain a better understanding of the situation. Unfortunately, other NGO staff working in the water and sanitation area were not available.

A final briefing was held with the team during which feedback was given and the recommendations explained which allowed for all those involved in the SCUK team to discuss and have a good understanding of the recommendations.

D. FINDINGS OF THE EVALUATION TEAM.

1. WATER AND SANITATION. (see appendix D for project spreadsheet)

Overview and main conclusions

• All water and sanitation targets of the Log Frame (LF) have been met or exceeded; • A continuous supply of water and adequate hygienic toilets has been provided to 16 schools, 6 clinics and 1 hospital in 7 communities • 5 (Gravity Flow) GF systems and 2 mixed GF and pump systems were built during this phase and within the 7 months construction season to a high standard of design reflecting the GF practices developed in Nepal – which has possibly the largest number of GF schemes in the world • with good cooperation from the local CMC partner, the project has been able to deliver water to 100% of the families in the area around the clinics and other institutions - almost 2500 families or about 11,000 people; the LF target that 10% of the houses surrounding clinics, nurseries and other institutions should receive water was thus overly cautious; • The GF water supply approach is only a few years old in DPRK and the nine built during this and the earlier phase (2004) are a local innovation for Tanchon county. Similarly, this model of compost toilets has been used for only a brief period in the country and those introduced by SCUK are again a local innovation for Tanchon. Their enthusiastic adoption by Tanchon CMC reflects the apparent utility of these innovations, their cultural appropriateness and a high degree of trust built between the development agency and local partner; • These innovations are important because these are technologies with the potential to work in large areas of DPRK and significantly improve daily lives of women and children; projects such as SCUK’s take these theoretically appropriate technologies and provide practical experience and demonstration required for these to go to scale • The project has been imaginatively opportunistic in expanding its scope of work (2 additional projects were completed at a cost of Euros 26,000), in building excess capacity for future expansion or extension and mobilising a few additional resources from WHO and DFID to undertake additional work in Tanchon; • Project progress has been impeded by contractors providing below-quality supplies or equipment, or the wrong pieces; • The project has helped Tanchon CMC to upgrade their warehouse infrastructure and also introduced a tally card system for the approximately 130 different items that require being stored; the warehouse official has also been trained and supported; this represents a useful investment that can be used by the CMC for future construction and other activities.

Water Supplies

• The innovation of gravity flow water supply systems have transformed domestic and institutional water supply to be more reliable, improved and sustainable • The average direct cost of providing one family with water over all seven schemes is about 95 Euros – about 21 Euros per person, a figure which is line with GF schemes in other countries elsewhere in Asia (direct cost includes only the materials and transport provided by SCUK and excludes local costs and SCUK project overheads) • Average per family direct costs for water supply ranges between different schemes from a low of 30 Euros to a high of 137 Euros; this range reflects the significant variation in distance to source, existing infrastructure, cluster size, etc which can determine total and average cost per family; it is worth noting, however, that in some countries it is those projects with a high per capita cost that reach the poorest who tend to reside in more remote locations; • Sustainability has been well addressed by training technicians (an operational workshop for 15 days covering both theory and practice), involving them in design, local involvement by communities in trenching and construction, providing tool kits to technicians and buying a “buffer stock” of spare parts that should provide enough support for a few years; • The system for Water Supply at all clinics is somewhat independent of the system for the remainder of the community, which should ensure clinics are never without water; • 10% of families received two water taps - who are these families? How are they selected?

Sanitation

• The innovation of twin pit VIP compost latrines have transformed domestic and institutional defecation management to be more hygienic and dignified without loss of traditional compost; traditional toilets provide privacy but do not hygienically contain the waste, thus opening them up to fly-borne disease transmission; the removal of the waste every few weeks to place on compost piles or onto fields or gardens further exposes humans to serious health risk; • The SCUK project, with other NGO projects and other health agencies, have succeeded in raising the profile of domestic sanitation as an important development issue; • Replication toilets have generally copied the design of the demonstration toilets to a meticulous level • the pace and extent of latrine replication has exceeded targets – 195 will be built before winter 2005 compared with a target of 60;

7 • SCUK has used a water seal toilet in delivery rooms, which thus allows women in delivery to stay indoors; • The unit cost of model latrines is estimated at 125 Euros and for replication latrines at 105 Euros; these costs have been divided to date as follows:

Latrine unit costs by demonstration / replication and cost allocation (Euro)

SCUK CMC / Ri / users total demonstration 85 40 125

replication 40 65 105

• 15% of families have received a toilet so far – as far as possible these were selected on the basis of getting leaders and key people in the project to be using these, so that they would know from first-hand experience their value; these included all five members of the mother’s group, the water technician as well as the village leader; members of the mother group were selected from all parts of the community before the allocation of latrines was made; • One cluster of Paekson - Ri is on schedule be a totally sanitized community before the winter with all families and institutions using a hygienic latrine. This, together with proper water supply and practicing the improved hygiene now possible, should demonstrate the reduction and possibly elimination of diarrhoea-related disease.

Water and sanitation

• Within the target communities, all clinics, nurseries, schools and one hospital have continuous running water and hygienic toilets reducing or removing the risk of diarrhoeal diseases and enhancing the daily quality of life for children, staff and patients; toilets have been designed to be child – friendly (i.e. appropriately sized, well lighted and an appropriate mix of privacy and access). • Design standards of one tap in the household delivering 60 litres per person per day and one toilet per household are in line with the aspirations of other nations with a similar socio-economic level (e.g. Eastern Europe, Central Asia) and far superior to those commonly installed in similar projects in South Asia and sub-Saharan Africa; • 15% of families (380) have received both water and sanitation within the project area to date at an average cost of about E 220 – about E 49 per person. • Overall construction and workmanship is excellent; minor problems such as shallow trenches and incorrect pipe sizes were identified and corrected;

Impacts

• These innovations have transformed domestic hygiene; mothers group members identified six important ways: 1. time and energy spent on water collection, mostly by women and children, has been eliminated – saving approximately one hour per day per household as well as 2. the frequency of bathing has increased from once a week to, during summer, a daily activity; perhaps even more frequently for some children who can now bathe both at home and at school or nursery; 3. the quality of water for drinking and cooking has improved as sources and delivery systems are now better protected 4. laundry washing, kitchen and home cleanliness are easier, thus done more frequently and to a higher standard 5. defecation now occurs in a clean, odourless, insect – free and stable location and faeces are hygienically contained; parents can assist their small children to use the new toilets until they are old enough to use them alone; 6. the management of human waste no longer requires moving raw waste to compost piles or kitchen gardens every few weeks but will be done only once a year when the waste is a hygienic , safe and relatively pleasant material

• As all schemes had only been in operation for a month or so, it was premature to evaluate usage or impacts but good public health benefits can be anticipated so long as new facilities are used as designed • Any subsequent evaluations should focus on system functioning and people’s proper usage – not health. This is in the approach advocated in the Minimum Evaluation Procedure (MEP) developed by WHO and others. This is based on the arguments that measuring health impacts is expensive to do well and that there already exist a body of rigorous studies that demonstrate health impacts from providing improved watsan facilities and their proper use in a wide range of different contexts. • Having mothers groups monitor health impacts such as diarrhoeal incidence, however, is important for other reasons: for example to identify families where help is required to ensure good hygiene is practiced, to raise awareness and to provide popular understanding of basic public health

Innovation

CMC and local leadership have embraced these innovations, displaying a high degree of skills in design, planning and construction.

The high rate of adoption of these innovations can be attributed to:

9 • The appropriateness of the innovations – not dependent on electrical power supply; relatively easy to maintain and thus attractive to users in the current reality of uncertain power supply; • The good inter-personal relationships established between SCUK and partner staff • The study tour visit to see similar water supply technology in Tibet at an early stage of the project, followed by a year of modest target – just 2 schemes- now followed by 7 schemes. • The high level of technical skill and knowledge of SCUK staff • The willingness to consider new approaches to solving public health problems displayed by CMC and their high level of responsibility and proactive and energetic take-up • The high level of mutual trust built up between both sides. The selection of Nepalese staff by SCUK was an inspired decision, as they come from a sister Asian country with no history of imperialism and possess a huge experience of building thousands of gravity flow water supplies over the past 3 decades;

The local Partner and partnership

• The local partner has completed seven projects in seven months; they have delivered on their commitments. • A good partnership has been established, that augers well for future joint work in Tanchon. • CMC has now allocated a 2 room office to SCUK and local staff in their office – a testimony to their trust in SCUK and their continued support to the project’s future phases.

Issues for further consideration

Water

1. Spare parts and adequate power for the two schemes that include water pumps 2. Spare parts for GF schemes after the buffer stocks run-out; 3. Can we encourage meetings with all GF system technicians to share experience, assist with problem-solving etc? 4. Large areas of the country are suitable for GF schemes – it would be useful for agencies and Government to document their design processes, develop standards and ensure exchange of best practices; 5. Similarly, it is important that SCUK and other NGOs and agencies provide a full listing of all schemes and their design features to local partners as well as a monitoring protocol. This should provide the basis for future assessments and help ensure these initial investments continue to provide a service over many years.

10 6. Wastewater management – providing 24 / 7 water in the house usually results in a massive increase in water consumption, compared with fetching water from a source away from the house. This means that wastewater must be managed, through additional infrastructure such as soak pits, or better drainage.

Sanitation

7. Management of the eco san toilets – will users wait for composting to take place? 8. Decomposition of human waste – will the combination of winter and a faeces / urine mix allow decomposition to occur in 12 months? 9. Public health benefits require total sanitation – will toilets be replicated to provide sufficient for each family to ensure all defecation happens in a hygienic manner? How long will this take? SCUK should use opportunities to argue that total sanitation coverage is important. 10. Costs of replication latrines – estimated to be 105 Euros per unit, serving average of 4.5 people – is this too expensive for unsupported replication? If so, can costs be reduced with design modifications? 11. Are infant faeces handled in a hygienic manner? – they are perceived to be less harmful than adult’s (but both WaterAid and SCUK think they are more harmful). Once we have confirmed this then the hygiene promotion should include this message. 12. As taps are usually installed in kitchens, will dirty nappies be washed there, or in the washing areas using a hosepipe of running water? Local partners need to monitor any risk of bringing contaminated material into food preparation areas.

Previous Evaluations

• The project had been included in two previous assessments. In October 2004, two staff from Concern NGO had reviewed outputs from phase one which had provided watsan facilities in 2 communities of Tanchon. All water points and latrines had been functioning. The evaluators had identified a number of useful innovations such as a gabion and installation of a clinic water supply outlet pipe below that of the main outlet. Unfortunately it was not possible for this current evaluation to visit the locations of the previous phase, which would have provided some sense of the short to medium term sustainability and usage. • The SCUK project was also included in a more comprehensive evaluation of all ECHO supported projects. This review described drinking water supply provision by SCUK and other NGOs as ‘ a large and successful area of ECHO support’ while noting shortcomings with waste water management and the absence of water quality data, observations with which we concur.

11 • The review was, however, very sceptical about domestic sanitation on the grounds that they are ‘not affordable to householders, making the interventions ineffective and unreplicable’. • We have considered this argument together with SCUK staff and while we accept that the unit costs are high, the current inclusion of a sanitation component is valid for the following reasons: o The project spends only a modest proportion – about 10% of its direct costs - on sanitation o As current domestic sanitation practice, using wooden outhouses provides privacy, any new latrine design must provide additional advantages for users to be motivated to construct them – the twin pit VIP compost design provides an approach that allows human waste to continue to be used for agriculture but in a much safer manner and for defecation to happen within a robust superstructure comparable to the materials and construction used in residences o Constructing them now could provide a proven and culturally appropriate approach for the future when economic or other conditions would permit their widespread replication o Even during the current economic stress, the current phase has shown some encouraging expansion with 195 replication units built during these seven months compared with 185 demonstration units built in the same period. We even noted some replication units built in one neighbouring Ri, in which the SCUK project had not worked, so we assume these were driven by internal dynamics about which we may only guess. In summary, therefore, we consider the latrine component to be worthwhile – a relatively modest investment for a potential significant benefit.

2. HEALTH.

When considering the project and whether or not it has met the objectives when concerned with health it is best to consider in turn each of the four results as laid out in the log frame and to see how far the project has met these results.

There will then be some general feedback on health matters from the visits and meetings and finally included in the recommendations there are a number of recommendations that specifically address the health matters.

It should be noted that all training and workshops have been carried out in collaboration with the Korean Family Planning and Maternal and Child Health Association (KFPMCHA) which has enabled capacity building to take place of KFPMCHA while introducing some new training methods, e.g. participatory learning, role plays and action planning. This relationship is a real strength of the health components as it enables training to take place by Koreans and enables the new ideas to be introduced smoothly. All training materials have been approved by the Ministry of Health and SCUK has drawn on the experience of UNICEF, WHO and Concern in developing the curricula used for the training.

Result one: ‘Rehabilitation work on the delivery rooms has been completed in a minimum of seven health facilities.’ • Visually seen four clinics with newly rehabilitated delivery suites which include a delivery room, a room for resting with a small kitchen facility to enable the family to cook for the new mother and also a bathroom with toilet and washing area and water 24 hours a day. Windows and doors ensure good winterization to enable the facility to be used in the winter months. Curtains provide privacy and improve the overall environment. • Clinic staff was involved in the design of their own delivery suites and are obviously happy with the improved facilities. • Staff have received training on “Perinatal Care” and clearly enjoyed the training and are keen to use their new knowledge. • Following on from the training equipment has been provided for the use of mothers who wish to deliver using more natural positions which will greatly enhance the overall environment for the delivery process. • The basic equipment in the delivery rooms remains very basic with a variety of beds and other equipment available. However Ri Clinic kits are due to arrive during October which will no doubt make a significant improvement to the overall care provided. (see appendix D for contents of the Ri clinic kits) • Positive feedback was provided to the evaluation team by the clinic staff and it is anticipated that the usage will increase.

13 Result two: ‘Rehabilitation of the water and sanitation provision for the health facilities has been completed in a minimum of 7 health facilities and their immediate communities.’ • Water in available in the clinics 24 hours a day with an independent pipe to each clinic to ensure that clinics have water even if the village supply has problems or demand exceeds supply although this is not anticipated • Latrines have been built in all the clinics – with separate latrines for men and women with hand washing facilities also provided. • Incinerators have also been built in each of the clinics – although unlike the other items that have been exact replications, not all incinerators have been built to the best specification which may impact on their effectiveness. • Mothers groups are up and running with hygiene promotion training provided to all groups. Each of these mothers has also received a demonstration latrine and water supply to their home. • In most communities the mothers have already begun to train other mothers and to share their new found knowledge. They are well motivated and keen to share their knowledge and understanding. They have generally understood well the information given to them and seem pleased with their role as hygiene promoters with one group explaining “We feel good about ourselves inside when we talk to other mothers.”

Result three: ‘Rehabilitation/renovation of a minimum of eighteen child welfare institutions, including water and sanitation rehabilitation has been completed’ • The kitchen areas in all targeted institutions have been rehabilitated and re-equipped with new doors and windows, flooring and stoves as well as bowls and cups and other cooking equipment. • The bathing areas have also been rehabilitated with the provision of water, taps and a bathing area for each targeted institution • Good quality of work was seen in almost all places although there was some concern that some of the floor tiling might be a little slippery – especially if children were playing in the bathing area. • Both the kitchen and bathing areas are much easier to keep clean and with a good water supply there is no doubt that food preparation will be more hygienic and the children will be bathed much more regularly • Training for carers on the keeping of kitchen areas clean and also the need for bathing etc has been provided • The children are being bathed more regularly – during the warm summer months – whether this will continue when fuel will be required to heat the water will require to be monitored. • Children are using the latrines and hand washing facilities but there is no soap available for hand washing or towels for drying hands or bodies after bathing.

14 Result four: ‘Improved capacity of nurses and midwives to target health facilities in the provision of mother and child health care.’ • There have been some difficulties developing the relationship with the School of Nursing including changes within the health personnel at the School and local partner level. This relationship there needs further development • Training materials – including a resuscitation “Annie” and in ‘injection arm’ have been purchased and are in transit. (currently in Pyongyang airport) • SCUK has taken the opportunity of the contact made and its presence in Tanchon County to undertake some winterization activities and the provision of mattresses, blankets and curtains and water and toilets for the School While these activities were not directly funded by the ECHO project it is hoped that the work done shows evidence of commitment from SCUK to the school of nursing and hopefully will help encourage the further development of the relationship with the school of nursing in the future. • Provision of new MOH/WHO approved curriculums has been delayed due to a lack of funding within WHO for printing. There may be opportunities in the future for this support to be provided if further printings are carried out in future years.

Issues for further consideration

Result one: Clinic rehabilitation: Each clinic has a delivery room and a resting area provided. WHO/MOH Ri clinic kits to arrive. It is too early to tell whether the numbers of deliveries has risen in the clinics since the rehabilitation work has been done. This will require some additional follow up work, as will all the monitoring of health impacts. One suggestion for the future is not to put a door with a large window on the indoor toilet to ensure privacy for the women.

Alongside the clinic rehabilitation was the provision of training to the midwives in the targeted clinics. There is a need to try and ensure that for future trainings all the midwives and nurses are encouraged to attend as it was found that some clinics sent their Directors and other health staff rather than those directly involved in the assistance of mothers in delivery. However it should be noted that by allowing Directors and others in charge of the clinic to also attend, a wider core of people will have the relevant knowledge and the Directors will be able to support their staff as they implemented the new procedures.

Result two: Water supply and basic sanitation facilities in the clinics have been achieved. One toilet was found to have a number of mosquitoes – indicating that the door has not been kept closed. This is an important message to ensure the community understands as with any community facility there will be a lack of individual

15 responsibility. Community education is important while some locally available closing mechanism might be considered.

Incinerators have been provided to the clinics for the burning of waste from the clinic. However it was not clear what was to be done with the non burnable rubbish and also the contents of the incinerator after burning. One incinerator seen did not meet the necessary specifications but otherwise the replication from an original model seemed to be followed well.

The setting up of the mothers groups and the training provided to them has been very well received. However there is a lack of clarity concerning the next step with some mothers groups already using opportunities to share their new knowledge and others waiting for more guidance. Therefore there is a need to put greater emphasis on the action planning process part of the training – to equip all groups with a clear plan for the next stage.

Result three: Institutional support: In the kindergartens, large mattresses have been supplied with the logic that the children are bigger and they can fit 2-3 children per mattress. It should be noted that the sleep is an important part of the day for all children. The larger size mattresses are more difficult to store and take up a lot of space in the classrooms but institutions are happy with them and there were no complaints.

The bathing areas have been rehabilitated and water is now provided to all institutions within the project area for bathing. However there are no hooks for towels and soap has not been provided for any of the institutions. There are questions of sustainability, particularly in relation to the provision of soap, but it might be worth considering the provision of hooks and towels for all children in the institutions and also to look at the provision of soap – perhaps supporting the local production of soap.

Various toys have been introduced on a trial basis in one kindergarten and are certainly well received. However the emphasis is on imported toys and there is an opportunity to consider the local production of toys using local materials. A toy making workshop could be considered to help the caregivers and others within the community gain a better understanding of the potential for toys made from locally available resources. There is a clear need to link toys to learning and this can be done without the need to import expensive toys from China or elsewhere.

Result four: The one area where the programme can be seen to have not met its objectives is with the support to the School of Nursing. This is due to changes in personnel within the school itself – including the Director which has slowed down the

16 negotiations needed to develop the relationship. While a number of training materials have been purchased and will be delivered very soon the delay was inevitable doe to difficulties of getting access and therefore being able to identify needs.

While the technical capacity of the nurses and midwives has not been improved due to the delays – SCUK has done work within the school to improve the overall living environment, utilizing other funds available at country level. This will certainly contribute to the training of the students as they will be living more comfortably and therefore able study better. It is hoped that this foundation can be used to build on in upcoming programmes in the next few years.

3. PROCUREMENT, WAREHOUSING AND LOGISTICS.

The procurement and tendering procedures were planned and implemented according to the SCUK and ECHO guidelines and are clear and effectively managed. The systems were developed by SCUK staff with clear guidance from the London office and have been supported throughout by programme management

A key issue concerned with procurement for projects in DPRK is the limited capacity of suppliers which leads to multiple suppliers and small orders rather than using a few key suppliers to provide a wide range of items. SCUK attempts to support local purchase where possible, but this can cause difficulties when China suppliers are able through economies of scale to produce goods at a cheaper rate than their Korean counterparts. While ECHO guidelines state that the cheapest suppliers should be used there are occasions when SCUK wishes to purchase a slightly more expensive item when it can be provided, for similar quality by a Korean supplier.

Further problems encountered by the programme occurred during the procurement process and involved the quality of the products versus the samples provided during the tendering process and also delays in delivery – often through no fault of the supplier or SCUK. (In two cases, rail fires led to a delay in delivery of the goods.) Three items had to be returned due to the poor quality of the final product – which was not the same as that provided when the initial contract had been signed. In all cases further shipments were provided but as the programme works to a short time frame – only able to access the project site for 8-9 months of the year delays may have a serious impact on project implementation.

The evaluation team visited the warehouse in Tanchon and met the warehouse manager who showed the team the warehouse management procedures which was clear and well organised. The quality of some of the buildings raises some concerns as not all warehouses would appear to be weatherproofed which could lead to damage to some goods. Also while SCUK has insurance arrangements in place there were no obvious fire prevention mechanisms and these should be put in place to try and reduce damage in the event of fire.

Stock keeping appears to be of a high level and the warehouse manager operated a first in first out process which ensures the rotation of stocks. All items – some 120 separate items have been purchased and supplied during the course of the project - were clearly marked and shelved correctly in a number of adjacent warehouses in one compound which appears safe and secure.

E. RECOMMENDATIONS.

E.1 Water

1. Install water meters on selected mains to provide consumption data to confirm or adjust the current planning standard of 60 l / person / day 2. water quality is always a risk until we know with confidence that it is safe; we endorse the intentions of the SCUK project to import water quality testing equipment, train technicians in their use and ensure water quality data are available for monitoring; 3. Partners should monitor the waste water management and ensure it does become a problem, for example, by building soak-aways or ensuring adequate drainage and discussing waste water in hygiene classes;

E.2 Sanitation

4. SC should consider continued support to toilet replication for non-locally available items such as the vent pipes (about Euros 10 each) and mosquito netting 5. Identify other compost latrine options (see for example the SIDA eco-san manuals) that may be worth experimenting with in this climate and organise a study tour for partners to see them in action and consider their adaptation to the area;

E.3 Urban project work

6. Building on the success of the rural programme, the next phase should identify a higher proportion of urban areas with clinics, nurseries and other mother and child institutions with water and sanitation stress, and include these in its target; large scale urban projects require huge funding, beyond SCUK’s capacity – its role should be to deliver watsan to children’s institutions and neighbouring areas demonstrating low –cost innovations such as rain water harvesting, small bore sewers as well as gravity flow water and eco-san toilets; 7. Consider a study tour to appropriate urban areas in Asia to see such innovations in practice (e.g. Orangi in Karachi Pakistan has a huge low cost drainage project for 1 million people – though it should be noted this is an anal-washing society)

E.4 Health

8. Action plans for mothers once hygiene training has been done to ensure that they actually do pass on the messages given and that there is some ability to follow up the promotion activities at community level.

19 9. Reinforcement of ORS message to the mothers to ensure they understand both how to make it and also how to use it correctly. 10. Hooks for towels in the bathing areas could be considered – perhaps with a one off provision of towels. 11. The provision of soap should be considered – perhaps considering the support of the local production of soap which can be used as an income generation activity as well as supporting the hygiene promotion messages within the project. 12. Soap should also be provided in the clinic and institution latrines – again to support the hygiene promotion messages provided at all levels within the community. 13. Training to include more on child care, first aid, care of sick children (with fever, with diarrhoea). 14. A follow up visit should be planned for next September to see whether the morbidity rates provide the evidence of health impacts that are expected. These would include consideration of rates of diarrhoea and skin infections. 15. There is a need to consider the development of indicators to see how to measure quality of care within institutions and to this end it is recommended that links are developed by SCUK between the ECCD work in Vietnam and Myanmar and the programme to learn from the experience of the programmes in these other countries. 16. Bringing toys for all institutions from China or elsewhere is expensive and reinforces the message that children need imported toys with which to learn. Therefore it is recommended that work is done with the caregivers and mothers groups on exploring the possibilities of the local production of toys using local materials. There are a number of resources available in the SCUK office which provide ideas for possible toys and these could be used as the basis for a workshop for caregivers and mothers groups on play and its importance and also on the manufacture of toys. 17. There is a need to continue to negotiate with School of Nursing to try and get access for support to curriculum development and further support to improving the quality of care provided by nurses and midwives in Tanchon County. It is therefore recommended that any future projects in Tanchon County should include the continued support to the School of Nursing.

E.5 Logistics and Warehousing

18. Contract an inspection agency or similar service to inspect goods before shipping from China and ensure their safe delivery to and within DPRK. 19. Basic precautions should be taken in the warehouses for the prevention of fire through the placing of sand buckets and the removal of all flammable material from the warehouses.

F. CONCLUSIONS.

From all of the above it is clear that in the most part SCUK has met and indeed exceeded its objectives in the implementation of the project in Tanchon County. The project has been well managed and the local partners have been extremely cooperative and the relationship between the partners and the SCUK team has greatly contributed to the success of the project.

Much of the work can and is being replicated, as this reports notes and there is great potential for replication in further counties as SCUK is already planning.

Obviously with an uncertain future it is unclear what opportunities there will be to learn from the evaluation and to implement the recommendations made. However it is hoped that if SCUK is able to remain in DPRK and operate effectively then future projects will be able to build on the success of this project.

The evaluation team wish to thank all those who assisted in the trip which was enjoyable as well as informative and certainly provided plenty of discussion and learning on all sides. The team particularly wishes to thank the people of Tanchon County and the Province of South Hamgyong who allowed us to enter their homes and institutions and view their work – and also who were willing to answer all our questions so patiently.

Appendix A

Terms of Reference for External Evaluation

Project Title: Improving Primary Health Care Access and Quality in South Hamgyong and North Pyongan Provinces – DPRK

1. Background For thirty years following the war, DPRK benefited from a comprehensive healthcare and childcare delivery system. Health professions developed education and clinical facilities using research and technology offered by such allies as the former Soviet Union and China.

An acute food and health care crisis has arisen since 1996 against the backdrop of a series of natural disasters that led to widespread food deprivation. This has been compounded by the economic setbacks that came as a result of former political allies and trading partners being unable to maintain and render earlier levels of support. This has resulted in a general degradation of domestic industrial food production facilities, which are now barely functioning.

Since 1996 DPRK has been receiving humanitarian assistance. The World Food Programme (WFP) currently has the largest food aid distribution programme. However, the food pipeline has proved difficult to maintain as donor contributions are cut. 6.4 million people, including pregnant and lactating mothers and children under five years of age, depend on food supplied by WFP.

Save the Children has re-established a presence in DPRK after a gap of a few years. Our long- term objectives are to improve capacity within the existing government structures to provide care and protection for vulnerable children and to improve health delivery services for mothers and children. In August 2003 we opened an office and commenced initial activities – improving conditions and standards in 31 child welfare institutions through renovation of food preparation areas and provision of warm clothing, blankets and mattresses. These institutions are mostly located in the southern part of the country not far from the capital city Pyongyang. In addition, an assessment mission was launched in October 2003 to explore the possibilities for SC to be involved in Northern parts of the country where the presence of international agencies is rather sparse. As a result SC managed to negotiate with the Flood Damage Rehabilitation Committee (FDRC) – the DPRK government’s sponsoring agency for international humanitarian aid in the country – to work in Tanchon County, South Hamgyong Province. The SC and FDRC agreed to work in the area of supporting local authorities in improving the access and quality of primary health care by improving the facilities in rural Ri and Dong clinics.

Ri clinics, the first access point for the DPRK population to health care, are ill-equipped to provide the basics of primary health care due to a shortage of drugs and equipment, restricted or non-existent in-service training opportunities for health staff, and limited or no access to clean water. There is a particular concern expressed by the stakeholders that the problems of water are leading to high rates of diarrhoea and other intestinal illnesses in both adults and children in the rural and urban areas of Tanchon County. A comprehensive water system has fallen into disrepair over recent years through low maintenance and a lack of resources to replace worn out and rusted pipes, pumps and tanks. Most clinics are dependent on uncovered wills or piped water arriving maybe one to two hours a day – the quality of which is extremely suspect due to the rusting of pipes, non-functioning water treatment facilities and a lack of supplies for the disinfection of the

22 water. This decay has been compounded by an energy crisis that has limited the quantity of water available.

In March 2004 an initial health project proposal was developed and funding agreement was reached with ECHO to initiate activities in Tanchon – in minimum two clinics and the immediate communities around – in addressing the acute shortage of clean water in health facilities and improve the sanitation, particularly in the birth room. IN addition, one of the central objectives of the intervention was to develop relation with local authorities in Tanchon so that the up-coming project can be implemented effectively. Theis initial phase is expected to be completed by the end of October. The final report will be submitted by the end of December 2004.

For this initial work, an internal evaluation involving SCUK’s environmental health manager and other two members from humanitarian community – INGOs – working in the sector in DPRK is planned at the beginning of October 2004. The internal evaluation will focus on the process applied in the implementation and the quality of work produced in order to achieve the intended impact of the project on child and mother’s health of the targeted communities.

This project – Improving primary health care access and quality in South Hamgyong and North Pyongan Provinces –, Tanchon II for year 2005/2006, is intended to build on the experience and development of the above-mentioned initial work in Tanchon.

1.1 Purpose and Objectives of the Project The main purpose of the project is to contribute to a reduction in the morbidity and mortality of children in a minimum of seven communities in Tanchon County, South Hamgyong Province and to eight child welfare facilities in North in North Pyongan Province.

The immediate direct beneficiaries will be children, their mothers and their immediate families from the seven communities: on an average more than 700,000 patients visit the health facilities per year in all health facilities in Tanchon County. In addition a minimum of 1,920 children aged between 3 months and 6 years in community-based care will be the direct beneficiaries in eight child welfare institutions in Tanchon County and eight in North Pyongan Province.

The project is scheduled to start form the beginning of October 2004 and will be completed by the end of September 2005.

1.2 Specific Project Objectives Minimum of seven health institutions and the communities in the immediate vicinity of the health facilities – minimum 100 families in each community – are provided with 24-hour running water, and the institutions (clinics and child care centres, eight in Tanchon and eight in North Pyongan) have the provision of safe and hygienic sanitation. In addition, all the welfare institutions are provided with warm clothes and bedding for children so that they are protected from harsh winters in the targeted areas. The project outputs include:

• Rehabilitation work on the birth room has been completed in a minimum of seven health facilities. • Rehabilitation of the water supply and sanitation provision for the health facilities has been completed in all seven health centres and their immediate communities. At least 28 demonstration latrines are constructed across the seven communities. • Rehabilitation / renovation of a minimum of sixteen child welfare institutions (particularly the renovation of kitchen, provision of water supply and safe sanitation) has been completed.

23 • Improved capacity of nurses and midwives in targeted health centres in provision of mother and child care in achieved.

2. The evaluation

2.1 Purpose of the External Evaluation The main purpose of the external evaluation is to determine the effectiveness of the project in achieving its broader objective of improving the health of children and their mothers in targeted communities. The evaluation purpose includes:

a) To determine whether project has been successful in achieving its objectives and outputs established in the project logframe. b) Determine how far the project has been able to address the issues of sustainability of the installed facilities through local capacity building initiatives. c) To determine the effectiveness of the project in addressing the issues outlined under risk and assumption in the project document. d) Determine the capacity of the SCUK project team in planning, implementing and monitoring the project progress and activities. e) Access to quality health services being one of the key issues in rural DPRK; recommend the effective ways to replicate the successful approaches in other Counties and Provinces. f) To advise on possible changes to project approaches for development of future interventions.

2.2 Methodology The team will be lead by an international, experienced environmental or community health expert. There will be at least one more international expert of appropriate expertise in addition to a staff member from Save the Children UK’s China programme or regional office: altogether three persons.

The actual evaluation process will involve:

• Review of project documents and reports • Field visit to observe the work done at community level • Discussion with local authorities, teachers, clinic staff, mothers, women’s group and ordinary community people wherever possible. • To visit at least three clinics in Tanchon and determine the effectiveness of the inputs. • Visit communities near by these clinics to assess the impact the project is likely to have on child an mother’s health. • Meet the Provincial FDRC and water authority to determine the usefulness of the approach applied by the project and their commitment to replicate the successful approaches in other Counties. • Visit at least four child welfare institutions in Tanchon and four in North Pyongan to determine the effectiveness of the project in addressing the children’s immediate need and the likely impact of improved facilities in their health and development. • Meet village representatives and local women’s group to gauge the level of awareness they have acquired as a result of the project intervention in utilising the improved facilities. • Determine whether the procurement, distribution and documentation systems are in line with SC policies and procedures.

24 • Write the report and recommendation for the use of both SC management and donor.

2.3 Available Resources

The following resources are available to carry out the review:

SC administration support staff Office workspace Vehicle with driver Translator

2.4 Timing

The evaluation will take place in September r 2005 (tentatively mid September to first week of October ) and the duration of evaluation will be for two weeks . The report must be presented by mid-October 2005.

2.5 Output of evaluation

A report will be produced describing and analysing the impact of ECHO funded Tanchon Health Project, and recommendations for future replication, particularly how the experience and lessons learned during this period can be taken forward to improve the health of children and women in DPRK. The team will be expected to make a presentation of these to ECHO, SC and partners at the end of the mission.

Appendix B

Timetable for evaluation.

Date Morning Afternoon 14.9.05 Depart Beijing OS 462 15.9.05 Arrive Beijing 16.9.5 Meet with Kate Wedgewood SCUK China Programme Director Collect DPRK visas 17.9.05 Flight to Pyongyang 18.9 .05 19.9.05 Prepare for field trip Leave for Wonsan 20.9.05 Travel 21.9 .05 Meet with City Management Visit Dolsan Ri 22.9.05 Visit FDRC warehouse Visit Daljon Ri 23.9.05 Visit Jagne Ri Visit paeksan Ri. 24.9.05 Report Writing 25.9.05 Report Writing 26.9.05 Debriefing with Local Partners Leave for Hamhung 27.9.05 Meet with Provincial FDRC Leave for Pyongyang 28.8.05 Meet with UNICEF Meet with ECHO 29.9.05 Meet with Red Cross Report Writing 20.9.05 Debrief with SCUK Team Report Writing 30.9.05 Depart to Beijing 1.10.05 Depart for London

Appendix C

List of key people met during the field trip.

Mr Jong Song Chol. FDRC Chairman of South Hamgyong Province Mr Pak Chong Mo. Vice Chairman of Tanchon City People’s Committee Mr Jon So Ju Vice Chairman of Tanchon City People’s Committee Mr Sin Chol Hyok. Chief of Health Department in Tanchon City.

Dolsan Vice Ri leader Daljon Village Leader Jagnae Ri Leader Paeksan Ri Leader

Mr Choe Chi Jun Warehouse manager

Representatives of Mothers Groups.

27 Appendix E

Professional Midwife Kit with Aluminium Case (as UNICEF 99 022 15)

ITEM DESCRIPTION UNIT QUANTITY 1 Disinfector instrument 222 x 82 x 41 mm stainless pc 1 2 Basin kidney 575 ml autoclavable polypropylene pc 1 3 Basin kidney 825 ml (28 oz) stainless steel pc 1 4 Bowl sponge 600 ml stainless steel pc 1 5 Apron utility 900 mm x 1 m opaque plastic pc 1 6 Bag polyethylene self-sealing 125 x 200 mm pc 10 7 Mucus extractor pc 1 8 Catheter urethral nelaton solid-tip one-eye 12FR pc 1 9 Catheter urethral nelaton solid-tip one-eye 14FR pc 1 10 Gloves surgeon’s latex size 7-1/2 pair 2 11 Pouch clear polypropylene 250 x 380 mm long pc 1 12 Sheeting plastic clear PVC 90 cm x 180 cm pc 1 13 Bottle dropping 10 ml amber glass pc 2 14 Thermometer clinical sub-normal 30 to 40C pc 1 15 Thermometer clinical oral/rectal Cels/Fahr pc 1 16 File for BCG ampoules 6 x 50 mm pc 4 17 Brush hand surgeon’s white nylon bristles pc 1 18 Case for midwife kit with lid empty aluminium pc 1 19 Case for 1 2-ml syringe + needles empty metal pc 1 20 Cotton wool absorbent non-sterile 100 g pc 1 21 Gauze-pad sterile 12-ply 76 x 76 mm, pack of 20 pk 1 22 Nail clipper/file pc 1 23 Soap-box 2-piece hinged plastic pc 1 24 Soap toilet 113 g bar unwrapped pc 6 25 Scale spring baby 5 kg x 25 g graduations pc 1 26 Suture catgut obstetric sterile w/needle, box of 36 box 1 27 Tape umbilical non-sterile 3 mm wide x 100 m spool pc 1 28 Tape-measure 1.5M/60” vinyl-coated fibreglass pc 1 29 Foil wrap to prevent neonatal heat loss 122 x 92 cm pc 1 30 Towel huck, 430 x 500 mm (17 x 20”) pc 4 31 Sphygmomanometer aneroid 300 mm with cuff pc 1 32 Stethoscope binaural complete pc 1 33 Stethoscope foetal Pinard monaural pc 1 34 Forceps hemostat straight Rochester-Pean 160 mm ss pc 2 35 Needle hypo 0.70 x 32 mm/22G 1-1/4” Luer, box of 12 box 1 36 Scissors surgical straight 145 mm B/B SS pc 1 37 Syringe hypo 2 ml Luer glass pc 1 38 Bottle N/M round screw-cap 50 ml amber glass pc 1 39 Bottle W/M round screw-cap 50 ml amber glass pc 1 40 Methylergometrine 0.2 mg amp 1 ml, box of 10 box 1 41 Book exercise A5 ruled 8 mm lines 48 pages pc 2

28 42 Pen ball-point blue pc 2

The complete kit consists of: Total No of packages: 2 cartons Total weight: 25 kgs Total volume: 0.12 m3 Total value: USD 155.--