Monitoring and Evaluation of Health System Support and Strengthening Programs in Papua New Guinea

Monitoring and evaluation of health system support and strengthening programs in Papua New Guinea

Emma Field

A thesis in fulfilment of the requirements for the degree of

Doctor of Public Health

UNSW Australia

School of Public Health and Community Medicine

Faculty of Medicine

October 2018

UNSW Supervisors:

Dr Sally Nathan

Dr Alexander Rosewell

Associate Professor Siranda Torvaldsen

Abt Associates Supervisor:

Mr Geoff Scahill

Table of contents

Table of contents ...... i Abbreviations and Acronyms ...... iii Acknowledgements ...... iv Abstract ...... v Publications and presentations ...... vii Topic and scope of this thesis ...... x References ...... xii CHAPTER 1: HEALTH SYSTEM SUPPORT AND STRENGTHENING IN PAPUA NEW GUINEA ...... 1 1. Papua New Guinea ...... 2 2. Health system support and strengthening ...... 9 3. Health system support and strengthening in Papua New Guinea ...... 11 4. Monitoring and evaluation of health system strengthening ...... 13 5. References ...... 19 CHAPTER 2: EVALUATION OF THE CMCA MIDDLE AND SOUTH FLY HEALTH PROGRAM ...... 28 1. Chapter prelude ...... 29 2. Background ...... 29 3. Project overview ...... 32 4. Contribution to project ...... 33 5. Project outputs ...... 34 6. Ethical considerations ...... 35 7. References ...... 37 8. Publication ...... 38 CHAPTER 3: LESSONS FROM THE VILLAGE HEALTH VOLUNTEER PROGRAM AND CONSIDERATIONS FOR POLICY ...... 51 1. Chapter prelude ...... 52 2. Background ...... 52 3. Project overview ...... 53 4. Contribution to the project ...... 53 5. Project outputs ...... 54 6. Ethical considerations ...... 54 7. References ...... 55 8. Manuscript ...... 56 i

CHAPTER 4: CONTEXTUAL FACTORS AND HEALTH SERVICE PERFORMANCE IN PAPUA NEW GUINEA ...... 89 1. Chapter prelude...... 90 2. Background ...... 90 3. Project overview ...... 91 4. Contribution to the project ...... 91 5. Project outputs ...... 91 6. Ethics considerations ...... 92 7. References ...... 93 8. Manuscript ...... 94 CHAPTER 5: CHALLENGES OF HEALTH PROGRAM MONITORING AND EVALUATION IN LOW RESOURCE SETTINGS ...... 113 1. Chapter prelude...... 114 2. Background ...... 114 3. Project overview ...... 115 4. Contribution to the project ...... 115 5. Project outputs ...... 115 6. Ethical considerations ...... 116 7. Manuscript ...... 117 CHAPTER 6: CONCLUSION AND RECOMMENDATIONS ...... 131 1. Summary of findings ...... 132 2. Recommendations ...... 141 3. Conclusion ...... 143 4. References ...... 145 Appendix. Strengthening Health services in Western Province, Papua New Guinea: Progress Report of the North Fly Health Services Development Program and the CMCA Middle and South Fly Health Program ...... 148

Abbreviations and Acronyms

ANC Antenatal care

CAP Community Action and Participation

CMCA Community Mine Continuation Agreement

DrPH Doctor of Public Health

MDG Millennium Development Goal

M&E Monitoring and evaluation

MRAC Medical Research Advisory Committee

NHIS National Health Information System

OTDF Ok Tedi Development Foundation

OTML Ok Tedi Mining Limited

PEPFAR The United States President’s Emergency Plan for AIDS Relief

PNG Papua New Guinea

RCT Randomised controlled trial

TB Tuberculosis

UNICEF United Nations Children’s Fund

VHV Village Health Volunteer

VBA Village Birth Attendant

WHO World Health Organization

iii

Acknowledgements

The journey through the Doctor of Public Health (DrPH) can be both challenging and rewarding. While we should embrace the challenges, we need to be supported. I was fortunate to have a team of supporters to navigate the DrPH and foster learning and development.

I had the privilege to have been able to undertake the DrPH as part of my work at Abt

Associates. My workplace supervisor, Mr Geoff Scahill, has been an endless source of encouragement and an advocate for me during the DrPH. I am also very thankful to my team at Abt Associates and teams for the programs on which I worked.

I would like to sincerely thank my academic supervisors Dr Sally Nathan, Dr Alexander

Rosewell and Associate Professor Siranda Torvaldsen, who went above and beyond the role of supervisor. I would also like to thank my current supervisor, Professor Ross

Andrews from Menzies School of Health Research, for providing time to complete the

DrPH.

I would like to thank my family and partner for their enduring support, for listening and just being there. In particular I would like to thank Archie for formatting, and Alison and

Daniel for proof reading parts of this thesis.

I would also like to thank the National Department of Health in Papua New Guinea for providing approval for me to undertake the DrPH on programs operating within the country. I would also like to acknowledge the Ok Tedi Development Foundation, which funds the CMCA Middle and South Fly Health Program.

Finally, I would like to thank the people of Papua New Guinea. It has been a privilege to work with inspiring health workers in such a beautiful country. I hope the work of this thesis will contribute to the knowledge to improve the health system.

Abstract

The health system in Papua New Guinea is in urgent need of strengthening in order to improve the quality, coverage, equity and responsiveness of health services.

Numerous programs aim to improve health services in Papua New Guinea through health system support or strengthening. Understanding what works in what context, through program monitoring and evaluation (M&E) is vital. This thesis for the Doctor of

Public Health is on monitoring and evaluation of health system support and strengthening programs in Papua New Guinea. It consists of four manuscripts for publication developed from work completed in my role as the Monitoring and

Evaluation Manager at Abt Associates for multi-intervention health system support and strengthening programs.

Chapter 1 provides an introduction to the health system in Papua New Guinea, describes health system strengthening and M&E. Chapters 2 and 3 are based on the

Community Mine Continuation Agreement (CMCA) Middle and South Fly health

Program, implemented by Abt Associates, and delivered in partnership with existing health service providers. In chapter 2, I describe the program model and the changes in service delivery in the first two years of implementation using a mixed methods evaluation. The focus of chapter 3 is the Village Health Volunteer Program implemented as part of the CMCA Middle and South Fly Health Program and describes the Village Health Volunteer Program from multiple perspectives and provides lessons for policy.

Chapter 4 is based on the Rural Primary Health Services Delivery Project. Recognising that there are limitations in measuring performance of complex projects through analysis of quantitative data alone, I undertook a sequential mixed methods study to understand the contextual factors that affect health services. The chapter describes the results of this contextual analysis, and the impact of this approach on the formative

evaluation. Chapter 5 is a reflective piece where I describe the lessons learned from conducting the M&E of the CMCA Middle and South Fly Health Program. The chapter aims to provide practical advice to M&E practitioners in low resource settings.

These chapters informed recommendations summarized in Chapter 6. Through the publication of manuscripts in peer-reviewed journals, this thesis provides information for M&E of health system support and strengthening in Papua New Guinea and other low resource settings.

Publications and presentations

The following manuscripts for publication and presentations were completed during the course of the Doctor of Public Health.

Journal manuscripts and publications

I strongly believe that any publications from the thesis should be freely accessible so that researchers and program implementers in low resource settings, particularly in

Papua New Guinea, will be able to access the papers. As I did not have access to any funding for publication I chose journals that had no article processing or open access fees but were still highly regarded in the field of rural and remote health and easily accessible to practitioners as well as academics.

 Emma Field, Dominica Abo, Louis Samiak, Mafu Vila, Georgina Dove,

Alexander Rosewell and Sally Nathan. A partnership model for improving

service delivery in remote Papua New Guinea: a mixed methods evaluation.

International Journal of Health Policy and Management. 2018; article in press.

doi:10.15171/ijhpm.2018.50

 Emma Field, Georgina Dove, Nelson Witi, Dominica Abo, Louis Samiak, Lisa

Vallely and Sally Nathan. Perceptions of the Village Health Volunteer Program

implemented in remote Papua New Guinea: lessons for policy [manuscript

ready for submission].

 Emma Field; Jethro Usurup; Alexander Rosewell; Sally Nathan. Contextual

factors and health service performance from the perspective of the provincial

health administrators in Papua New Guinea. Rural and Remote Health,

[accepted 14 May 2018].

vii

 Emma Field, Mafu Vila, Laina Runk, Fiona Mactaggart, Alexander Rosewell and

Sally Nathan. Lessons for Health Program Monitoring and Evaluation in a low

resource setting. Rural and Remote Health, [accepted 7 June 2018].

Report

Abt JTA (2016) Strengthening Health services in Western Province, Papua New

Guinea: Progress Report of the North Fly Health Services Development Program and the CMCA Middle and South Fly Health Program. Available from: http://www.abtassociates.com.au/wp-content/uploads/2016/07/Progress-Report-

NFHSDP-CMCA-MSFHP_DFAT-Print_PA-signed.pdf

Conference presentations

 Emma Field, Laina Runk, Louis Samiak, Dominica Abo, Mafu Vila, Sally

Nathan, Alexander Rosewell and Georgina Dove. A model for improving health

service delivery in Papua New Guinea: the experience from the CMCA Middle

and South Fly Health Program. PNG Update: Sustaining Development (in PNG)

Beyond the Resources Boom. Port Moresby, Papua New Guinea, November

2016 (presented by Dominica Abo).

 Field, EJ; Vila, M; Dove, G and Kewa K. Lessons from the field: Monitoring and

Evaluation of Health Programs in Papua New Guinea. Population Health

Congress, Hobart, September 2015 (presented by Emma Field).

viii

Topic and scope of this thesis

This thesis for satisfying the requirement for the Doctor of Public Health (DrPH) is on monitoring and evaluating (M&E) health system support and strengthening programs in

Papua New Guinea. The DrPH is a workplace-based degree where students contribute research undertaken in their role to a thesis. The work presented in this thesis was undertaken as part of work while at Abt Associates (previously Abt JTA and JTA

International), a health and social sector consulting company based in Brisbane, from

2014-2016. The thesis primarily draws on work from two health programs in Papua

New Guinea: the Community Mine Continuation Agreement (CMCA) Middle and South

Fly Health Program and the Rural Primary Health Services Delivery Project. For both programs I undertook M&E activities as part of my core work at Abt Associates in the role of Monitoring and Evaluation Manager of several health programs. Chapter 1 provides an introduction to the health system in Papua New Guinea and describes health system strengthening and M&E concepts. Chapter 2 focusses on the midline evaluation of the CMCA Middle and South Fly Health Program. Chapter 3 takes the

Village Health Volunteer program data from the CMCA Middle and South Fly Health

Program and explores the policy issues for such programs. Chapter 4 describes a contextual analysis completed for the formative evaluation of the Rural Primary Health

Services Delivery Project to provide an understanding of the baseline performance for health indicators for participating districts. Chapter 5 is a reflective chapter where lessons learned from the M&E for the CMCA Middle and South Fly Health Program are described. Finally, chapter 6 summarises the thesis.

This thesis differs from a traditional Doctor in Philosophy. The emphasis is on workplace-based research, also known as applied research. Applied research is research conducted in the real world to solve real world problems. While applied research aims to generate new knowledge, it also can be used to validate existing knowledge (1). The latter is particularly relevant for health program M&E as programs

usually include interventions with an established evidence-base, so the focus is on validating the effectiveness of the intervention in the specific context.

Applied research is underrepresented in published literature on global health, particularly for low resource settings where arguably the need to understand how to improve health systems is greatest (2). Further, the real world context for applied research often means the most rigorous studies designs cannot be used due to practical, ethical and financial reasons (3). Nonetheless, applied research forms an important element of research, allowing health providers, planners and decision- makers understand how programs are implemented, how to make key improvements and to document what the outcomes were. The work presented in this thesis is written in a way that it is to meet the requirements of the DrPH and also provide insights and learnings for health service providers and program implementers, who seek to positively impact the health of the people they serve with the time and resources they have available.

References

1. Applied research [Miller-Keane Encyclopedia and Dictionary of Medicine,

Nursing, and Allied Health, Seventh Edition]. 2003 [cited 2017 7 April]. Available from: http://medical-dictionary.thefreedictionary.com/applied+research.

2. Ridde V. Need for more and better implementation science in global health.

BMJ Global Health. 2016;1(2):e000115.

3. Simmons RK, Ogilvie D, Griffin SJ, Sargeant LA. Applied public health research

-- falling through the cracks? BMC Public Health. 2009;9:362.

xii

CHAPTER 1: HEALTH SYSTEM SUPPORT AND STRENGTHENING IN PAPUA NEW GUINEA

Chapter 1: Health system support and strengthening in Papua New Guinea

1. Papua New Guinea

The Independent State of Papua New Guinea, often touted as “The land of the unexpected” is a culturally diverse, majority Christian, tropical island country (1). The population, as measured through the census in 2011, was 7.3 million with a rapid growth rate of 3.1% (2). Papua New Guinea is divided up into 22 provinces, including the Autonomous Region of Bougainville, and four regions: Southern, Momase, Islands and Highlands. A majority of the population live in rural and remote areas with only

15% in urban areas (3). While Papua New Guinea has three official languages: Tok

Pisin, English and Kiru Moto, there are more than 800 languages spoken, comprising

15% of the world’s languages (4).

Papua New Guinea has recently experienced a period of economic growth, largely due to foreign investment in the resource industry (5). However, as the economy in Papua

New Guinea is largely reliant on the resource industry, it is particularly exposed to fluctuations in commodity prices (5). This has been most evident in recent years where commodity prices stagnated or declined from 2011 (6). Recent economic growth and improved fiscal capacity at the provincial level has not translated into improved government service delivery, including health service delivery, due to inconsistent and late receipt of funds at the provincial level, and funding not filtering down to service delivery points (5-7). More broadly, a multitude of reasons for the lack of progress in development in Papa New Guinea have been identified, including corruption (Papua

New Guinea ranks 139 out of 167 on the corruptions perceptions index (8)), political and administrative decentralisation; weak institutions; poor law and order; frequent political and administrative staff turnover; ineffective coordination and planning; lack of capacity or inertia in implementing changes; and rapid population growth (9, 10).

Chapter 1: Health system support and strengthening in Papua New Guinea

1.1. Health status in Papua New Guinea

Life expectancy at birth is 63 years in Papua New Guinea, low compared to similar pacific countries Solomon Islands (69 years) and Fiji (70 years) and neighbouring

Australia (83 years) (11). Maternal and child mortality indicators are poor. While the

2015 Millennium Development Goal (MDG) target for maternal mortality was met, the rate is still high at 215 per 100,000 live births (4). Further, some argue that due to data quality issues the actual maternal mortality rate may be as high as 500 per 100,000 live births (12). Infant and child mortality are similarly high (44 and 56 deaths per 1000 live births, respectively) but again the rates met, or were close to meeting, the modest 2015

MDG targets of 43 and 56 deaths per 1000 live births respectively (4, 13).

Papua New Guinea has both a high burden of infectious diseases and an emerging burden of non-communicable diseases. Communicable diseases and obstetric and maternal conditions are leading causes for admissions to hospitals and deaths although there are increasing trends in diabetes mellitus, kidney disease and ischaemic health disease related deaths (14). Most causes for admissions and deaths for communicable diseases include malaria, tuberculosis, and lower respiratory infections

(3, 14). Malaria incidence has been declining across the country, likely due to the implementation of the Malaria Program initiatives such as long-lasting bed net distribution, introduction of rapid diagnostic testing prior to treatment and an updated treatment protocol (15). However, Papua New Guinea has one of the highest incidences of tuberculosis (TB) in the world at 417 per 100,000 population (13). The ongoing high incidence of TB in Papua New Guinea has received much publicity and highlights the need for an effective National Tuberculosis Program and basic health services delivered through a strong health system (16).

Chapter 1: Health system support and strengthening in Papua New Guinea

1.2. The health system in Papua New Guinea

The World Health Organization defines a health system as consisting of “all organizations, people and actions whose primary interest is to promote, restore or maintain health” (17). There are six sub-systems or ‘building blocks’ for health systems: leadership and governance; health care financing; health workforce; medical products and technologies; information and research; and service delivery (17). The health system in Papua New Guinea will be described using these building blocks. However, it should be noted that a health system is not simply these six building blocks but rather a complex set of dynamic relationships and interactions between the building blocks (18).

1.2.1. Leadership and governance

Leadership and governance includes the key functions of policy guidance; intelligence and oversight; collaboration and coalition building; regulation; system design; and accountability (17). Prior to independence of Papua New Guinea from Australian rule in

1975, the health system was centralised with vertical programs (10). From the 1970s to

1990s the health system was progressively decentralised (10, 19). Now the National

Department of Health is responsible for policies, standards and guidelines, the procurement of medical supplies and managing provincial hospitals, regional hospitals and the national referral hospital. The provincial health administration is responsible for rural health services. In 2007 the Provincial Health Authorities Act was enacted which provides the structure for an authority which will have responsibility for not only the rural health services but the provincial hospital as well (20). Many of the 22 provinces in Papua New Guinea have either transitioned to a Provincial Health Authority or will do so in the near future.

The National Health Plan 2011-2020 sets the overarching goal of the health sector to strengthen primary health care for all and improved service delivery for the rural majority and urban disadvantaged (21). The plan strategies sit under eight key result

Chapter 1: Health system support and strengthening in Papua New Guinea

areas corresponding to health system strengthening and priority health issues: Improve

Service Delivery; Strengthen Partnerships and Coordination with Stakeholders;

Strengthen Health Systems; Improve Child Survival; Improve Maternal Health; Reduce the Burden of Communicable Diseases; Promote Healthy Lifestyles; and Improve our

Preparedness for Disease Outbreaks and Emerging Population Health Issues (21).

Leadership and governance, as in all sectors in Papua New Guinea, is weak in the health sector throughout all tiers of the health system (national, provincial, and district)

(10, 22, 23). While there has been substantial sound policy reforms in health, the implementation of the these policies is poor (10).

There are multiple reasons for the failure in effective policy implementation. Due to decentralisation of the health system, responsibility for implementation of national policies lies at the provincial and district level (10). However, the allocated finances and human resources at the sub-national level may not be sufficient for implementing these national policies(10). Further the health workers who become managers at district and provincial level, and therefore the leaders for policy implementation, have not received adequate training in leadership and management to do so (22). Implementation is further hampered by poor communication from national to province and district level; lack of accountability; and inadequate administrative systems (10).

1.2.2. Health care financing

Health care financing refers to the availability, distribution and efficient use of funds and supportive policies to improve people’s financial access to health services and progress towards universal health coverage (24). In Papua New Guinea total health expenditure is 4.5% of Gross Domestic Product, slightly lower than the recommended

5% or more but is higher than Fiji (4.1%) and lower than Solomon Islands (5.1%) and

Australia at 9.4% (13, 24). While policy states that primary health care and specialised health services are to be delivered free of charge, there is evidence that due to the delays or inadequate distribution of Health Function Grants, which are for recurrent 5

Chapter 1: Health system support and strengthening in Papua New Guinea

funds for health service delivery, some health facilities are forced to charge user fees in order to provide services (7, 25, 26).

1.2.3. Health workforce

The health workforce in Papua New Guinea consists largely of community health workers (34%) and nursing officers (28%), with smaller numbers of health extension officers (a cadre of health worker between a nurse and a medical officer) (4%), medical officers (4%), medical laboratory technicians (2%) (27). Allied health and administrative personnel make up over a quarter (26%) of the health workforce (27). Papua New

Guinea has one of the lowest densities of physicians globally at 0.055 per 1000 population. Neighbouring Solomon Islands has almost four times the density of physicians at 0.203 per 1000 population and even higher in Fiji (0.437) and neighbouring Australia (3.374) (28). The density of nurses/midwifery personnel is higher at 0.532 per 1000 respectively (28). However, again, this is much lower than

Solomon Islands (2.098), Fiji (2.297) and Australia (12.344) (28).

The current shortage of health workers is expected to worsen as the aging workforce retires (27). The supply of health workers has been limited by the number of places available at training institutions, and although the number of places has increased in recent years this is yet to address the shortage (29). The capacity of health workers and the quality of care that they provide is further hampered by the lack of professional development opportunities throughout their careers. Once health workers undergo their initial training there is minimal ongoing training, with supervision and in-service training insufficient or non-existent (27).

1.2.4. Medical products

Availability of quality and affordable medical supplies is critical for providing health services. The medical supplies system in Papua New Guinea consists of both a push and pull system. The push system is the distribution of 40% and 100% kits of supplies,

Chapter 1: Health system support and strengthening in Papua New Guinea

which are kits consisting of the stated proportions of standardized annual quantities of medical supplies for a health facility. The pull system is where health facilities order additional supplies required through the national supplies which are then distributed to facilities. There has been an improvement in the months that a facility has adequate medical supplies to 87% in 2014 and 2015 from 45-53% in 2010-2013, which has been attributed to the push system (30, 31). Inexplicably this figure plummeted in 2016 to

51%, and, although the reasons have not yet been explored, this indicates problems with the system still exist (32).

1.2.5. Health information systems

The National Health Information System (NHIS) is the primary platform for health information in Papua New Guinea. Health facilities collect aggregate information on outpatients, inpatients and services provided and report via paper forms to the provincial level. At the provincial level the data is entered into the provincial database and then the paper forms are forwarded to the National level for data entry into the

National database (33). There are significant issues with data quality, completeness and timeliness of the NHIS (33, 34). The National reporting completeness rate has ranged from 84% to 91% in the five years from 2012-2016 (32).

Accurate birth, death (including cause of death) and population data is required to calculate many health indicators. However, in Papua New Guinea there is not an adequate civil registration and vital statistics system in place. For example, birth and death registration coverage is only 5%, making it difficult to accurately measure health indicators, particularly specific-cause mortality indicators (35). Further, there are questions about the accuracy of the latest available population data from the 2011 census data which impacts the accuracy of any health indicator that uses census population data as the denominator (36).

Chapter 1: Health system support and strengthening in Papua New Guinea

1.2.6. Service delivery

In Papua New Guinea there are seven levels of health services which are, from the lowest to highest level: aid posts, health sub-centres, health centres, district hospitals, provincial hospitals, regional hospitals and the national referral hospital (37). The

Health Vision for 2050 describes the replacement of the two lowest level facilities, aid posts and health sub-centres, with community health posts over the coming years (37).

The community health posts are to be comprehensive primary health care facilities located within and servicing as a hub within communities (38).

The National Health Service Standards 2011-2020 clearly delineate the roles and services to be provided at each of the seven level of health services (37). The health facilities are largely run by the Government or faith-based organisations with a small number of facilities run by private providers (29).

Service delivery refers to the outputs of the health system, providing quality, people- centred health care to the population when and where needed (17, 39). In terms of health system outputs, that is the services provided, at the National level the targets are not being met and trends in performance generally show declines in the past five years. Outreach clinics, an important activity for increasing service coverage, declined to the lowest level in five years in 2016 at 29 per 1000 children under 5 years of age, well below the target of 50 (32). Family planning has been increasing since 2013 and was 81 couple years protection per 1000 women of reproductive age in 2016 (32). The proportion of women who receive at least one antenatal care visit has been stagnant over the past five years and was 54% in 2016, the lowest in the five-year period (32).

The majority of births take place without a skilled birth attendant. The proportion of pregnant women who had a supervised delivery has been relatively stable over the last five years at 40% in 2016 (32). Measles vaccination coverage for children less than 1 year was 36% in 2016, again the lowest in the five-year period and well below the target of 80% (32). 8

Chapter 1: Health system support and strengthening in Papua New Guinea

2. Health system support and strengthening

Historically, multi- and bilateral donor funded health programs in developing countries have been selective or vertical programs. Vertical health programs have mandates defined in terms of specific health outcomes to reduce the impact of a specific disease.

While success may be achieved in the focus area there can be diversion of resources to the vertical programs from other areas of the health system or missed opportunities for improving the overall functioning or capacity of the health system strengthening (40-

42). The early implementation of the President’s Emergency Plan for AIDS Relief

(PEPFAR) in Uganda provides an example of a vertical program with a focus on a single disease that missed opportunities to have an impact beyond addressing HIV.

There was no evidence of this vertical program contributing to health system strengthening as demonstrated by no increase in utilisation of services not related to

HIV (43). Further, PEPFAR funded activities were perceived to overburden already limited human resources in health (44).

More recently the focus in international development has been for health system strengthening programs or to broaden the support from vertical programs to include elements of health system strengthening. Health system strengthening has varying definitions and is not universally understood (45). In this thesis, there will be a distinction between health system strengthening and health system support. Health system strengthening, as defined by the World Health Organization, refers to improving the six health system building blocks and managing their interactions in ways that achieve more equitable and sustained improvements across health services and health outcomes (17). Health system strengthening typically works across the building blocks and seeks to create long term sustainable and transformative improvements to the system. Health system support refers to improving services through increasing health system inputs, for example, training of health workers and provision of funds or

Chapter 1: Health system support and strengthening in Papua New Guinea

supplies. Health system support is argued to only create short term improvements in service delivery (46).

Chee et al. describes health system support working at the input level of the health system building blocks while the typical vertical programs and health system strengthening works across the performance drivers of policy and regulation, organisational structures and behaviour (46). Both approaches are important and the appropriate use of either health system support or strengthening interventions depends on the context. For example, Newbrander (47) discusses rebuilding health systems in fragile states, where governments lack legitimacy and effectiveness in providing services, and the need for donors to provide short-term relief (health system support) along with longer-term health system strengthening.

In terms of what works for strengthening health systems, the consensus is that there is no single approach. Health system strengthening is a process that needs to be adapted to the context (17, 48). This is demonstrated by reviews of evidence on health system strengthening that identify a myriad of diverse approaches that have some evidence of effectiveness, including, but not limited to: contracting non-government organisations

(NGOs) or private sector providers to provide health services, user fee exemptions, health care subsidies for the poor, performance related pay and incentives, health worker training, medical supplies system strengthening, health insurance, service integration, task-sharing or task shifting, reorganising outreach workers, social marketing, and community engagement (49, 50).

The context-dependent nature of health system strengthening is illustrated by the example in the state of Tamil Nadu in India. The health system in this state has been promoted as a successful model for low resource settings (51). In particular, literature reported the success of the centralised tendering and purchasing of medicines (52).

However, when other states attempted to apply the model, it did not always translate to

Chapter 1: Health system support and strengthening in Papua New Guinea

similar results. The model had to be adapted to the local context, to the point that it was no longer similar to the original model (53). This example demonstrates the importance of adapting interventions to the context, not only at the country level but also within the country.

3. Health system support and strengthening in Papua New Guinea

After independence in 1975, the Australia Government was the sole funder of the

Papua New Guinea budget for the health sector; the country received no assistance from any other bilateral donors. This support transitioned to program support in the

1980s (19). Currently Papua New Guinea receives external assistance for health system support and strengthening through a number of initiatives. There are multiple programs supported by a range of donor organisations that aim to support or strengthen the health system in Papua New Guinea:

 The Health and HIV Program

 The Health and Education Procurement Facility

 The Rural Primary Health Service Delivery Project

 The CMCA Middle and South Fly Health Program

 The North Fly Health Services Development Program

 The health system receives additional support through various organisations:

e.g., World Vision, Oil Search Health Foundation, World Health Organization,

UNICEF, Rotarians Against Malaria.

The Australian Aid Program remains a large contributor to health system support and strengthening in Papua New Guinea. For example the Health and HIV Program aims to 11

Chapter 1: Health system support and strengthening in Papua New Guinea

support the Government of Papua New Guinea to implement the National Health Plan

2011-2020 and the National HIV and AIDS Strategy 2011-2015 with a focus on five priority provinces: Western, Manus, Eastern Highlands, Autonomous Region of

Bougainville and Western Highlands (54). Additionally, the Australian Aid Program funds the Health and Education Procurement Facility (HEPF). The health component of

HEPF aims to build health facilities to the National Health Service Standards of Papua

New Guinea and expand the infrastructure of health worker training institutions (55).

The Rural Primary Health Services Delivery Project is implemented by a Project

Support Unit within the National Department of Health. The Project is funded through: a loan from the Asian Development Bank to the Government of Papua New Guinea; a grant from the Australian Aid Program; the Organization of the Petroleum Exporting

Countries Fund for International Development and the Government of Papua New

Guinea. In-kind support is provided by the World Health Organization, and the Japan

International Cooperation Agency. The Project aims to strengthen the rural health system by increasing the coverage and quality of primary health care in partnership with both state and non-state service providers through a number of initiatives including supporting development and implementation of national policies and standards, supporting provincial partnerships between state and non-state actors, human resource development, construction of community health post facilities and health promotion

(56). The Project operates in two districts each of eight provinces in Papua New

Guinea. The Project commenced in 2012 and will run for six years.

There are two programs that operate at the sub-national level that work within all building blocks of the health system in Western Province of Papua New Guinea. The

North Fly Health Services Development Program is funded by Ok Tedi Mining Limited and covers the whole of North Fly District and the Nomad Local Level Government in

Middle Fly District (57) . The program commenced in 2009 and will run for 10 years

(58). The Community Mine Continuation Agreement (CMCA) Middle and South Fly 12

Chapter 1: Health system support and strengthening in Papua New Guinea

Health Program is funded by the CMCA portion of the Western Province People’s

Dividend Trust Fund and is managed through the Ok Tedi Development Foundation.

The program covers the CMCA villages along the Fly River in Middle and South Fly

Districts The Program commenced in 2013 and will run for five years (59). Both programs are implemented by Abt Associates (58, 59).

The work in this thesis is based on two of these programs in Papua New Guinea: the

Rural Primary Health Service Delivery Project and the CMCA Middle and South Fly

Health Program. These programs contribute to all health system building blocks and aim to improve service delivery to rural and remote populations.

4. Monitoring and evaluation of health system strengthening

Health system support and strengthening programs generally consist of a series of initiatives, implemented over a number of years, to achieve specified outcomes. In order to understand if a program is implemented as planned and achieving the intended outcomes, program M&E is conducted.

Program monitoring is a continuous ongoing process of documenting program implementation and progress towards achieving program outputs, objectives and goals and is sometimes called process evaluation (60). Evaluation is a periodic and systematic application of scientific methods to assess the design, implementation, outcomes or impacts of a program (61). Evaluations use a range of methodologies, both quantitative and qualitative, depending on the evaluation questions (62).

Program designs should be based on a program theory of change or program logic and are integral to M&E (62). Program logic refers to a diagrammatic representation of the program’s inputs, processes, outputs, outcomes and impacts (30). Similarly the

Chapter 1: Health system support and strengthening in Papua New Guinea

program theory of change documents the inputs, processes, outputs, outcomes and impacts but also documents the assumptions about ‘why’ the program will lead to changes in each step along the results chain (30). The program theory of change or program logic can be used to develop the M&E framework and evaluation questions.

There are two main types of evaluation: formative and summative (61). Formative evaluation can include needs assessments and process evaluation; typically focus on program inputs, processes and outputs; and are carried out prior to or during program implementation (63). Formative evaluations can be used to identify program design failures, implementation failures and evaluation failures (64). Ultimately, the purpose of a formative evaluation is to inform implementers so improvements can be made during program implementation to ensure desired outcomes and impacts are achieved.

Summative evaluation includes evaluations of outcomes or impacts and assesses the contribution of the program on achieving the desired outcomes and impacts (61). While the outcomes and impacts are typically measured towards the end of program implementation, summative evaluations need to be planned prior to implementation to ensure the relevant data are collected during the implementation period. Summative evaluations can be used to assess whether the program objectives and goals have been achieved and if such programs should be continued or expanded (63).

The best practice is that M&E is embedded within the program and starts from commencement of the program, rather than being undertaken after program implementation. Increasingly it is a requirement of health programs to incorporate an

M&E system to ensure the program is implemented as planned and the intended outcomes are achieved. A sound M&E system is one mechanism to ensure that program implementers are transparent and accountable to funders and that there are quantifiable results from the program that provide evidence for effectiveness and value

Chapter 1: Health system support and strengthening in Papua New Guinea

for money. The M&E system should also contribute to informed decision making by implementers (65).

After introducing the principles for health system strengthening the World Health

Organization developed a framework for M&E of health system strengthening (Figure

1). This framework combined both elements of program logic and an M&E framework whereby the data sources, analyses methods and communication strategy are detailed.

The framework is high level, therefore the specifics for inputs, processes, outputs, outcomes, impacts, data collection, analysis and communication are then detailed for each program in each context. This framework has been adapted to disease-specific programs and programs that address the health system building blocks (66).

Chapter 1: Health system support and strengthening in Papua New Guinea

Figure 1: Monitoring and evaluation framework for health system strengthening (67)

The reality of conducting program M&E for health system support and strengthening in low resource settings creates challenges in applying best practices (68, 69). An experimental design, or randomised controlled trial (RCT), is the most rigorous method for program evaluation (69). This method requires identification of suitable controls or a counterfactual, for comparison to the intervention unit (70). The intervention unit, in the context of a health program, may be a health facility or community (70). However, it may be unethical or politically unpopular to withhold an intervention from communities

Chapter 1: Health system support and strengthening in Papua New Guinea

so that they serve as controls (70, 71). Quasi-experimental methods, where controls are identified retrospectively from non-participating units, does overcome these ethical and political issues (71). However, for both experimental and quasi-experimental designs, identification of controls can be difficult or impossible, firstly, as health system support and strengthening programs may be implemented at the district, province or even national level and secondly the context may be diverse such as in Papua New

Guinea. Finally, experimental and quasi-experimental designs for evaluation are only feasible where there is a large number of units for analysis (70, 71).

Even if experimental or quasi-experimental designs for evaluation are possible, the ability to replicate the outcomes in other settings is limited. Often evaluations do not document how well the program has been implemented according to design

(monitoring or process evaluation) nor do they adequately describe and examine the contexts in which it has been implemented (72). This limits the understanding of what works for whom in what context. Further, for programs with many interventions, it is difficult to know which intervention(s) contributed to the outcomes (68).

More often, evaluations of multi-intervention programs in the developing context require multiple methods and study designs involving both quantitative and qualitative methods to validate results and are based on a program logic or theory of change (73).In the context of program evaluation, this approach is called a mixed methods evaluation and can be broadly categorised as parallel, sequential or multi-level mixed-methods evaluations (74). Parallel mixed methods evaluations involve data collection and analysis occurring separately, but concurrently, and the quantitative and qualitative findings are triangulated and synthesised (74, 75). Sequential mixed methods evaluations use quantitative and qualitative methods chronologically (in either order) where one method informs the next (75, 76). Multi-level mixed methods evaluations use multiple methods for each system level of the program (e.g. community, health workers, and health administrators) (74, 75). For multi-level mixed methods 17

Chapter 1: Health system support and strengthening in Papua New Guinea

evaluations, data may be collected in parallel or sequentially(74, 75). The complexity of methods used for program evaluation reflects that programs consist of multiple objectives/goals with multiple interventions being implemented in the real-world context.

Publication of multi-interventional program evaluations in peer-reviewed journal articles is limited, with most evaluations, if published at all, found in grey literature. For example, the USAID funded MEASURE Evaluation provides tools and lessons learned from monitoring and evaluation and the Development Experience Clearinghouse provides access to a database of USAID monitoring and evaluation reports (76, 77).

Outside of such initiatives, evaluation reports are often not available publicly, resulting in missed opportunities to build on learnings from other health programs. This thesis aims to contribute to the publicly available literature on multi-interventional health service delivery programs.

Chapter 1: Health system support and strengthening in Papua New Guinea

5. References

1. The World Factbook: Papua New Guinea: Central Intelligence Agency [cited

2016 2 January ]. Available from: https://www.cia.gov/library/publications/the-world- factbook/geos/pp.html.

2. National Statistical Office Papua New Guinea. National Population & housing

Census 2011 Final Figures Papua New Guinea 2013 [cited 19 October 2017].

Available from: http://www.nso.gov.pg/index.php/document-library.

3. Government of Papua New Guinea. National Health Plan 2011-2020 Volume 2

(Part A) Reference Data and National Health Profile. 2010.

4. Department of National Planning and Monitoring, Government of Papua New

Guinea. Papua New Guinea - Millennium Development Goals Final Summary Report

2015 [cited 2017 29 December]. Available from: file:///C:/Users/Emma/Downloads/015UNDP%20PNG%20MDG%20Progress%20Repo rt_100216.pdf.

5. Hayward-Jones J. Papua New Guinea in 2015 - At a crossroads and beyond

Sydney: Lowy Institute; 2015 [cited 2018 5 June]. Available from: https://www.lowyinstitute.org/publications/papua-new-guinea-2015-crossroads-and- beyond.

6. The National Economic and Fiscal Commission. Provincial Expenditure Review.

2014.

7. World Bank. Below the Glass Floor: Analytical Review of Expenditure by

Provincial Administrators on Rural Health from Health Function Grants & Provincial

Internal Revenue 2013 [cited 2018 5 June]. Available from: https://openknowledge.worldbank.org/bitstream/handle/10986/17567/845210WP0v20B e0x382123B00PUBLIC0pdf.pdf?sequence=1&isAllowed=y.

8. Transparency International. Corruptions Perceptions Index 2015 [cited 2016 8

June]. Available from: http://www.transparency.org/cpi2015#results-table.

Chapter 1: Health system support and strengthening in Papua New Guinea

9. Hayward-Jones J. The future of Papua New Guinea: Old challenges for new leaders Sydney: Lowy Institute; 2016 [cited 2018 5 June]. Available from: https://www.lowyinstitute.org/publications/future-papua-new-guinea-old-challenges- new-leaders.

10. The Australian National University. Policy Making and Implementation: Studies from Papua New Guinea: ANU E Press; 2009. Available from: http://press- files.anu.edu.au/downloads/press/p78541/pdf/book.pdf?referer=355.

11. World Health Organization. Life expectancy: Data by country 2016 [cited 2017

10 April 2017]. Available from: http://apps.who.int/gho/data/view.main.SDG2016LEXv?lang=en.

12. Mola G, Kirby B. Discrepancies between national maternal mortality data and international estimates: the experience of Papua New Guinea. Reproductive Health

Matters. 2013;21(42):191-202.

13. World Health Organization. Global Health Observatory Data Repository WHO

Western Pacific Region: Papua New Guinea statistics summary (2002 - present) [cited

2014 18 August]. Available from: http://apps.who.int/gho/data/node.country.country-

PNG?lang=en.

14. World Health Organization. Papua New Guinea: WHO statistical profile 2015

[cited 2018 5 June]. Available from: http://www.who.int/gho/countries/png.pdf?ua=1.

15. Department of Health. National Malaria Strategic Plan 2014-2018. 2014.

16. Uplekar M, Raviglione MC. The "vertical-horizontal" debates: time for the pendulum to rest (in peace)? Bull World Health Organ. 2007;85(5):413-4.

17. World Health Organization. Everybody’s Business: Strengthening Health

Systems to Improve Health Outcomes: WHO’s Framework for Action. Geneva2007

[cited 2018 16 May]. Available from: http://www.who.int/healthsystems/strategy/everybodys_business.pdf.

18. Systems thinking for health systems strengthening. Geneva: World Health

Organization 2009. 20

Chapter 1: Health system support and strengthening in Papua New Guinea

19. Thomason J, Kase P, Ndugwa N. Working together to get back to basics-- finding health system solutions. Papua and New Guinea Medical Journal. 2009;52(3-

4):114-29.

20. Provincial Health Authorities Act, No. 9 of 2007 (2007).

21. Government of Papua New Guinea. National Health Plan 2011-2020: Volume 1

Policies and Strategies 2010 19 October 2017. Available from: http://www.health.gov.pg/publications/PNGNHP%20Vol1_2014.pdf.

22. Asante A, Hall J. A review of health leadership and management capacity in

Papua New Guinea. Sydney: University of New South Wales, 2011.

23. Australian Department of Foreign Affairs and Trade. Papua New Guinea

National Department of Health: Independent Capacity Diagnostic Assessment 2013

[cited 2017 27 December]. Available from: http://dfat.gov.au/about- us/publications/Documents/2013-ndoh-capacity-assessment.pdf.

24. World Health Organization. Health financing strategy for the Asia Pacific region

(2010-2015). Manila, Philippines: 2009.

25. Wiltshire C, Mako AA. Financing PNG's Free Health Care Policy: User Fees,

Funding and Performance 2014.

26. National Department of Health. Free Primary Health Care and Subsidized

Specialist Services Policy: Implementing the Alotau Accord2013 23 November 2017.

Available from: http://www.health.gov.pg/publications/FPHCSSSP_2014.pdf.

27. The World Bank. PNG Health Workforce Crisis: A Call to Action 2011 [cited

2018 27 August]. Available from: http://documents.worldbank.org/curated/en/216511468332461651/pdf/NonAsciiFileNa me0.pdf.

28. World Health Organization. The 2016 update, Global Health Workforce

Statistics27 December 2017. Available from: http://www.who.int/hrh/statistics/hwfstats/.

Chapter 1: Health system support and strengthening in Papua New Guinea

29. National Department of Health. Supporting Evidence to the Arrest Plan 2013-

2016 2013 [cited 2018 5 June]. Available from: http://www.health.gov.pg/publications/V2%203rd_2013.pdf.

30. Clark H, Anderson AA. Theories of change and logic models: telling them apart

Atlanta, Georgia2004 [cited 2018 5 June]. Available from: https://www.theoryofchange.org/wp- content/uploads/toco_library/pdf/TOCs_and_Logic_Models_forAEA.pdf.

31. National Department of Health. Sector Performance Annual Review:

Assessment of Sector Performance 2010-2014. Port Moresby: Government of Papua

New Guinea; 2015.

32. National Department of Health. Assessment of Sector Performance 2012-2016:

National Report 2017 [cited 2017 20 November]. Available from: http://www.health.gov.pg/publications/SPAR_2016.pdf.

33. Rosewell A, Makita L, Muscatello D, John LN, Bieb S, Hutton R, Ramamurthy

S, Shearman P. Health information system strengthening and malaria elimination in

Papua New Guinea. Malaria Journal. 2017;16(1):278.

34. Ashwell HE, Barclay L. Problems measuring community health status at a local level: Papua New Guinea's health information system. Rural and Remote Health.

2010;10(4):1539.

35. Bloomberg Philanthropies. Papua New Guinea: CRVS country overview [cited

2017 27 December]. Available from: http://mspgh.unimelb.edu.au/__data/assets/pdf_file/0020/2330408/Overview_PNG_02

1701.pdf.

36. Allen B. Papua New Guinea National Census 2011: Rates of Population

Change in Local Level Government Areas: Australian National University 2014 [cited

2017 27 December]. Available from: http://bellschool.anu.edu.au/sites/default/files/publications/attachments/2015-12/IB-

2014-44-Allen-ONLINE_0.pdf. 22

Chapter 1: Health system support and strengthening in Papua New Guinea

37. Government of Papua New Guinea. National Health Service Standards for

Papua New Guinea 2011-2020: Volume 1 2011.

38. National Department of Health. PNG Community Health Post Policy 2013.

39. Monitoring the Building Blocks of Health Systems: A Handbook of Indicators and Their Measurement Strategies. Geneva: World Health Organization, 2010.

40. Kawonga M, Blaauw D, Fonn S. Aligning vertical interventions to health systems: a case study of the HIV monitoring and evaluation system in South Africa.

Health Research Policy and Systems / BioMed Central. 2012;10:2.

41. Keugoung B, Macq J, Buve A, Meli J, Criel B. The interface between the national tuberculosis control programme and district hospitals in Cameroon: missed opportunities for strengthening the local health system -a multiple case study. BMC

Public Health. 2013;13:265.

42. Biesma RG, Brugha R, Harmer A, Walsh A, Spicer N, Walt G. The effects of global health initiatives on country health systems: a review of the evidence from

HIV/AIDS control. Health Policy and Planning. 2009;24(4):239-52.

43. Luboga SA, Stover B, Lim TW, Makumbi F, Kiwanuka N, Lubega F, Ndizihiwe

A, Mukooyo E, Hurley EK, Borse N, Wood A, Bernhardt J, Lohman N, Sheppard L,

Barnhart S, Hagopian A. Did PEPFAR investments result in health system strengthening? A retrospective longitudinal study measuring non-HIV health service utilization at the district level. Health Policy and Planning. 2016;31(7):897-909.

44. Lohman N, Hagopian A, Luboga SA, Stover B, Lim T, Makumbi F, Kiwanuka N,

Lubega F, Ndizihiwe A, Mukooyo E, Barnhart S, Pfeiffer J. District Health Officer

Perceptions of PEPFAR's Influence on the Health System in Uganda, 2005-2011.

International Journal of Health Policy and Management. 2016;6(2):83-95.

45. Kutzin J, Sparkes S. Health systems strengthening, universal health coverage, health security and resilience. Bull World Health Organ. 2016;2(94).

Chapter 1: Health system support and strengthening in Papua New Guinea

46. Chee G, Pielemeier N, Lion A, Connor C. Why differentiating between health system support and health system strengthening is needed. The International Journal of Health Planning and Management. 2013;28(1):85-94.

47. Newbrander W. Rebuilding Health Systems and Health Services in Fragile

States 2007 [cited 2017 30 December]. Available from: https://www.msh.org/sites/msh.org/files/rebuilding-health-systems-and-providing- health-services-in-fragile-states.pdf.

48. Task Force on Global Action for Health System Strengthening. Global Action for

Health System Strengthening: Policy Recommendations for the G8 2009 [cited 2017 29

December]. Available from: http://www.jcie.org/researchpdfs/takemi/full.pdf.

49. Janovsky K, Peters DH. Improving Health Services and Strengthening Health

Systems: Adoption and Implementing Innovative Strategies 2006 [cited 2017 14

February ]. Available from: http://www.who.int/management/working_paper_5_en_opt.pdf?ua=1.

50. Hatt L, Johns B, Connor C, Meline M, Kukla M, Moat K. Impact of Health

Systems Strengthening on Health Bethesda, M.D.: Health Finance & Governance

Project, Abt Associates 2015 [cited 2017 31 December]. Available from: https://www.hfgproject.org/impact-hss-health/.

51. Parthasarathi R, Sinha SP. Towards a Better Health Care Delivery System: The

Tamil Nadu model. Indian Journal of Community Medicine : official publication of Indian

Association of Preventive & Social Medicine. 2016;41(4):302-4.

52. Chokshi M, Farooqui HH, Selvaraj S, Kumar P. A cross-sectional survey of the models in Bihar and Tamil Nadu, India for pooled procurement of medicines. WHO

South-East Asia Journal of Public Health. 2015;4(1):78-85.

53. Singh PV, Tatambhotla A, Kalvakuntla RR, Chokshi M. Replicating Tamil

Nadu's Drug Procurement Model 2012 [cited 2017 29 December]. Available from: http://apps.who.int/medicinedocs/documents/s19919en/s19919en.pdf.

Chapter 1: Health system support and strengthening in Papua New Guinea

54. Health and HIV Implementation Services Provider. [cited 2016 9 September].

Available from: http://hhisp.org/about-hhisp.

55. HKL - HEPF. Health and Education Procurement Facility - About the HEPF

[cited 2016 2 January ]. Available from: http://www.hepf.net/about-the-hepf/.

56. Asian Development Bank. Project Administration Manual: PNG Rural Primary

Health Service Delivery Project 2011 [cited 2018 27 August]. Available from: https://www.adb.org/sites/default/files/project-document/74336/41509-013-png- pam.pdf.

57. JTA International. Feasibility Studies for the Middle and South Fly Health

Development Program. 2012.

58. Abt JTA North Fly Health Services Development Program - About Us 2013

[cited 2016 2 January]. Available from: http://www.nfhsdp.org/about-us/.

59. CMCA Middle and South Fly Health Program: Abt Associates; 2013 [cited 2018

16 May]. Available from: http://www.cmsfhp.org/about-us/.

60. United Nations Development Programme. Handbook on Planning, Monitoring and Evaluating for Development Results. New York 2009.

61. Rossi PH, Freeman HE, Lipsey MW. Evaluation: A Systematic Approach. Sixth ed. United States of America: Sage Publications; 2004.

62. Bamberger M, Rugh J, Marby L. Real World Evaluation: Working under budget, time, data and political contraints Sage Publications; 2006.

63. Centers for Disease Control and Prevention. Building our understanding: key concepts of evaluation: What is it and how do you do it? [cited 2016 3 January].

Available from: http://www.cdc.gov/nccdphp/dch/programs/healthycommunitiesprogram/tools/pdf/eval_ planning.pdf.

64. Katz J, Wandersman A, Goodman RM, Griffin S, Wilson DK, Schillaci M.

Updating the FORECAST formative evaluation approach and some implications for

Chapter 1: Health system support and strengthening in Papua New Guinea

ameliorating theory failure, implementation failure, and evaluation failure. Evaluation and Program Planning. 2013;39:42-50.

65. Kusek JZ, Rist RC. Ten steps to a results-based monitoring and evaluation system : a handbook for development practitioners. Washington: The World Bank;

2004 [cited 2018 16 May]. Available from: https://openknowledge.worldbank.org/handle/10986/14926.

66. Patel MS, Afsana K, Phillips C. Tracking Progress toward Health and Equity through Millennium Development Goals-Lessons for Public Health Surveillance. In:

McNabb SJ, Conde JM, Ferland L, MacWright W, Memish Z, Okutani S, et al., editors.

Transforming Public Health Surveillance2016. p. 377-89.

67. World Health Organization. Monitoring and evaluation of health system strengthening: An operational framework Geneva2009 [cited 31 December 2017].

Available from: http://www.who.int/healthinfo/HSS_MandE_framework_Nov_2009.pdf.

68. Improving Service Delivery in Developing Countries: From Evidence to Action.

Washington, DC: The World Bank; 2009.

69. Campbell M, Fitzpatrick R, Haines A, Kinmonth AL, Sandercock P,

Spiegelhalter D, Tyrer P. Framework for design and evaluation of complex interventions to improve health. BMJ (Clinical research ed). 2000;321(7262):694-6.

70. White H. An introduction to the use of randomised control trials to evaluate development interventions. Journal of Development Effectiveness. 2013;5(1):30-49.

71. White H, Sabarwal S. Quasi-experimental Design and Menthods Florence:

UNICEF Office of Research, ; 2014 [cited 2017 31 December]. Available from: https://www.unicef-irc.org/publications/pdf/brief_8_quasi- experimental%20design_eng.pdf.

72. Adam T, Hsu J, de Savigny D, Lavis JN, Rottingen JA, Bennett S. Evaluating health systems strengthening interventions in low-income and middle-income countries: are we asking the right questions? Health Policy and Planning. 2012;27

Suppl 4:iv9-19. 26

Chapter 1: Health system support and strengthening in Papua New Guinea

73. Mookherji S, Meck K. How Can We Better Evaluate Complex Global Health

Initiatives? Reflections From the January 2014 Institute of Medicine Workshop. Global

Health, Science and Practice. 2015;3(2):174-9.

74. USAID. Technical Note: Conducting Mixed Method Evaluations 2013 [cited

2018 11 September]. Available from: https://www.usaid.gov/sites/default/files/documents/1870/Mixed_Methods_Evaluations

_Technical_Note.pdf.

75. Schoonenboom J, Johnson RB. How to Construct a Mixed Methods Research

Design. Kolner Zeitschrift Fur Soziologie Und Sozialpsychologie. 2017;69(Suppl

2):107-31.

76. USAID. Development Experience Clearinghouse [cited 2017 30 December].

Available from: https://dec.usaid.gov/dec/content/evaluations.aspx.

77. MEASURE Evaluation [cited 2017 30 December]. Available from: https://www.measureevaluation.org.

CHAPTER 2: EVALUATION OF THE CMCA MIDDLE AND SOUTH FLY HEALTH PROGRAM

Chapter 2: Evaluation of the CMCA Middle and South Fly Health Program

1. Chapter prelude

This chapter is based on the report: Strengthening Health services in Western

Province, Papua New Guinea: Progress Report of the North Fly Health Services

Development Program and the CMCA Middle and South Fly Health Program (1), specifically chapters 1 and 3 of the report. This report was a substantial piece of work that combined the midline evaluation of the CMCA Middle and South Fly Health

Program and annual report of the North Fly Health Services Development Program.

These two programs operated in Western Province in Papua New Guinea and would be integrated over the coming years. The report provided a comprehensive account of the progress the programs to date and the future directions.

2. Background

2.1. Ok Tedi Mining Limited and Ok Tedi Development Foundation

The Ok Tedi Mine has operated in Western Province of Papua New Guinea since

1984, first extracting gold and later expanding to copper (2). Shortly after production began, landslides destroyed a dam containing tailings (crushed rock and water) (2).

The mine subsequently became fully owned by Papua New Guinean interests in 2013, with the Papua New Guinea Government the majority shareholder, following the exit of the original owners - BHP in 2002 and Inmet Mining Corporation in 2011 (2).

A community mine consultation process was initiated by Ok Tedi Mining Limited to consult with the communities along the Fly River that were directly impacted from the mine operations to provide consent, or otherwise, for the mine to continue operation.

Consent from the communities is provided in the form of Community Mine Continuation

Agreements (CMCA) (2).

Chapter 2: Evaluation of the CMCA Middle and South Fly Health Program

Ok Tedi Mining Limited contributes to development in Western Province through a number of Corporate Social Responsibility initiatives. In terms of the health sector, Ok

Tedi Mining Limited has funded the construction and operation of the level 4 Tabubil

Hospital in the mining town of the same name and the redevelopment of the level 4

Kiunga Hospital (3). Ok Tedi Mining Limited also funds the North Fly Health Services

Development Program. Ok Tedi Development Foundation manages the community development benefits from Ok Tedi mine operations on behalf the beneficiaries (3).

The North Fly Health Services Development Program is a multi-interventional program to improve health services covering in the North Fly District of Western Province (4).

The program is implemented in partnership with the existing health service providers, specifically the North Fly District Health Services (government), Evangelical Church of

Papua New Guinea health services and Catholic Health Services. The program commenced in 2009 and is due to continue until 2018 (4).

Like the North Fly Health Services Development Program, the CMCA Middle and South

Fly Health Program is a comprehensive health program that aims to improve service delivery to the CMCA communities in Middle and South Fly (5). The Program began in

July 2013 and would be implemented over five years (6).

2.2. CMCA Middle and South Fly Health Program

The CMCA Middle and South Fly Health Program is a multi-intervention program that includes:

 Coordination of support through a partnership with existing health service

providers;

Chapter 2: Evaluation of the CMCA Middle and South Fly Health Program

 Provision of outreach clinics to communities including outpatient clinics,

immunisation, antenatal care, provision of family planning methods, health

promotion and child nutrition checks and counselling;

 Provision of essential medical equipment including vaccine refrigerators;

support for ordering, distributing and managing medical supplies;

 Infrastructure improvements to health facilities including rehabilitation of water

supply, renovations and construction of staff housing and provision of transport

and communications;

 Coordination of health worker training through formal training and on-the-job

training; support management capacity and clinical patient care at health

facilities; and

 Implementation of community-based initiatives such as VHV Program and

Healthy Village concept.

The program activities are aligned with the Papua New Guinea National Health Plan

2011-2020.

The program has an emphasis on health system support. Health system support was deemed appropriate given that existing health service providers did not have the capacity and resources to deliver health services to the minimum standards, exacerbated by the remote setting, in the program area. While it is too early to determine the sustainability of this approach, it is certainly worth investigating as the program is due to finish in 2018.

The CMCA Middle and South Fly Health Program is a novel model of improving health services in that a private organisation (Abt Associates) was contracted to implement the program by a non-government organisation (Ok Tedi Development Foundation) and it was funded by compensation payments from Ok Tedi Mining Limited (the CMCA 31

Chapter 2: Evaluation of the CMCA Middle and South Fly Health Program

portion of the Western Province Peoples Dividend Trust Fund). This model has both elements of contracting-out for service delivery and public-private partnership.

3. Project overview

The CMCA Middle and South Fly Health Program design included a comprehensive

M&E plan including ongoing monitoring and reporting, and a baseline, midline and endline evaluation. A baseline evaluation was conducted in July-September 2013, at the commencement of the program. The findings of the baseline were used to identify immediate needs and inform the 2014 annual activity plan. A comprehensive midline evaluation was conducted in late 2015 to review progress to date and inform further implementation. The objectives of the midline evaluation were to:

 Review progress to date on program activities, outputs and outcomes, including

progress towards achieving the National targets as detailed in the National

Health Plan Monitoring and Evaluation Framework;

 Assess the effectiveness of the partnership model and coordination

mechanisms; and

 Identify lessons learned and recommendations for improving overall program

performance to achieve outcomes by 2018 and beyond.

The evaluation questions were developed with input from the program partners. The evaluation methods were then developed based on the objectives and the identified evaluation questions. The evaluation used a multi-level parallel evaluation design whereby both qualitative and quantitative methods are used concurrently to gain multiple perspectives across the levels of the program (community, health workers,

Village Health Volunteers, health facilities, health service providers, other stakeholders). The findings from each method were triangulated and synthesised to 32

Chapter 2: Evaluation of the CMCA Middle and South Fly Health Program

develop a comprehensive picture of the program. Quantitative methods included a before-after analysis of key indicators using the NHIS data and health facility assessments; and qualitative methods included inductive thematic analysis of key informant interviews with health workers, Village Health Volunteers, program partners and stakeholders; and focus group discussions with communities. While it is best practice for at least two people to undertake the analysis of qualitative data, resources only permitted one person. I acknowledge this is a limitation of the analysis. However, the field staff who collected the data were involved in the development of the manuscript and had opportunity to provide their input. Both program staff and independent evaluators were involved in the midline evaluation.

4. Contribution to project

The CMCA Middle and South Fly Health Program Midline Evaluation was my responsibility as the Monitoring and Evaluation Manager. I developed the evaluation design, developed the semi-structured interview and focus group discussion guides, developed training materials for the field evaluators, supervised the independent evaluators while in the field, analysed the data and wrote the report and journal articles.

Two independent field evaluators were engaged to undertake the interviews and focus group discussions in the communities. An international independent reviewer was engaged to undertake interviews with Program Partners and other stakeholders, and wrote a short report which was included in overall report: Strengthening Health services in Western Province, Papua New Guinea: Progress Report of the North Fly Health

Services Development Program and the CMCA Middle and South Fly Health Program.

I led the development and revisions of the manuscript “A Partnership Model for

Improving Service Delivery in Remote Papua New Guinea: A Mixed Methods

Evaluation” with input from the co-authors. I also led the writing of the report: 33

Chapter 2: Evaluation of the CMCA Middle and South Fly Health Program

“Strengthening Health services in Western Province, Papua New Guinea: Progress

Report of the North Fly Health Services Development Program and the CMCA Middle and South Fly Health Program” with inputs from the team at Abt Associates.

Specifically, I wrote chapters 1, 3 and 5 and substantially contributed to Chapters 2 and

5. Project outputs

There were multiple project outputs from this evaluation. The Midline Evaluation of the

CMCA Middle and South Fly Health Program formed a large part of the report:

Strengthening Health services in Western Province, Papua New Guinea: Progress

Report of the North Fly Health Services Development Program and the CMCA Middle and South Fly Health Program. The findings from this report were disseminated through presentations to stakeholders (Ok Tedi Development Foundation, the CMCA

Trust Leaders and program partners) and posters summarising the results were provided to health facilities and communities in the Program catchment area. In addition to the report, a presentation on the midline evaluation was given at the PNG

Update conference and a journal article was published. The outputs are detailed below.

Journal publication

Emma Field, Dominica Abo, Louis Samiak, Mafu Vila, Georgina Dove, Alexander

Rosewell and Sally Nathan. A partnership model for improving service delivery in remote Papua New Guinea: a mixed methods evaluation. International Journal of

Health Policy and Management. 2018; article in press. doi:10.15171/ijhpm.2018.50

Chapter 2: Evaluation of the CMCA Middle and South Fly Health Program

Declaration

I certify that this publication was a direct result of my research towards this DrPH, and that reproduction in this thesis does not breach copyright regulations.

......

Emma Field

Report

Abt JTA (2016) Strengthening Health Services in Western Province, Papua New

NFHSDP-CMCA-MSFHP_DFAT-Print_PA-signed.pdf

The full report is available in the Appendix of this thesis.

Conference presentation

Emma Field, Laina Runk, Louis Samiak, Dominica Abo, Mafu Vila, Sally Nathan,

Alexander Rosewell and Georgina Dove. A model for improving health service delivery in Papua New Guinea: the experience from the CMCA Middle and South Fly Health

Program. PNG Update: Sustaining Development (in PNG) Beyond the Resources

Boom. Port Moresby, Papua New Guinea, November 2016

6. Ethical considerations

The CMCA Middle and South Fly Health Program Midline Evaluation included semi- structured interviews and focus group discussions. Ethical review and approval was

Chapter 2: Evaluation of the CMCA Middle and South Fly Health Program

provided by the Papua New Guinea Medical Research Advisory Committee (MRAC

No. 15.08) and University of New South Wales (HC15466).

Chapter 2: Evaluation of the CMCA Middle and South Fly Health Program

7. References

1. Abt JTA. Strengthening Health Services in Western Province, Papua New

Guinea: Progress Report of the North Fly Health Services Development Program and the CMCA Middle and South Fly Health Program 2016 [cited 2017 28 April]. Available from: http://www.abtassociates.com.au/wp-content/uploads/2016/07/Progress-Report-

NFHSDP-CMCA-MSFHP_DFAT-Print_PA-signed.pdf.

2. Ok Tedi Mining Limited. Ok Tedi Mining Limited - History [cited 2018 12

January]. Available from: http://www.oktedi.com/index.php/about-us/history.

3. Ok Tedi Mining Limited. Ok tedi Mining Limited - Health Services [cited 2016

28 September]. Available from: http://www.oktedi.com/our-corporate-social- responsibility/health-services.

4. Abt JTA North Fly Health Services Development Program - About Us 2013

[cited 2016 2 January]. Available from: http://www.nfhsdp.org/about-us/.

5. CMCA Middle and South Fly Health Program: Abt Associates; 2013 [cited 2018

16 May]. Available from: http://www.cmsfhp.org/about-us/.

6. Abt JTA. CMCA Middle and South Fly Health Program: Map of key program indicators [cited 2017 15 February ]. Available from: http://www.cmsfhp.org/health- facilities/key-indicators/

Chapter 2: Evaluation of the CMCA Middle and South Fly Health Program

8. Publication

Chapter 2: Evaluation of the CMCA Middle and South Fly Health Program

CHAPTER 3: LESSONS FROM THE VILLAGE HEALTH VOLUNTEER PROGRAM AND CONSIDERATIONS FOR POLICY

Chapter 3: Lessons from the VHV Program and considerations for policy

1. Chapter prelude

This chapter details a qualitative descriptive study on the Village Health Volunteer

Program implemented in Western Province as part of the CMCA Middle and South Fly

Health Program. The initial analysis of the qualitative data for this project was reported for the midline evaluation of the CMCA Middle and South Fly Health Program in the report: Strengthening Health services in Western Province, Papua New Guinea:

Progress Report of the North Fly Health Services Development Program and the

CMCA Middle and South Fly Health Program (1). Subsequent to this publication, further analyses were undertaken and contextualised through international literature on lay health worker programs.

2. Background

Lay health workers are a cadre of health workers who do not have a formal qualification; have received training in one or more interventions; work in the communities in which they live; and may or may not receive remuneration (2). While globally a common name for lay health workers is community health workers, this is not the case in Papua New Guinea where community health workers are a cadre of paid health worker who have completed a two-year formal qualification. In Papua New

Guinea, lay health workers are currently called Village Health Volunteers (VHVs).

As discussed in Chapter 1, there is a critical shortage of health workers in Papua New

Guinea, particularly in rural areas (3). VHVs provide an opportunity to increase the coverage of health services. The National Department of Health sets the minimum standards and curriculum for VHV Programs. The VHV Programs are often implemented within provinces by NGOs. In the Middle and South Fly Districts of

Western Province, the VHV Program was implemented as one of the many

Chapter 3: Lessons from the VHV Program and considerations for policy

interventions of the CMCA Middle and South Fly Health Program. For this part of

Papua New Guinea, it would be the first time the VHV Program had been implemented.

3. Project overview

This project was undertaken as part of the midline evaluation of the CMCA Middle and

South Fly Health Program (Chapter 2). There were evaluation questions identified by the Program partners related to the Village health Volunteer Program. Specifically, these questions were:

 What are VHVs doing in the communities?

 How well are the VHVs integrated into the health system?

 Do the VHVs continue to provide services?

 How much time do VHVs spend doing VHV work?

 How are VHVs supported by the community?

During the midline evaluation, independent evaluators sought to conduct semi- structured interview all VHVs available at each village they visited about their experience of the VHV program. In addition, they interviewed health workers at selected facilities and conducted male and female focus group discussions at selected communities for the midline evaluation of the CMCA middle and South Fly Health

Program, with several questions specifically related to the VHV Program.

4. Contribution to the project

I developed the interview and focus group discussion guides, analysed the data and wrote the report and drafted the manuscript in collaboration with the co-authors. 53

Chapter 3: Lessons from the VHV Program and considerations for policy

5. Project outputs

Manuscript

Emma Field, Georgina Dove, Nelson Witi, Dominica Abo, Louis Samiak, Lisa Vallely and Sally Nathan. Perceptions of the Village Health Volunteer Program implemented in remote Papua New Guinea: lessons for policy [manuscript ready for submission].

Declaration

I certify that this manuscript was a direct result of my research towards this DrPH, and that reproduction in this thesis does not breach copyright regulations.

......

Emma Field

Report

Abt JTA (2016) Strengthening Health services in Western Province, Papua New

NFHSDP-CMCA-MSFHP_DFAT-Print_PA-signed.pdf

The full report is available in the Appendix of this thesis.

6. Ethical considerations

The descriptive study on the VHV Program was carried out as part of the CMCA Middle and South Fly Health Program Midline Evaluation and included semi-structured interviews and focus group discussions. Written informed consent was obtained from all participants prior to interview or focus group discussion. Ethical review and approval

Chapter 3: Lessons from the VHV Program and considerations for policy

was provided by the Papua New Guinea Medical Research Advisory Committee

(MRAC No. 15.08) and University of New South Wales (HC15466).

7. References

1. Abt JTA. Strengthening Health Services in Western Province, Papua New

NFHSDP-CMCA-MSFHP_DFAT-Print_PA-signed.pdf.

2. Lehmann U, Sanders D. Community health workers: What do we know about them? Geneva: World Health Organization, 2007.

3. The World Bank. PNG Health Workforce Crisis: A Call to Action 2011 [cited

2018 27 August]. Available from: http://documents.worldbank.org/curated/en/216511468332461651/pdf/NonAsciiFileNa me0.pdf.

Chapter 3: Lessons from the VHV Program and considerations for policy

8. Manuscript

Title: Perceptions of the Village Health Volunteer Program implemented in remote

Papua New Guinea: lessons for policy

Authors: Emma Field1,2,3, Georgina Dove1, Nelson Witi1, Dominica Abo1, Louis

Samiak4, Lisa Vallely5 and Sally Nathan3

Authors Institutions:

1. Abt Associates

2. Global and Tropical Health, Menzies School of Health Research, Charles

Darwin University, Brisbane, Australia

3. School of Public Health and Community Medicine, UNSW Australia

4. University of Papua New Guinea

5. Kirby Institute

Abstract

In Papua New Guinea, Village Health Volunteers work in their communities to support healthy lifestyles and form an important link between the community and the formal health system. In the Middle and South Fly Districts of Western Province, the VHV

Program was implemented as one of the many interventions of the CMCA Middle and

South Fly Health Program. For this region of Papua New Guinea, it would be the first time the VHV Program had been implemented. We describe how the VHV program was implemented and documented the perspectives of the VHV Program from the

VHVs, their supervising health workers and the communities they were engaged to serve through focus group discussions and interviews. The qualitative data revealed that some health workers and community members had expectations of the VHVs that differed from their role, particularly in regards to the expectation that VHVs do clinical 56

Chapter 3: Lessons from the VHV Program and considerations for policy

work. The importance of compensation for VHVs for their time was identified by communities, health workers and the VHVs. VHVs and the community also noted the conflicting demands of the family of the VHV and their work in the community. The final theme identified was about the interaction between the VHV and the formal health system, where VHVs identified needs for access to medical supplies, transport, ongoing training and supervision. These themes are discussed in the context of international literature and the VHV policy in Papua New Guinea.

Chapter 3: Lessons from the VHV Program and considerations for policy

INTRODUCTION

Lay or ‘community’ health workers are a cadre of health workers, who receive training in one or more health interventions, usually do not have a formal qualification in health, and provide health services to their communities with or without remuneration (1).

These lay health workers are argued to have the advantage of an in-depth understanding of the community culture and language and are considered to be well placed to provide culturally appropriate health services to the community in which they work (2). There are a myriad of terms used for lay health workers from traditional birth attendant, community health representative and village health volunteers (VHVs) (1).

These lay health workers often form an important link between communities and health facilities (1).

Globally, lay health workers undertake a variety of roles, often including: disease prevention and control; environmental health; reproductive health; maternal and child health, school health; annual census data collection; health education and communication; treatment for specific diseases (e.g. HIV, TB); and provision of basic health care (3). Lay health workers have been shown to be effective for interventions such as immunisation coverage, uptake of breastfeeding, improving outcomes for TB, mental health, acute respiratory infection and malaria (4-8). However many lay health worker programs have not been successfully sustained or scaled up (9). Determining what works is also a challenge as there is a lack of information on how such lay health workers programs are developed and implemented (10).

In Papua New Guinea (PNG) Village Health Volunteers (VHVs) are a recognised cadre of health worker within the health system, with a national curriculum covering general health and maternal and child health. At the local level, VHVs are not universally available as the VHV Program is rolled out largely by non-government organisations in specific locations. As far back as the 1980s there were concerns raised about the

Chapter 3: Lessons from the VHV Program and considerations for policy

sustainability of such VHVs in PNG due to poor supervision (11). However there has been evidence the VHVs have been significant contributors to maternal health in some remote communities (12). While there has been a proliferation of publications on lay health workers in recent years, there have been limited publications from PNG and other Pacific countries (3). Nonetheless, lay health worker programs are particularly relevant to the PNG context due to a shortage of formal health workers and the challenges associated with accessing health services, such as poor infrastructure, lack of transport, difficult terrain as well as numerous socio-cultural barriers (13, 14).

The VHV Program has been in existence in PNG in some form for decades and the current program combines all previous lay health worker programs including village birth attendant, community based distributor, marasin meri (medicine woman), village health promoter, men’s health educator, and village health assistant (15). A review of the village birth attendants (VBA) program in Milne Bay Province in 2014 found that the

VBAs had been providing assistance to women during childbirth, with limited support from their communities or local health facilities, for 15-20 years (16). Despite the lack of support, VBAs were motivated by internal factors such as a desire to serve, religious beliefs and dedication to their communities (16). The review highlighted the need for clarity on the role of the VBA and better support and supervision of VBAs. However, it is not known if this experience in the VBA program is similar to that of the broader VHV

Program in PNG.

The Community Mine Continuation Agreement (CMCA) Middle and South Fly Health

Program is implemented in the Western Province of Papua New Guinea (PNG) (17).

The Program aligns with and supports the national policies and plans, one of which is the Village Health Volunteer Program. For many villages, the Program provided the first opportunity for people to have access to and support from a VHV. As part of the midline evaluation of the CMCA Middle and South Fly Health Program, we examined how the VHV program was implemented and documented the experience of the VHVs, 59

Chapter 3: Lessons from the VHV Program and considerations for policy

their supervising health workers and the communities they were engaged to serve. We synthesise the findings from this study in this paper with global literature for lay health worker programs to identify considerations for policy for PNG and other similar settings with low rates of formal health workers and challenging geography.

METHODS

Context

In PNG, the majority of the population (85%) live in rural areas (18). Health services in rural areas are delivered through aid posts, health sub-centres and health centres (19).

Aid posts are usually staffed by one community health worker (a formal health worker in PNG with two years of training, not to be confused with lay health workers in other countries who are called community health workers) (19). Health sub-centres and health centres are usually staffed by a mix community health workers and nursing officers (19). The World Bank identified a crisis in the health workforce where there is not only a shortage in the number of health workers but also an unequal distribution with fewer health workers in rural areas (13). Lay health workers are one mechanism to counteract the shortfall in health workers (1).

In PNG the primary role of a VHV is to be a link between communities and health services; to coordinate community based projects and organisations in relation to health activities; to promote a healthy community; and provide support for prevention of health problems (20).

Chapter 3: Lessons from the VHV Program and considerations for policy

Box 1: Duties that Village Health Volunteers are trained to undertake (20)

The Village Health Volunteer will be trained to: 1. Build and maintain linkages between the village and health service:  Link with the Aid Post CHW and mobile MCH team;  Encourage individuals and families to be self-reliant and to use health services when needed; and to use the support provided by the VHV. 2. Coordinate and cooperate with church, school, women’s and youth groups, village leaders, other community based projects and organisations:

 Work with other partners such as the Local Level Government, councillors, community-based organisations, etc.;  Assist village leaders to coordinate community activities in relation to health. 3. Promote a healthy community:  Be a positive role model for the village by practicing healthy habits;  Encourage safe, healthy practices in the village through health education;  Promote traditional beliefs and practices that support a healthy lifestyle;  Encourage men to support the health needs of women, pregnant mothers and both male and female children;  Mobilize village / clan members to be responsible for their own health;  Assist the rural health worker, program staff or other program health motivators to educate men and women about the support that the VHV can provide;  Assist health motivators to educate men and women about relevant health topics, using culturally appropriate methods and language.

4. Provide support to prevent health problems: Preventive roles for assisting pregnant women:  Identify pregnant women in the village and encourage them to go for regular ante-natal check by a health worker;  Advise pregnant women to go to a health centre for delivery;  Identify pregnant women at ‘high risk’ and encourage them to go to an equipped health facility for antenatal care and delivery. Preventive roles for assisting pregnant women with village delivery:  Assist a woman to have a safe normal delivery in the village (2nd, 3rd or 4th baby);*  Accompany a pregnant woman who has an obstetric emergency to an equipped health facility. Preventive roles for post-natal care:  Check the mother and baby in the village daily during the first five days after giving birth;  Encourage the mother to care for herself and the baby during the first six weeks after giving birth. Preventive roles for infant and child health:

 Encourage and support immunization of infants, children and pregnant women;  Assist in checking the growth and development of children;  Provide information to parents about nutrition;  Educate parents about diarrhoea and dehydration; fever; and respiratory problems in children; Preventive roles for child spacing:  Provide information on child spacing to all interested persons;  Provide information to individuals and couples on infertility;  Provide appropriate family planning methods to all interested persons;  Refer individuals and couples to a health centre to obtain other temporary and permanent methods or for management of complications. Preventive roles for sexually transmitted infection:  Educate individuals about sexually transmitted infections;  Encourage individuals and couples who may have a sexually transmitted infection to seek care at a health facility. Preventive roles for illness  Educate individuals and families about injuries; malaria; diarrhoea and dehydration; respiratory problems; anaemia; skin problems; and ear problems.

* In 2014 the National Department of Health provided a directive that VHVs were to refer all pregnant women to health facilities to give birth(21) 61

Chapter 3: Lessons from the VHV Program and considerations for policy

VHV Program implemented through the CMCA Middle and South Fly Health

Program

The VHV Program has been implemented in the Middle and South Fly Districts of

Western Province since 2014 as part of a larger program called the Community Mine

Continuation (CMCA) Middle and South Fly Health Program which aimed to improve health service delivery. Through the CMCA Middle and South Fly Health Program two

VHV Coordinators were employed and have overall responsibility for implementing the

VHV Program in the villages in the program catchment area. The catchment area covers 84 villages along the Fly River in Western Province, known as CMCA villages, starting from south of the town of Kiunga to the mouth of the Fly River (Figure 1). The villages are remote or very remote with extremely limited road infrastructure. Travel is mostly via boat on the river and then on foot from the river to villages. The CMCA

Middle and South Fly Program covers an estimated population of 50,813 (22).

Chapter 3: Lessons from the VHV Program and considerations for policy

Figure 1: Map of Western Province, Papua New Guinea, with CMCA Middle and

South Fly Health Program area shaded (*approximation only)

The VHV program has been progressively implemented in villages in the CMCA area using a cascading model, whereby VHV Training of Trainers was conducted followed by training of VHVs. All VHV training was carried out by the VHV Trainers, supported by the VHV Coordinators. First, VHV Trainers were identified through consultation with health workers and communities. These people were either health workers or community members (n=11) and participated in a 10 day training course. This training focused on teaching skills such as developing lesson plans and training materials, VHV training, and proposal writing to seek funds for VHV activities. These VHV Trainers where then supported by the VHV Coordinator to undertake village mobilisation to inform community members about the VHV Program. Also, during the mobilisation, a 63

Chapter 3: Lessons from the VHV Program and considerations for policy

Memorandum of Understanding between the VHV Program and the community was signed. The community also had to establish a Village Health Committee that would support the VHV.

During the mobilisation, the community identified two community members, one male and one female, for VHV training. The selection criteria for VHVs included the ability to read and write, spend a majority of their time living in the community, can speak up for the community and, for women, are married. The VHV training was then conducted in clusters of about 20 participants. A total of 11 VHV Trainers and 76 VHVs were trained covering 35 villages, largely in the Suki Fly Gogo, Dudi and Middle Fly CMCA Regions of the Middle and South Fly Districts. The VHV Training curriculum consisted of five modules: “Being a Better Volunteer,” “Learning about Health,” “Safe Motherhood,”

“Healthy Children,” and “Self-Help Health Care.”

The Village Planning Committee chairman, the Village Health Committee chairman, local health workers, and school teachers were invited to attend the final week of training where Community Action and Participation (CAP) training was undertaken (23,

24). Healthy Islands is one of the healthy settings, which are “settings-based approaches to health promotion, (which) involve a holistic and multi-disciplinary method which integrates action across risk factors” (25). The intention of CAP was to train the community to create a greater discussion in the villages to identify community health needs in order to guide the activities that the VHVs would implement.

After the VHVs were trained, the Program VHV Coordinators conducted visits to the

VHVs in the communities (1-4 times per year). In addition, local health workers were to provide supervision to the VHVs.

Chapter 3: Lessons from the VHV Program and considerations for policy

Study design

An explorative qualitative study was undertaken in September 2015 with interviews and focus group discussions, as outlined below. Data collection for the VHV study occurred as part of the CMCA Middle and South Fly Health Program midline evaluation (26).

Sample Villages for VHV Interviews and Focus Group Discussions

A sample of villages was selected for participation in the CMCA Middle and South Fly

Health Program Midline Evaluation as described elsewhere (26). The study was undertaken in two geographically distinct regions where the VHV Program had been implemented (referred to as Region 1 and Region 2). At each village, an attempt was made to conduct a focus group discussion with males and females separately and interview any VHVs or VHV trainers available in individually.

Key Informant Interviews with Health Workers

All available health workers at the selected facilities at the time the study team visited in 2015 were invited to participate in a semi-structured interview. The interview focussed on exploring the activities that the VHVs had undertaken, and how the VHV program could be improved.

Key Informant Interviews for the VHV Program

VHVs and VHV Trainers located in villages and a centrally located VHV Coordinator were invited to participate in a semi structured interview. The interview topics included activities undertaken since receiving VHV training, interaction with the supervising health facilities, and barriers and enablers to undertaking VHV work.

Focus Group Discussions with Communities

Focus group discussions allow for documentation of perceptions where participants can build on others’ comments and identify issues of most relevance or significance to the group (27). Focus group discussions were carried out for one group of 10 adult males and one group of 10 adult females, where possible, in each of the selected 65

Chapter 3: Lessons from the VHV Program and considerations for policy

villages. All participants were 18 years or older and were selected to cover a range of age groups. The following topics were covered in the focus group discussions: activities VHVs had undertaken, barriers and enablers to VHVs carrying out their work from a community perspective, community support for VHVs and what they thought

VHVs required to improve their work.

Data analyses

Interviews and focus group discussions were conducted in English or Tok Pisin with the assistance of an interview/focus group discussion guide with questions and prompts by two trained evaluators (authors LS and DA). The interviews and focus group discussions were recorded with an audio recorder. The interviews and focus group discussions were transcribed, translated if not in English, and uploaded into NVivo

(QSR International, 2011). Inductive thematic analyses were conducted by one author,

EF, whereby themes were identified from the transcriptions without a prior framework nor an extensive search of the literature (28). Common themes or patterns across interviews and group discussions were sought as well as areas of difference between different participants and participant groups (28).

Ethics

All key informants and focus group discussion participants provided written informed consent. The study was reviewed and approved by the PNG Medical Research

Advisory Committee (MRAC No. 15.08) and the University of New South Wales

(HC15466).

RESULTS

Key informant interviews were carried out with seven VHVs, three VHV trainers and eight health workers and eight focus group discussions with 10 people per group, across four villages (one male and one female focus group discussion per village) were conducted (Table 1).

Chapter 3: Lessons from the VHV Program and considerations for policy

Table 1: Number of focus group discussions and interviews by location

Interviews Community Village Focus Village Location Health Health Group Health Volunteer worker Discussion Volunteer Trainer Region 1 locations Health sub-centre and village 1 2 1 2 1 Village 2 2 - - - Aid Post - - - 1 Region 2 locations Health sub-centre - - - 5 Aid Post and village 1 - 1 - 1 Village 2 2 3 1 - Village 3 2 1 - - Village 4 - 1 - - Total 8 7 3 8

Themes from the qualitative data

The main themes drawn from the qualitative data highlighted the conflict between designated duties of a VHV and actual duties reported by VHVs, health workers and community members. In addition, issues of community support for VHVs and participation in VHV activities; incentives and remuneration; and the relationship of the

VHV program to the health system were common topics of discussion.

1. Conflict between designated duties of a VHV and actual duties reported

by VHVs, health workers and community members

The VHVs and Trainers interviewed reported undertaking a range of activities in line with their expected role: assisting community members who are sick to get to the health facility; assisting the community to build toilets, bathrooms, dish racks, rubbish pits; nutritional gardening; conducting awareness on health issues; encouraging mothers to attend the health facility for child health checks; and providing First Aid.

Several of the focus group discussions across villages did initially say the VHVs were not active, but on further querying it appears that the VHVs were active, but not necessarily doing the activities the community had expected, for example, one focus

Chapter 3: Lessons from the VHV Program and considerations for policy

group discussant described activities in line with the role (awareness, keeping the community clean). This may be due to a perception from both the community and health workers that the work of VHVs is similar to that of health workers working in a health facility. For example, when asked what could be done to improve the work of

VHVs, one community suggested to build an aid post for the VHV to work in.

I want the aid post to be here in [Village name], so that we’ll send these village

health volunteers for training so that they will come and help us over here.

(Female focus group discussion 1)

In some instances, the work described by the VHVs themselves was clinical, beyond that of the role of a VHV. Some VHVs cited working in a health facility, rather than doing community-based work, assisting Maternal and Child Health Clinics with the health workers, cleaning the health facility and assisting women in labour in the village and at the health facility.

That sister gives me that key in the afternoon. When it is night duty I take over.

How long I sit with that woman when she delivers, that's my work. But I don't

give her medicine. When everything is cleared I don't give her medicine. I just

leave her and tell her that sister will come and help you. I go back and tell sister

everything is done. (VHV 7)

The VHVs are to only supervise births if necessary but their main role in maternal health is to encourage and accompany women to the health facility to give birth.

However, from the interviews with VHVs, health workers and the community focus group discussions it appears that they are regularly supervising births either in the village or assisting health workers in the health facilities. In this instance, there was only one health worker and the VHV provided assistance to women who are in labour during the evenings. Another health worker, male, who cited the VHVs assisting with

Chapter 3: Lessons from the VHV Program and considerations for policy

births in the village, reasoned that the women prefer to be attended to by another woman.

Sometimes mothers used to get frightened and scared. They get frightened to a

man to help. They feel a female can train up to feel that they can help each

other. (Health Worker 2)

Furthermore, a couple of health workers expressed that VHVs required more clinical training, for example in treating illness in the community or being trained in a health facility. However, some VHVs and health workers were clearer about the VHV role not being about providing clinical care, but rather on prevention.

We are not allowed to practice medicine, but we give advice only, to look after

themselves when they are in need. (VHV Trainer 6)

We don't get support from them (health workers), but they send the word for us,

or they tell us to go and help them in the hospital. But we say that we did not

train for that, we are trained on the prevention side, not the curing. (VHV 4)

2. Incentives and remuneration

While the VHV position is not a paid position in Papua New Guinea, a common theme from VHVs and a couple of the health workers and community focus group discussions, all in different locations, was that there should be financial compensation for VHVs in recognition of their work. A majority of the VHVs expressed the desire to be paid for their work. One VHV saw the lack of pay as a disincentive when comparing their work to that of a paid health worker:

I want to do that work but in other side I say, I can do all the work but I am not

paid. That's the part sometimes it puts me down as I am not paid. You are

giving me this job every time, and I'm finding it hard… You [health worker] are

paid for it; I am not paid for it. (VHV 7)

Chapter 3: Lessons from the VHV Program and considerations for policy

A couple of the male focus group discussions raised the point that VHVs should receive some financial support.

The VHVs themselves, to my observations, they are really struggling and facing

hardships, like they are not paid. (Male focus group discussion 1)

Another thing I found out was that these volunteers need to be paid to because

they will improve their work. (Male focus group discussion 2)

Interviews with the health workers revealed that some health workers were informally making payments to the VHVs for their services.

They have a rate of, especially with deliveries, it's 20 kina per mother…. as a

gift because they are volunteers for the village. They are not any sort of a paid

health worker. (Health Worker 8)

Another health worker paid the VHVs similar rates (20-30 kina) for services. When queried as to why, they replied:

I feel sorry for them, so I say okay, I'll help you just for this. (Health Worker 1)

Generally, the reasons provided for payment for VHVs was for recognition for their hard work and their time spent doing activities which limited their time for collecting food and caring for their families.

In addition to comments on financial remuneration, one health worker stated that VHVs could be remunerated through non-monetary incentives that would specifically support the VHVs to undertake their work:

Like buy whatever they need like soap and all that stuff. For the community

also, they should be supporting, giving at least food or that to keep them going

to help the community. (Health Worker 4)

Chapter 3: Lessons from the VHV Program and considerations for policy

3. Community support for VHVs and participation in VHV activities

The community members were largely appreciative of the work of the VHVs and noted the changes in the village, particularly with the construction of toilets, showers and handwashing facilities. There was also an appreciation from community members of the VHV work focussing on health promotion. While the VHVs generally felt that they were well supported by the community, some VHVs noted that some community members did not listen to them or participate in their activities. One VHV offered a reason for a lack of community support:

We are in the village, we are hunters and gatherers. So some parents or some

families, they can stay at home and some they go out looking for food. (VHV 6)

One focus group discussion described poor health as a barrier for communities to participate in the VHV activities and also family obligations of the VHVs themselves creating a barrier for them to carry out their work:

If I have tooth pain, how can I help myself and go and do this work. (Male focus

group discussion, 2)

But the bad side of it I see is that, they are married, sometimes when their

children they are short of sago, they go to the bush and leave the work. That’s

one of the bad side of it I see. Maybe single is okay. (Male focus group

discussion, 2)

These quotes indicate that providing for families is both a factor in the VHV’s ability to undertake the work and the community’s ability to participate in and support VHV activities.

Chapter 3: Lessons from the VHV Program and considerations for policy

4. VHV support from the health system

The final main theme is the support VHVs received or would like to receive from the health system. Within this main theme, the sub-themes of medical supplies, transport, training and supervision were identified.

Lack of access to medical supplies was often cited by VHVs as an issue that affected their ability to undertake their work. Although the issue of medical supplies was not raised directly in the community focus group discussions, one focus group noted that if the VHVs don’t have supplies for first aid, they cannot carry out their work:

If they send the supply from [the program] yes that’s the time they will help us,

like sick person there or any big cut they will come and help. But if they have

nothing, or no supply…they will only help the people to take them to the

hospital, sub centre there (Women’s Focus Group Discussion 1).

Another common need cited by VHVs and VHV Trainers was transport. For VHVs the need was for transporting patients to health facilities. The VHV Trainers, who may be located in a different village to the VHVs they are required to supervise, cited the need for transport to go and supervise their VHVs in the communities. For VHV Trainers located near health facilities, even though they had access to a dinghy and outboard motor, they noted that access to fuel was a problem.

VHV must have our own transport of the communities so that [VHV] trainers,

they can visit all communities, do what they learn and to emphasise again to do

it. (VHV 1)

More training for VHVs was cited by VHVs, VHV Trainers, health workers and the community members as a need. VHVs wanted refresher training or training that focussed on delivering health awareness on priority topics. VHVs also cited that they

Chapter 3: Lessons from the VHV Program and considerations for policy

need ongoing training. One VHV Trainer noted that revision of training should occur more frequently than it currently does.

I would like if we can do some revision after three month. At the moment I wait

for maybe four, five months and then do some revision. I really want to go

around and try to motivate the VHVs to do something on the ground. (VHV

Trainer 2)

The VHVs and Trainers reported that they had variable contact with the health workers at their supervising health facility. The health workers that did cite providing supervision typically described the supervision occurring when the VHVs were working in the facility or assisting the health worker with a birth in the village. In some cases, the VHV worked independently of the health facility. One health worker did not believe they had a responsibility to supervise VHVs:

I [sort of] supervise them, but they're not there with me. I don't supervise them.

(Health worker 2)

The qualitative data revealed some important themes which were: conflict between designated duties of a VHV and actual duties reported by VHVs, health workers and community members; the importance of incentives and remuneration; conflicting obligations of the VHV between family and the community; and the VHV program and

VHV support from the health system, specifically in relation to supplies, training and supervision.

DISCUSSION

This study has described the experience of the implementation of the VHV Program in the Middle and South Fly Districts of PNG from the perspective of VHVs, VHV Trainers, health workers and community members. The key themes raised by the qualitative

Chapter 3: Lessons from the VHV Program and considerations for policy

findings are discussed in this section in the context of the global literature: conflict between designated duties of a VHV and actual duties reported by VHVs, health workers and community members; incentives and remuneration; community support for

VHVs and participation in VHV activties; and the integration of the VHV program within the health system.

Conflict between designated duties of a VHV and actual duties reported by VHVs, health workers and community members

The role of a VHV, as defined in the Minimum Standards for Village Health Volunteers in Papua New Guinea, is to be a link between communities and health services; coordinate community based projects and organisations in relation to health activities; promote a healthy community; and provide support to prevent health problems (15).

The VHVs are to work primarily in the community in which they live. However, in this study we found that some VHVs were routinely working in health facilities or there was an expectation from health workers and communities that they work in a health facility.

The most common activity that VHVs cited was attending to births. However, in 2014 the National Department of Health provided a directive that VHVs were to refer all pregnant women to health facilities to give birth (21). It is likely that the VHVs are assisting women to give birth to meet a community need due to issues of transportation from communities to health facilities, limited human resources available at health facilities or a preference for a female to assist with a birth rather than a male health worker. Furthermore there may also be an expectation from VHVs, health workers and the communities that VHVs attend to births based on knowledge of the previous Safe

Motherhood Program that was implemented in other parts of the province.

More broadly, there was an expectation from the community and health workers that the VHV work in health facilities. This expectation may be a reflection of community need for curative services alongside the constrained services available at health

Chapter 3: Lessons from the VHV Program and considerations for policy

facilities in PNG (29). Further, if health facilities to which VHVs refer to cannot provide adequate services, this can be a dis-incentive for VHVs as identified in India with lay health workers (30). Others have argued that inclusion of curative services for a lay health worker program can improve lay health worker motivation and also acceptance within their community (31, 32) which has implications for the VHV program in PNG.

While Byrne and Morgan (2011) outline potential for a larger role of VHVs in curative services in the critical areas of maternal and child health, such expansion of their role would have to be accompanied by greater integration with and support from the health system (33). Considering the challenges with health system support identified from this study, expansion of the role to curative services may be challenging in PNG. While the community and health workers were briefed on the role of VHVs at the initiation of the program, perhaps ongoing clarification of the VHV role is required to ensure community and health worker expectations are in line with the intended role of VHVs. Alternatively, the role of VHVs may need to be reviewed.

Incentives and remuneration

In this study there was a desire from VHVs, the community and health workers that there should be some sort of financial incentive for their work. Indeed, some health workers reported that they were already providing a fee for VHV services. Incentives are vitally important to the sustainability of lay health worker programs, recognising that lay health workers usually have low-incomes and the time spent doing work takes them away from other activities used to support themselves and their families (e.g. gardening) as identified in this and other studies (34). Furthermore, in PNG the wantok system, which solicits a strong sense of obligation between people from the same clan or extended family, may priorities family duties over VHV duties (35).While the World

Health Organization and literature suggests that lay health workers are to be paid to improve retention and motivation (1, 32, 36), Glenton (2010) argues that the evidence for this view is weak and cites the lay health worker program in Nepal with no 75

Chapter 3: Lessons from the VHV Program and considerations for policy

remuneration but a very low attrition rate (5%)(37). The authors suggest this is potentially due to the culture of volunteerism and providing non-financial incentives that meet the needs of the lay health worker (37). Further, remuneration requires a system to enable regular payment. Irregular payment can lead to attrition and demotivation as the remuneration may be their sole source of income (38, 39). Remuneration is likely to be difficult in PNG, due to limited financial resources and the complexity of enabling a regular system for payment.

The PNG VHV policy discourages payment of cash to VHVs, and other non-financial incentives are to be determined at the local village level (15). The VHV Policy states that such incentives should be agreed upon at the village level (20). However, there are incentives identified in other settings that a community is not likely to be able to provide to a VHV, such as job aides, supplies, ongoing training and a career pathway into the formal health sector (40, 41). These types of incentives should be coordinated through the health sector. There needs to be an understanding by the health sector and VHV program implementers of the VHV motivations and link to incentives in the context of

PNG (37). Appropriate incentives may contribute towards addressing the issue of

VHVs’ obligations to family as a barrier to their VHV work.

Community support for VHVs and participation in VHV activities

Community support is clearly a factor in sustainability of lay health worker programs (1,

34, 40). The community can be a source of motivation to become a lay health worker as they are valued, have social prestige and credibility, and provide recognition for the lay health workers’ activities(34). Lack of community participation can be a de- motivator leading to attrition in lay health worker programs (34).

The VHV Program in PNG fosters community support through involving the community in the selection of VHVs from the community itself and a needs assessment at the community mobilisation stage to guide VHV activities. The literature supports

Chapter 3: Lessons from the VHV Program and considerations for policy

community selection of VHVs as the community is likely to select people that are already trusted and supported by the community (32, 42, 43). In East Sepik Province in

Papua New Guinea, community members were more likely to seek treatment for their febrile child if the VHV had higher social capital, specifically social capital obtained through links with people in formal positions outside their village (44). Identifying people who are well linked externally to their village could potentially be a criterion for selection of VHVs to improve community engagement and participation. In this study, some

VHVs from different locations cited that there was low participation from the community in their activities. While community selection of VHVs did occur, one VHV suggested there was a barrier for community participation in VHV activities due to their own commitments to providing resources to their family. Other explanations for low community participation were not identified from the interviews in this study and is an area for further investigation.

In this study, one focus group discussant suggested married VHVs may have difficulties in finding time to do their work. However, in Solomon Islands, when younger lay health workers married they were more likely to drop out than older, married workers (38). In a systematic review the evidence on the effect of marriage on retention was mixed (32). The issue may in fact be around time available to contribute to VHV activities in the context of household obligations as lay health workers with fewer household duties were more active in Kenya and Bangladesh (32, 45, 46).

VHV Support from the health system

Shortage of supplies, inadequate or irregular supervision, lack of transport, insufficient initial and ongoing training and inadequate linkage to the health system have been identified as issues in the current study, and have been well documented as factors impacting lay health worker motivation and program sustainability and scale up in other country contexts (29, 32, 34, 40, 47, 48). These issues are reflective of the challenges

Chapter 3: Lessons from the VHV Program and considerations for policy

experienced in the PNG health system overall and other low resource settings. The concerns raised about poor supervision of lay health workers in the 1980s (11) and more recently in the review of VBAs (16) in PNG was again raised in this study.

Supervision has been demonstrated as a factor for maintaining motivation of lay health workers (39, 40) and is associated with better performance (49). The evidence on the effectiveness of supervision approaches is limited, but good supervision should generally involve supportive approaches, community monitoring, quality assurance and problem solving (50). Potentially, specific training for supervisors of VHVs, i.e. health workers at nearby health facilities, on their supervisory role may be useful as the attitude of the supervisor and the quality and frequency of the supervision are important for lay health worker retention and performance (29, 32). While technological solutions for improving supervision, such as mobile apps and mobile phones on a closed user groups (group members can make unlimited calls to other groups members), have been trialled, they did not result in improved in supervision, job satisfaction or motivation in Sierra Leonne (51). Frequent supervision of VHVs by health workers should be planned and budgeted for, including budget for travel to VHVs in their communities. As health workers need to travel to villages where VHVs are located to conduct outreach clinics, efficiencies may be gained if supervision were to occur in conjunction with outreach clinics.

Monitoring and evaluation

There are limited rigorous evaluations of lay health worker programs in the global literature. While this study contributes to the knowledge of a lay health worker program in PNG, it is also not a comprehensive evaluation. VHVs are integral to the implementation of the National Health Plan (18) and improving access to health care in rural and remote populations in PNG, yet there are no VHV specific indicators in the

National Health Monitoring and Evaluation Plan (52).

Chapter 3: Lessons from the VHV Program and considerations for policy

There is a need to understand attrition rate, length of time in service and factors affecting VHV retention in PNG in order to plan for ongoing recruitment and training. An evaluation of a lay health worker program in Solomon Islands identified that lay health workers worked on average 4.9 years and over a 13 year period there was a 47% attrition rate (38). While there are some VBAs in PNG who have provided services for decades (16), we do not know the attrition rate and average length of service for the

VHV Program in PNG. A minimal set of nationally agreed indicators and targets, with data collection integrated within the existing National Health Information System, should be considered. The data from the National Health Information System can potentially be reportable at a village level, which previously has not been possible, if the recent pilot of a mobile reporting platform is rolled out nationally (53, 54).

Limitations

There are several limitations to this study. Firstly, we attempted to interview at least 10

VHVs, however, as the interviewers were only in a village for a brief period, not all

VHVs were available for interview at the time of visit. Secondly, the review of literature was largely limited to publicly available literature. For lay health worker programs there is likely to be a plethora of evaluations that may not be publicly available. Despite these limitations, the combination of qualitative data triangulated from multiple perspectives

(VHVs, VHV trainers, health workers and community members) is a strength for discussing issues with lay health worker programs in the context of Papua New

Guinea.

CONCLUSION This study provided the perspectives of VHVs, health workers and the community on the VHV Program in Western Province, PNG. The main themes identified were conflict between designated duties of a VHV and actual duties reported by VHVs, health

Chapter 3: Lessons from the VHV Program and considerations for policy

workers and community members; community support for VHVs and participation in

VHV activities; incentives and remuneration; and VHV support from to the health system, specifically supplies, training and supervision. This study offers considerations for the National VHV Policy in PNG. In order for the VHV Program to be sustained,

VHVs must be better integrated into the health system in order to be provided with adequate supervision, ongoing training, and medical supplies. Community participation and support, through selection of community members for VHV training, active engagement in the activities of VHVs, and provision of appropriate incentives for time spent doing VHV activities, is also vital for the success of the VHV Program.

Chapter 3: Lessons from the VHV Program and considerations for policy

REFERENCES

1. Lehmann U, Sanders D. Community health workers: What do we know about them? Geneva: World Health Organization, 2007.

2. Olaniran A, Smith H, Unkels R, Bar-Zeev S, van den Broek N. Who is a community health worker? - a systematic review of definitions. Global Health Action.

2017;10(1):1272223.

3. Schneider H, Okello D, Lehmann U. The global pendulum swing towards community health workers in low- and middle-income countries: a scoping review of trends, geographical distribution and programmatic orientations, 2005 to 2014. Human

Resources for Health. 2016;14(1):65.

4. Lewin SA, Dick J, Pond P, Zwarenstein M, Aja G, van Wyk B, Bosch-Capblanch

X, Patrick M. Lay health workers in primary and community health care. The Cochrane

Database of Systematic Reviews. 2005(1):Cd004015.

5. Glenton C, Scheel IB, Lewin S, Swingler GH. Can lay health workers increase the uptake of childhood immunisation? Systematic review and typology. Tropical

Medicine & International Health. 2011;16(9):1044-53.

6. Lewin S, Munabi-Babigumira S, Glenton C, Daniels K, Bosch-Capblanch X, van

Wyk BE, Odgaard-Jensen J, Johansen M, Aja GN, Zwarenstein M, Scheel IB. Lay health workers in primary and community health care for maternal and child health and the management of infectious diseases. The Cochrane Database of Systematic

Reviews. 2010(3):Cd004015.

7. van Ginneken N, Tharyan P, Lewin S, Rao GN, Meera SM, Pian J,

Chandrashekar S, Patel V. Non-specialist health worker interventions for the care of mental, neurological and substance-abuse disorders in low- and middle-income countries. The Cochrane Database of Systematic Reviews. 2013(11):Cd009149.

8. Dudley L, Azevedo V, Grant R, Schoeman JH, Dikweni L, Maher D. Evaluation of community contribution to tuberculosis control in Cape Town, South Africa. The

Chapter 3: Lessons from the VHV Program and considerations for policy

International Journal of Tuberculosis and Lung Disease : the official journal of the

International Union against Tuberculosis and Lung Disease. 2003;7(9 Suppl 1):S48-55.

9. Perry H, Crigler L, Hodgins S. Developing and Strengthening Community

Health Worker Programs at Scale. USAID: Maternal and Child Health Integrated

Program, 2014.

10. de Vries DH, Pool R. The Influence of Community Health Resources on

Effectiveness and Sustainability of Community and Lay Health Worker Programs in

Lower-Income Countries: A Systematic Review. PloS One. 2017;12(1):e0170217.

11. Garner PA. Voluntary village health workers in Papua New Guinea. Papua and

New Guinea Medical Journal. 1989;32(1):55-60.

12. O'Keefe D, Davis J, Yakuna G, Van Gemert C, Morgan C. Formal and informal maternal health care: comparing the service provision of health facilities and village health volunteers in East Sepik Province. Papua and New Guinea Medical Journal.

2011;54(3-4):147-53.

13. The World Bank. PNG Health Workforce Crisis: A Call to Action 2011 [cited

2018 27 August]. Available from: http://documents.worldbank.org/curated/en/216511468332461651/pdf/NonAsciiFileNa me0.pdf.

14. Vallely LM, Homiehombo P, Kelly AM, Vallely A, Homer CS, Whittaker A.

Exploring women's perspectives of access to care during pregnancy and childbirth: a qualitative study from rural Papua New Guinea. Midwifery. 2013;29(10):1222-9.

15. National Department of Health. Policy for village health initiatives and Village

Health Volunteer activities. 2000.

16. Vallely L, Paul R, Naidi B, Morewaya J, Kariwiga G, Vallely A, Morgan C,

Homer C. “I have a heart to help the mothers” : 25 years of the Village Birth Attendant

Program in Milne Bay Province, Papua New Guinea. Papua New Guinea Medical

Journal. 2016;59(3-4):164-77.

Chapter 3: Lessons from the VHV Program and considerations for policy

17. CMCA Middle and South Fly Health Program: Abt Associates 2013 [cited 2016

2 January ]. Available from: http://www.cmsfhp.org/about-us/.

18. Goverment of Papua New Guinea. National Health Plan 2011-2020: Volume 1

Policies and Strategies Port Moresby 2010.

19. National Health Service Standards for Papua New Guinea 2011-2020.

Government of Papua New Guinea; 2011.

20. Minimum Standards for Village Health Volunteers in Papua New Guinea. In:

National Department of Health, editor. 2003.

21. Papua New Guinea Department of Health. Public Health Policies. Port

Moresby2014. p. 117-44.

22. National Statistical Office. 2011 National Population & Housing Census Ward

Population Profile Southern Region. Port Moresby 2014.

23. Temu P, Chen PC. The Papua New Guinea vision of Healthy Islands. Pacific

Health Dialog. 1999;6(2):253-8.

24. National Policy on Health Promotion for Papua New Guinea. In: Ministry of

Health, editor.

25. World Health Organization. Healthy Settings 2017 [cited 2017 24 February].

Available from: http://www.who.int/healthy_settings/en/.

26. Abt JTA. Strengthening Health Services in Western Province, Papua New

NFHSDP-CMCA-MSFHP_DFAT-Print_PA-signed.pdf.

27. Ritchie J. Not everything can be reduced to numbers. In: Berglun C, editor.

Health Research Melbourne: Oxford University Press; 2001.

28. Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Research in Psychology 2006;3(2):77-101.

Chapter 3: Lessons from the VHV Program and considerations for policy

29. Phiri SC, Prust ML, Chibawe CP, Misapa R, van den Broek JW, Wilmink N. An exploration of facilitators and challenges in the scale-up of a national, public sector community health worker cadre in Zambia: a qualitative study. Human Resources for

Health. 2017;15(1):40.

30. George MS, Pant S, Devasenapathy N, Ghosh-Jerath S, Zodpey SP. Motivating and demotivating factors for community health workers: A qualitative study in urban slums of Delhi, India. WHO South-East Asia Journal of Public Health. 2017;6(1):82-9.

31. Curtale F, Siwakoti B, Lagrosa C, LaRaja M, Guerra R. Improving skills and utilization of Community Health Volunteers in Nepal. Social Science & Medicine.

1995;40(8):1117-25.

32. Kok MC, Dieleman M, Taegtmeyer M, Broerse JE, Kane SS, Ormel H, Tijm

MM, de Koning KA. Which intervention design factors influence performance of community health workers in low- and middle-income countries? A systematic review.

Health Policy and Planning. 2015;30(9):1207-27.

33. Byrne A, Morgan C. Improving maternal, newborn and child health in Papua

New Guinea through Family and Community Health Care Policy Report [Internet]. 2011

23 August 2017. Available from: https://www.worldvision.com.au/docs/default- source/publications/australia-and-the-pacific/policy-report---family-and-community- care-png-(2).pdf?sfvrsn=6.

34. Shipton L, Zahidie A, Rabbani F. Motivating and Demotivating Factors for

Community Health Workers Engaged in Maternal, Newborn and Child Health Programs in Low and Middle-Income Countries: A Systematic Review. Journal of the College of

Physicians and Surgeons--Pakistan. 2017;27(3):157-65.

35. Asante A, Hall J. A review of health leadership and management capacity in

Papua New Guinea. Sydney: University of New South Wales, 2011.

36. Task Force for Scaling Up Education and Training for Health Workers GHWA.

Scaling Up, Saving Lives 2008 24 August 2017. Available from:

Chapter 3: Lessons from the VHV Program and considerations for policy

http://www.who.int/workforcealliance/documents/Global_Health%20FINAL%20REPOR

T.pdf.

37. Glenton C, Scheel IB, Pradhan S, Lewin S, Hodgins S, Shrestha V. The female community health volunteer programme in Nepal: Decision makers’ perceptions of volunteerism, payment and other incentives. Social Science & Medicine.

2010;70(12):1920-7.

38. Chevalier C, Lapo A, O'Brien J, Wierzba TF. Why do village health workers drop out? World Health Forum. 1993;14(3):258-61.

39. Rabbani F, Shipton L, Aftab W, Sangrasi K, Perveen S, Zahidie A. Inspiring health worker motivation with supportive supervision: a survey of lady health supervisor motivating factors in rural Pakistan. BMC Health Services Research. 2016;16(1):397.

40. Bhattacharyya K, Winch P, LeBan K, Tien M. Community Health Worker

Incentives and Disincentives: How They Affect Motivation, Retention, and

Sustainability. Arlington, Virginia: Published by the Basic Support for Institutionalizing

Child Survival Project (BASICS II) for the United States Agency for International

Development; 2001.

41. Sanou AK, Jegede AS, Nsungwa-Sabiiti J, Siribie M, Ajayi IO, Turinde A,

Oshiname FO, Serme L, Kabarungi V, Falade CO, Kyaligonza J, Afonne C, Balyeku A,

Castellani J, Gomes M. Motivation of Community Health Workers in Diagnosing,

Treating, and Referring Sick Young Children in a Multicountry Study. Clinical infectious diseases : an official publication of the Infectious Diseases Society of America.

2016;63(suppl 5):S270-s5.

42. Turinawe EB, Rwemisisi JT, Musinguzi LK, de Groot M, Muhangi D, de Vries

DH, Mafigiri DK, Pool R. Selection and performance of village health teams (VHTs) in

Uganda: lessons from the natural helper model of health promotion. Human Resources for Health. 2015;13:73.

43. Abbey M, Bartholomew LK, Nonvignon J, Chinbuah MA, Pappoe M, Gyapong

M, Gyapong JO, Bart-Plange C, van den Borne B. Factors related to retention of 85

Chapter 3: Lessons from the VHV Program and considerations for policy

community health workers in a trial on community-based management of fever in children under 5 years in the Dangme West District of Ghana. International Health.

2014;6(2):99-105.

44. Inoue Y, Takahashi D, Kondo N, Yoshii A, Sekihara M, Hombhanje FW,

Tsukahara T. Village health volunteers' individual social capital and caretakers' health service utilization for febrile children in Malaria-endemic villages in Papua New Guinea.

Transactions of the Royal Society of Tropical Medicine and Hygiene.

2017;111(11):490-6.

45. Alam K, Tasneem S, Oliveras E. Retention of female volunteer community health workers in Dhaka urban slums: a case-control study. Health Policy and

Planning. 2012;27(6):477-86.

46. Olang'o CO, Nyamongo IK, Aagaard-Hansen J. Staff attrition among community health workers in home-based care programmes for people living with HIV and AIDS in western Kenya. Health Policy (Amsterdam, Netherlands). 2010;97(2-3):232-7.

47. Bhutta ZA, Lassi ZS, Pariyo G, Huicho L. Global Experience of Community

Health Workers for Delivery of Health Related Millenium Development Goals: A

Systematic Review, Country Case Studies, and Recommendations for Integration into

National Health Systems: World Health Organization; 2010.

48. Abbatt F. Scaling Up Health and Education Workers: Community Health

Workers. 2005.

49. Wanduru P, Tetui M, Tuhebwe D, Ediau M, Okuga M, Nalwadda C, Ekirapa-

Kiracho E, Waiswa P, Rutebemberwa E. The performance of community health workers in the management of multiple childhood infectious diseases in Lira, northern

Uganda - a mixed methods cross-sectional study. Global Health Action.

2016;9(1):33194.

50. Hill Z, Dumbaugh M, Benton L, Kallander K, Strachan D, ten Asbroek A,

Tibenderana J, Kirkwood B, Meek S. Supervising community health workers in low-

Chapter 3: Lessons from the VHV Program and considerations for policy

income countries--a review of impact and implementation issues. Global Health Action.

2014;7:24085.

51. Vallieres F, McAuliffe E, van Bavel B, Wall PJ, Trye A. There's No App for That:

Assessing the Impact of mHealth on the Supervision, Motivation, Engagement, and

Satisfaction of Community Health Workers in Sierra Leone. Ann Glob Health.

2016;82(5):936-49.

52. Papua New Guinea Health Sector Monitoring and Evaluation Strategic Plan:

National Health Plan (2011-2020) Port Moresby: National Department of Health 2012.

53. Ashwell HE, Barclay L. Problems measuring community health status at a local level: Papua New Guinea's health information system. Rural and Remote Health.

2010;10(4):1539.

54. Rosewell A, Makita L, Muscatello D, John LN, Bieb S, Hutton R, Ramamurthy

S, Shearman P. Health information system strengthening and malaria elimination in

Papua New Guinea. Malaria Journal. 2017;16(1):278.

CHAPTER 4: CONTEXTUAL FACTORS AND HEALTH SERVICE PERFORMANCE IN PAPUA NEW GUINEA

Chapter 4: Contextual factors and health service performance

1. Chapter prelude

Abt Associates was contracted to undertake a formative evaluation for the Rural

Primary Health Services Delivery Project, which was implemented by the Project

Support Unit of the National Department of Health, Papua New Guinea. This chapter details the analysis of baseline indicators and contextual analysis performed as part of the formative evaluation.

2. Background

The Rural Primary Health Services Delivery Project (the Project) is an initiative of the

Government of Papua New Guinea and aims is to improve the coverage and quality of primary health care services delivered in rural Papua New Guinea through a range of initiatives at the national, provincial, district, health facility and community level. The

Project operates in two districts in each of eight provinces: Central and Southern

Region in the Autonomous Region of Bougainville; Wewak and Maprik in East Sepik;

Okapa and Kainantu in Eastern Highlands; Laiagam and Kompian-Ambum in Enga;

Alotau and Kiriwina-Goodenough in Milne Bay; Bulolo and Menyamya in Morobe;

Talasia and Kandrian-Gloucester in West New Britain and Tambul Nebilyer and Mul-

Baiyer in Western Highlands (1).

A formative evaluation is defined as a “rigorous assessment process designed to identify potential and actual influences on the progress and effectiveness of implementation efforts” (2). Contextual analyses form an important part of this assessment. Context can help explain why a program did or did not work (3). In this project we brought together health indicator data and key informant interviews to describe the current performance of health facilities and factors which affect performance.

Chapter 4: Contextual factors and health service performance

3. Project overview

This project used a sequential explanatory mixed methods design. The quantitative data, the health indicators that describe health service delivery performance at the district and health facility level, were analysed to described and compare performance between and within districts. Subsequently these indicators were used for presentation to key informants, provincial health administrators, in semi-structured interviews to identify factors which affect performance.

4. Contribution to the project

I led the design, analysis and writing of this project. While health indicators are available at the provincial and district level in the annual Sector Performance Review

Reports in Papua New Guinea, indicators at the facility level are not. In this study, I used annual data from the National Health Information System at the facility and district level to calculate the indicators. To understand the context of performance, myself and two team members conducted interviews with eight provincial health advisers (or equivalent position). I then coded and analysed the interview transcripts. I led the writing of the report for the Project and the manuscript with input from the co-authors.

5. Project outputs

The main output of this project was a report for the Rural Primary Health Services

Delivery Project (not included in this thesis). This report was used to develop a manuscript for publication.

Chapter 4: Contextual factors and health service performance

Journal publication

Emma Field; Jethro Usurup; Alexander Rosewell; Sally Nathan. Contextual factors and health service performance from the perspective of the provincial health administrators in Papua New Guinea. Rural and Remote Health, [accepted 14 May 2018].

Declaration

I certify that this manuscript was a direct result of my research towards this DrPH, and that reproduction in this thesis does not breach copyright regulations.

......

Emma Field

6. Ethics considerations

The key informant interviews were conducted only after informed consent was obtained where they were assured their anonymity would be maintained. The audio recording was used for transcription purposes only and the transcripts were de-identified. The study was approved by the University of New South Wales (HC15208) and the Papua

New Guinea Medical Research Advisory Committee (15.10).

Chapter 4: Contextual factors and health service performance

7. References

1. Asian Development Bank. Project Administration Manual: PNG Rural Primary

Health Service Delivery Project 2011 [cited 2018 27 August]. Available from: https://www.adb.org/sites/default/files/project-document/74336/41509-013-png- pam.pdf.

2. Stetler CB, Legro MW, Wallace CM, Bowman C, Guihan M, Hagedorn H,

Kimmel B, Sharp ND, Smith JL. The role of formative evaluation in implementation research and the QUERI experience. Journal of general internal medicine. 2006;21

Suppl 2:S1-8.

3. Adam T, Hsu J, de Savigny D, Lavis JN, Rottingen JA, Bennett S. Evaluating health systems strengthening interventions in low-income and middle-income countries: are we asking the right questions? Health Policy and Planning. 2012;27

Suppl 4:iv9-19.

Chapter 4: Contextual factors and health service performance

8. Manuscript

Title: Contextual factors and health service performance from the perspective of the provincial health administrators in Papua New Guinea

Authors: Field, Emma1,2,3; Usurup, Jethro1; Rosewell, Alexander3; Nathan, Sally3

Institutions

1. Abt Associates

2. Menzies School of Health Research

3. School of Public Health and Community Medicine, UNSW Australia

Abstract

Background: The Rural Primary Health Services Delivery Project aims to improve the quality and coverage of health services to rural populations in Papua New Guinea.

There are limitations in measuring performance of such projects through analysis of health information system data alone due to data quality issues and a multitude of unmeasured factors which affect performance. A mixed methods study was undertaken to understand the contextual factors that affect health service performance.

Methods: A performance assessment framework was developed including service delivery indicators derived from the National Health Information System. Prior to implementation, a baseline analysis of the indicators was undertaken. Subsequently semi-structured interviews were conducted with health administrators where they were asked about factors they perceived to influence health facility performance. During the interviews, key informants were provided with health indicators for their province and asked to interpret the performance of facilities. Interviews were transcribed and inductive thematic analysis performed.

Results: Performance indicators varied greatly within and between districts. Key informants cited a number of reasons for this variation. Health facilities accessible by road in urban areas, with competent and/or higher-level staff and health services 94

Chapter 4: Contextual factors and health service performance

operated by churches or private companies were cited as contributors to high performance. For high performing districts, key informants also discussed use of health information, planning and targeted strategies to improve performance. Inadequate numbers of staff, poorly skilled staff, funding delays and challenging geography were major contributors noted for poor performance.

Conclusion: Analysis of quantitative indicators needs to be performed at health facility level in order to understand district level performance. Interpretation of performance through key informant interviews provided useful insight into previously undocumented contextual factors affecting health delivery performance. The sequential explanatory mixed-methods design could be applied to evaluations of other health service delivery programs in similar contexts.

Chapter 4: Contextual factors and health service performance

Introduction

The Rural Primary Health Services Delivery Project (the Project) is an initiative of the

Government of Papua New Guinea (PNG). The Project aims to improve the coverage and quality of rural primary health care services through a range of initiatives at the national, provincial, district, health facility and community level (1). The Project is being piloted in two districts in eight provinces, with the intention of further roll out if effective.

The Project is expected to improve health system performance in a range of access, quality and maternal and child health indicators.

While health programs are largely measured through quantitative indicators, it is acknowledged by the World Health Organization that “Health systems are complex and their performance and impact are difficult to capture using only quantitative indicators”

(2). The context within which a program is implemented can have an effect on the success or otherwise of the program. In order to measure progress and impact of a program, or scale up an intervention, it is important to understand the contextual factors which may influence performance (3, 4). In PNG, there are several factors that have been identified to negatively impact health facility performance, including: remoteness; access to funds; and cultural beliefs of the community, particularly around family planning, pregnancy and birth (5-7). These factors have largely been identified through quantitative studies where data on predefined variables were used but which may not capture the full range of factors that could influence performance. A key challenge for this evaluation was to measure changes in coverage and quality of primary health care in eight very different provinces.

Qualitative data is regularly used in monitoring and evaluation of health programs (4).

Health administrators can offer a useful perspective on the reasons for high or low performance at the health facility and district level due to their contextual knowledge. In order to understand the contextual factors which may affect performance of the Project, we analysed the performance of key indicators and conducted a contextual analysis 96

Chapter 4: Contextual factors and health service performance

through interviews with senior Provincial health administrators to document perceived factors affecting performance at baseline. We propose that this sequential explanatory mixed-methods approach to the contextual analysis could be beneficial for evaluating complex programs in low-resource settings.

Methods

Setting

The Project commenced in 2012 and operates in two districts in each of eight provinces: Central and Southern Region in the Autonomous Region of Bougainville;

Wewak and Maprik in East Sepik; Okapa and Kainantu in Eastern Highlands; Laiagam and Kompian-Ambum in Enga; Alotau and Kiriwina-Goodenough in Milne Bay; Bulolo and Menyamya in Morobe; Talasia and Kandrian-Gloucester in West New Britain and

Tambul Nebilyer and Mul-Baiyer in Western Highlands (1). These eight provinces are spread across the four regions of Papua New Guinea (Highlands, Islands, Momase and Papua Regions). The Project supports the review or development of national policies and standards; establishment of partnerships at provincial level with state and non-state partners in health; health worker training; infrastructure development through construction of two Community Health Posts in each district; and community level health promotion activities to improve demand for services (1).

Study design

Two health system consultants were engaged to conduct a formative evaluation during implementation from 2013 to 2017. A Performance Assessment Framework was developed for the Project. In line with best practice, and not to duplicate data collection systems, the indicators were aligned with the National Health Plan Monitoring and

Evaluation Framework and data for the indicators were taken from the existing National

Health Information System (NHIS) at the facility and district level (8). A sequential explanatory mixed-methods study design was employed involving analyses of the 97

Chapter 4: Contextual factors and health service performance

indicators in the Performance Assessment Framework for the baseline year (2013) and subsequent key informant interviews with provincial health administrators from participating provinces to help understand the context of the performance indicator (9).

Indicator analysis

The NHIS is used to record data on outpatient and other services provided at health facilities. Data were transcribed from automated reports from the NHIS for 2013 into

Microsoft Excel. The indicators calculated were outpatients per person per year, outreach clinics per 1000 children less than 5 years of age, measles immunisation coverage at 9-11 months, family planning couple years protection per 1000 women of reproductive age, proportion of pregnant women who at least one antenatal care visit

(ANC1) and proportion of pregnant women had a supervised delivery using the methods defined in the National Health Plan Monitoring and Evaluation Plan (8). All indicators were adjusted for missing monthly summary reports by dividing by the percentage of reports submitted for the year. Health facilities with no catchment population provided were excluded as rates or per cent coverage could not be calculated. Indicators at the facility and district level for each indicator were compared to National targets (8).

Key Informant Interviews

Face-to-face key informant interviews were conducted with all Provincial Health

Advisers or Provincial Health Authority Chief Executive Officers (interviewees) from the

Project provinces. The interviews were conducted in English by three trained monitoring and evaluation staff. The interviewees were considered key informants having in-depth knowledge of the health system with over 10 years of experience.

The interviews were semi-structured with questions about factors affecting access, utilization and quality of health services. The interviewees were provided with a table that included the indicators from the NHIS data at the facility level for districts within

Chapter 4: Contextual factors and health service performance

their provinces. Some of the probing questions were derived from these tables, specifically asking about the context in which facilities with exceptionally high or low performance in indicators operate. Audio recordings were transcribed and data were coded using Nvivo 10 (QSR International). Firstly, the data were coded according to the question. Secondly, inductive thematic analysis was conducted on three of the interviews which means that themes were identified from the data (10). From this initial analysis, the main themes identified were geography, cultural and religious beliefs, human resources, financing and health information, with further themes identified within these main themes. A coding framework with these themes was developed and applied to all interviews.

Ethical considerations

The key informant interviews were conducted only after informed consent was obtained. The audio recording was used for transcription purposes only and the transcripts were de-identified. While a diversity of quotes was sought for presentation in this paper, interviewee numbers have been removed to protect anonymity. The study was approved by the PNG Medical Research Advisory Committee (15.10) and the

University of New South Wales (HC15208).

Results

NHIS analysis

For brevity, only four of the most relevant indicators from the Performance Assessment

Framework are presented covering health system access, and maternal and child health. The indicators presented are outreach clinics per 100 children less than 5 years of age; measles vaccination coverage at 12 months of age; ANC1 coverage; and supervised delivery coverage. The performance indicators varied widely across the 16 districts (Table 1 and 2). Both districts in Milne Bay met all four targets for the 99

Chapter 4: Contextual factors and health service performance

indicators presented. Conversely no targets were met in Central Region in the

Autonomous Region of Bougainville, Kandrian-Gloucester in West New Britain and both Bulolo and Menyamya in Morobe.

Six districts met the outreach clinic rate coverage target. However, of these districts that met the target 100% (7/7) of facilities in Kiriwina-Goodenough met the target whereas in Kompian-Ambum only 29% of facilities (2/7) met the target. Seven districts met the target for measles vaccination with three districts notably exceeding 100%.

Similar to the outreach clinic rate, there was a variable proportion of facilities within these districts that met the target for measles vaccination from 29% (2/7) in Maprik to

100% (3/3) in Kainantu. Seven districts met the ANC1 target. In Kainantu, the ANC1 target was not met despite all facilities listed in the NHIS having met the target. Six districts met the target for supervised deliveries and of these districts Laiagap-Porgera and Wewak had only one facility that met the target (1/6 and 1/9 respectively) while in

Kiriwina-Goodenough 86% of facilities (6/7) met the target.

100

Chapter 4: Contextual factors and health service performance

Table 1: Indicator performance by district, 2013

Province District Indicator performance Outreach Measles ANC 1st Supervised clinics/1000 vaccination visit delivery population coverage coverage coverage <5yrs (%) (%) (%) Autonomous Central 34 41 41 23 Region of 74 37 73 58 Bougainville South Eastern Okapa 7 61 37 9 Highlands Kainantu 22 77 39 34 Laiagap- 17 45 92 47 Porgera Kompian- 54 109 55 33 Enga Ambum Wewak 15 59 86 56 East Sepik Maprik 11 91 54 19 Alotau 101 72 72 73 Kiriwina- 110 102 80 53 Milne Bay Goodenough Bulolo 15 51 53 15 Morobe Menyamya 20 22 41 11 Kandrian- 33 51 50 31 West New Gloucester Britain Talasea 32 51 85 46 Mul Baiyer 57 100 52 19 Western Tambul- 85 216 70 23 Highlands Nebilyer 2013 national target 44 64% 70% 41% Number of districts where 6 (38%) 7 (44%) 7 (44%) 6 (38%) target met (%) *Shaded cells indicate 2013 target met See Discussion for possible explanations for % coverage exceeding 100%.

101

Chapter 4: Contextual factors and health service performance

Table 2: Number of facilities where indicator met (%), by district, 2013

Province District Number of Number of facilities where indicator met/ Total facilities in number of facilities in analysis analysis Outreach Measles ANC 1st Supervised (Facilities clinics/1000 vaccination visit delivery excluded*) pop <5yrs coverage coverage coverage* (%) Autonomous Central 6 (1) 4/6 4/6 2/6 2/6 Region of Bougainville South 14 (1) 8/14 2/14 3/14 9/14 Eastern Okapa 4 (0) 1/4 0/4 0/4 0/4 Highlands Kainantu 3 (0) 1/3 3/3 3/3 2/3 Enga Laiagap- Porgera 6 (2) 2/6 1/6 3/6 1/6 Kompian- Ambum 7 (2) 2/7 5/7 1/7 1/7 East Sepik Wewak 9 (0) 1/9 2/9 2/9 1/9 Maprik 7 (0) 0/7 2/7 1/7 1/7 Milne Bay Alotau 15 (1) 13/15 10/15 9/15 4/14 Kiriwina- Goodenough 7 (0) 7/7 7/7 5/7 6/7 Morobe Bulolo 4 (3) 0/4 0/4 1/4 0/4 Menyamya 4 (1) 1/4 1/4 1/4 1/4 West New Kandrian- Britain Gloucester 13 (2) 2/13 1/13 0/13 0/13 Talasea 14 (3) 3/14 4/14 4/14 5/14 *Facilities excluded from analysis due to lack of data for health facility catchment population. Results for Western Highlands not presented due to a high number of facilities without catchment population data.

Key informant interviews

All Provincial Health Advisers or Provincial Health Authority Chief Executive Officers

(interviewees) from provinces covered by the Project participated in the interviews. The interviewees (n=8) had been working in health for 12-43 years with a majority of that time working in the province in PNG where they were currently employed and represented each of the eight provinces. The responses presented here relate to discussion of all indicators presented: outpatients per person per year, outreach clinics per 1000 children less than 5 years of age, measles immunisation coverage at 9-11 months, family planning couple years protection per 1000 women of reproductive age, proportion of pregnant women who at least one antenatal care visit and proportion of pregnant women had a supervised delivery.

102

Chapter 4: Contextual factors and health service performance

Geography

Difficult terrain for travel, either mountainous regions or locations only accessible by sea, was a reason given by many interviewees for poor performance of health facilities

(Table 3). The negative impacts cited from challenging geography include: people presenting to the health facility only if they are seriously ill: restricting health workers’ ability to conduct outreach in the catchment communities of their facilities; limiting support and distribution of essential supplies from the district and provincial level to remote facilities; and increased costs for providing health services in remote locations, particularly if air travel was required. By contrast, high performance was attributed to the health facility location being accessible by road and in urban areas where people frequently travel to for other purposes.

103

Chapter 4: Contextual factors and health service performance

Table 3. Quotes from key informant interviews demonstrating themes

Geography On factors that affect access, utilisation and quality of health services:: For an ill person, for their health - if they are sick, they wouldn't think of walking and feeling sick, to travel from one point to the other. So these are some of the challenges or difficulties in terms of geographical settings in our province. On differences in performance within a district: [Health Facility A] is 10 minutes off the main highway. That's why you can see that it's all - it's achieved all its purposes whereas everyone in [location of Health Facility B] is way out there; it so difficult. Cultural and religious beliefs On low supervised deliveries coverage: Delivering and getting delivered by another person that is a non-immediate relative to you by an unknown person and sometimes when they come and they find that there's a male who's going to help to deliver. So all those cultural issues tend to affect. If you have older women who has helped deliver in a village you'll probably find that easier to go to her to get her to attend to your delivery then. On low family planning coverage: [Community 1] and [Community 2] I think it really is to do with their beliefs, … that's putting shame that people don't really want to come in for family planning. Human resources On challenges for health service delivery: A lot of our facilities are not fully staffed and so that's a concern to us, that's a big challenge for us. Even if we staff them, we have to train them to make sure that they are good and effective. Financing On factors that affect access, utilisation and quality of health services: So for example, I need 1000 Kina to go out for the month of January, but the cash comes to you, it's only about 100 or less. What do you do for that? So you have to then prioritise. So these are situations that do affect the service delivery. On factors that affect access, utilisation and quality of health services: Free health care funding was depleted almost within two months….So because of that, there's no longer any funds for their up keep and operations. So, they have gone back, more than six months ago, to start charging again patients. Health information On reasons for high performance in indicators: So we have tried to work out that trend to see what is our plan target based on the performance that we had, and compared with the actual achievements. …. Now I'll give you an example for a facility... It has almost zero - or less than maybe 10 per cent supervised delivery, and since we introduced the incentive [baby bundles and a rest house for expectant mothers], that has actually gone up by almost 100 per cent. That's because of the incentives.

Cultural and religious beliefs

When asked if cultural beliefs affect access and utilisation of health services, interviewees often cited the beliefs in sorcery, the preference of women to deliver

104

Chapter 4: Contextual factors and health service performance

babies in the village and religious beliefs around the use of family planning. For example, in some communities people believe that illness is caused by sorcery rather than believing in traditional Western beliefs about causation which influences their health seeking behaviour, including late presentation to health facilities. There were several cultural issues that were attributed to women preferring to deliver babies in the village noted by two of the interviewees. Specifically, these included the expectation by male health workers that they do not deliver babies and that women prefer to deliver in the village with a known person rather than delivering at a facility with the possibility of a male health worker in attendance. The low uptake of family planning was attributed to the community beliefs around family planning and some church-run health facilities not providing methods of contraception.

Human resources

Human resources was the theme mentioned most frequently for impacting people’s access and utilisation of health services. Having an adequate number and distribution of skilled staff was seen as a vital component for improving health services and access by the community. Half of the interviewees noted that the existing health workers and support staff did not have adequate skills to do their job.

In remote locations, some interviewees said health workers face issues with accommodation and limited support. Health workers in remote locations may frequently travel away from their facility to go to urban areas and when staff are not at their health facility, people will by-pass the facility and go to a higher-level facility such as a hospital. The corollary is that health workers were perceived to be the reason for high performance of some facilities. For two interviewees, the availability of staff was a reason for high performance and they reported that people will seek out these facilities even if they have to travel further. In addition to formal health workers, high performance was attributed to village health volunteers by one respondent, particularly in encouraging women to attend antenatal care and have supervised deliveries. 105

Chapter 4: Contextual factors and health service performance

Financing

The majority of interviewees mentioned funding as a factor contributing to low performance. In particular, delays in funds were mentioned as a factor affecting the delivery or services. One interviewee stated there were sufficient funds but the main issue was with the delay in receiving the funds. Another interviewee also stated there was no issue with insufficient funding but rather the use of the funds available and the capacity of staff were more pertinent issues. The experience of the recently introduced free health care policy demonstrates the possible impact of the misuse and poor management of funding with the reintroduction of user-fees to ensure facilities can remain open after funds have been depleted.

Health Information

Several interviewees stated that the NHIS data were not accurate either because of inaccurate reporting from facilities or inaccurate population estimates for calculating percentages and rates. However, regardless of questions around data quality, two interviewees mentioned using data for reviewing performance at the facility level and creating targeted interventions to improve performance on a specific indicator.

Discussion

Provincial health administrators perceive multiple factors from the health system and the context in which health facilities operate affect performance. Health facilities accessible by road in urban areas with competent and/or higher level staff and health services operated by churches or private companies were cited as contributors to high performance. Conversely, inadequate numbers of staff, poorly skilled staff, delays in funding and challenging geography were the major contributors noted for poor performance. We also identified a large variation in performance indicators between and within the Project districts and while a district may meet the national target, a high 106

Chapter 4: Contextual factors and health service performance

proportion of facilities within the district may not, suggesting that coverage of services is not optimal.

Several factors affecting performance identified in this study had been identified previously, specifically: workforce of appropriate number, cadre mix and skills (11); challenging geographical access to health facilities (6, 12, 13); delays in receiving funds for health services (5); and cultural factors, particularly for maternal and reproductive health indicators. (14). However this contextual analysis offered new information on factors affecting performance such as the availability of health workers at a facility and use of data to develop targeted interventions to improve performance.

Further, the contextual analysis aided understanding of quantitative data used for the indicator analysis.

The interviewees noted issues with data quality and district and health facility catchment populations used within the NHIS to calculate coverage and rates. Issues with the NHIS data quality and completeness have been documented (15, 16). The population catchments used for calculating percent coverage were estimates based on projections from the 2000 population census. In this study there were many coverage rates well above 100%, which could be due to inaccurate numerator data from the

NHIS, inaccurate population estimates or people from outside the catchment area may access the facility who would not be included in the population denominator. According to the interviewees, some facilities with coverage rates well above 100% were accessed by people outside the catchment area either due to the location in urban areas or perceived quality of services.

The indicators used in this study are usually only presented at the national, provincial or district level (12, 16, 17). As seen in this study, individual health facility performance is highly variable and reporting at a district level may mask underperformance in a number of facilities. While previous studies have suggested district level benchmarking is important, it is also important to review data at each facility. The Reaching Every 107

Chapter 4: Contextual factors and health service performance

District approach encourages monitoring of data and plan development at the health facility and district level to improve immunisation coverage and the approach has been adapted for prevention of mother-to-child transmission of HIV (18, 19). The Reaching

Every District strategy is supported by the National Department of Health for immunisation (20, 21). However, most districts had multiple facilities well below the target coverage for measles immunisation suggesting that the strategy has not been effectively implemented. Two interviewees did talk of using a strategy similar to

Reaching Every District, whereby they reviewed indicators and developed interventions and observed improvements in indicators.

There are several limitations to this study. As noted by some of the interviewees, the validity of the NHIS data may be questionable not only due to inaccurate reports but also inaccurate populations for coverage and rate calculations. This limitation highlights the benefit using mixed-methods for this evaluation, particularly sequential explanatory mixed methods where qualitative data can provide explanation for the quantitative data

(9). A second limitation is that the key informant interviews provide qualitative information that is the perception of a limited number of people and does not necessarily mean that this is the only view. These interviews form just one perspective of the health system in PNG.

Undertaking a contextual analysis using a sequential explanatory mixed methods design for the baseline of the formative evaluation of the Rural Primary Health Services

Delivery Project proved useful for understanding the context in which the health facilities operate and potential issues with the data for use in the evaluation. Further, analysis of indicators at both district and health facility level enabled a richer understanding of the heterogeneity of performance. Subsequent to this study, the focus of the evaluation methods shifted from predominantly quantitative to predominantly qualitative methods to document the contextual factors that enabled and or prevented

Project progress. Further, analyses of performance in indicators were undertaken at

108

Chapter 4: Contextual factors and health service performance

the district and health facility level rather than just the district level as planned. Such ongoing contextual analyses and facility level analysis of indicators may be useful for health service delivery programs in similar contexts where health information data quality is poor and the context is not adequately documented elsewhere.

Acknowledgements

The authors would like to sincerely thank the health administrators who participated in this study.

Conflicts of interest

The authors declare no conflicts of interest.

Funding

This work was carried out as part of the Formative Evaluation of the Rural Primary

Health Service Delivery Project. The Rural Primary Health Services Delivery Project is a joint initiative of the Government of PNG, the Asian Development Bank, Australian

Department of Foreign Affairs and Trade, OPEC Fund for International Development,

World Health Organization, United Nations Children’s Fund and the Japanese

International Cooperation Agency.

109

Chapter 4: Contextual factors and health service performance

References

1. Asian Development Bank. Project Administration Manual: PNG Rural Primary

Health Service Delivery Project 2011 [cited 2018 27 August]. Available from: https://www.adb.org/sites/default/files/project-document/74336/41509-013-png- pam.pdf.

2. Monitoring the Building Blocks of Health Systems: A Handbook of Indicators and Their Measurement Strategies. Geneva: World Health Organization, 2010.

3. Systems thinking for health systems strengthening. Geneva: World Health

Organization 2009.

4. Bamberger M, Rao V, Woolcock M. Using Mixed Methods in Monitroing and

Evaluation: Experiences from International Development 2010 [cited 2018 16 May].

Available from: https://openknowledge.worldbank.org/handle/10986/3732.

5. World Bank. Below the Glass Floor: Analytical Review of Expenditure by

Provincial Administrators on Rural Health from Health Function Grants & Provincial

Internal Revenue 2013 [cited 2018 5 June]. Available from: https://openknowledge.worldbank.org/bitstream/handle/10986/17567/845210WP0v20B e0x382123B00PUBLIC0pdf.pdf?sequence=1&isAllowed=y.

6. Howes S, Mako AA, Swan A, Walton G, Webster T, Wiltshire C. A lost decade?

Service Delivery and Reforms in Papua New Guinea 2002-2012 2014 [cited 2017 20

October]. Available from: http://devpolicy.org/publications/reports/PEPE/PEPE_A_lost_decade_FULL_REPORT. pdf.

7. Lesley L, O'Connor M. Barriers to Family Planning Services in PNG: Supply and

Demand Report and Recommendations: Australian Doctors International; 2010.

8. Papua New Guinea Health Sector Monitoring and Evaluation Strategic Plan:

National Health Plan (2011-2020) Port Moresby: National Department of Health 2012.

110

Chapter 4: Contextual factors and health service performance

9. Creswell JW, Clarck VLP. Designing and Conducting Mixed Methods Research.

2nd Revised edition ed. Thousand Oaks, United States: SAGE Publications Inc; 2011.

10. Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Research in Psychology 2006;3(2):77-101.

11. The World Bank. PNG Health Workforce Crisis: A Call to Action 2011 [cited

2018 27 August]. Available from: http://documents.worldbank.org/curated/en/216511468332461651/pdf/NonAsciiFileNa me0.pdf.

12. Bauze AE, Tran LN, Nguyen KH, Firth S, Jimenez-Soto E, Dwyer-Lindgren L,

Hodge A, Lopez AD. Equity and geography: the case of child mortality in Papua New

Guinea. PloS One. 2012;7(5):e37861.

13. Gething PW, Johnson FA, Frempong-Ainguah F, Nyarko P, Baschieri A,

Aboagye P, Falkingham J, Matthews Z, Atkinson PM. Geographical access to care at birth in Ghana: a barrier to safe motherhood. BMC Public Health. 2012;12:991.

14. Vallely LM, Homiehombo P, Kelly AM, Vallely A, Homer CS, Whittaker A.

Exploring women's perspectives of access to care during pregnancy and childbirth: a qualitative study from rural Papua New Guinea. Midwifery. 2013;29(10):1222-9.

15. Ashwell HE, Barclay L. Problems measuring community health status at a local level: Papua New Guinea's health information system. Rural and Remote Health.

2010;10(4):1539.

16. National Department of Health. Sector Performance Annual Review:

Assessment of Sector Performance 2010-2014. Port Moresby: Government of Papua

New Guinea; 2015.

17. World Health Organization. Global Health Observatory Data Repository WHO

Western Pacific Region: Papua New Guinea statistics summary (2002 - present) [cited

2014 18 August]. Available from: http://apps.who.int/gho/data/node.country.country-

PNG?lang=en.

111

Chapter 4: Contextual factors and health service performance

18. Kanyuuru L, Kabue M, Ashengo TA, Ruparelia C, Mokaya E, Malonza I. RED for PMTCT: an adaptation of immunization's Reaching Every District approach increases coverage, access, and utilization of PMTCT care in Bondo District, Kenya.

International Journal of Gynaecology and Obstetrics: the official organ of the

International Federation of Gynaecology and Obstetrics. 2015;130 Suppl 2:S68-73.

19. World Health Organization. Microplanning for immunization service delivery using the Reaching Every District (RED) Strategy. Geneva: 2009.

20. Datta SS, Toikilik S, Ropa B, Chidlow G, Lagani W. Pertussis outbreak in

Papua New Guinea: the challenges of response in a remote geo-topographical setting.

Western Pac Surveill Response J. 2012;3(4):3-6.

21. Apeng D, Drorit D, Mabong P, Theo L, Fitzwarryne C, Bunat A. The 'Reach

Every Village' strategy for community-based health improvement interventions in the

Momase Region of Papua New Guinea. Papua and New Guinea Medical Journal.

2010;53(1-2):37-47.

112

CHAPTER 5: CHALLENGES OF HEALTH PROGRAM MONITORING AND EVALUATION IN LOW RESOURCE SETTINGS

113

Chapter 5: Challenges of health program M&E in low resource settings

1. Chapter prelude

In my role at Abt Associates, I was the Monitoring and Evaluation Manager for three large health programs in Papua New Guinea: the North Fly Health Services

Development Program, the CMCA Middle and South Fly Health Program and the

Formative Evaluation of the Rural Primary Health Services Delivery Project. During my

Doctor of Public Health journey, I reflected on the challenges of conducting monitoring and evaluation in a context such as Papua New Guinea, and the lessons I had learned.

This chapter documents the challenges and lessons, with a focus on the CMCA Middle and South Fly Health Program.

2. Background

The CMCA Middle and South Fly Health Program commenced in July 2013. Rigorous

M&E was a priority for the program from the design stage, ensuring that program progress and outcomes would be measurable, transparent and accountable to beneficiaries, program partners, and the managing organisation: Ok Tedi Development

Foundation. The program design documentation provided the principles for program

M&E, including a program theory of change and M&E framework. However, the specifics of how the M&E would be implemented had to be developed.

I worked with the Program Health Information Officer, based within the Program Team in Papua New Guinea, to develop the Program M&E System to ensure the right data were collected from program commencement and could be easily analysed for regular reporting. A priority of the Program M&E System was to demonstrate equity in terms of program inputs across and within the five CMCA Regions that were contributing funding for the Program. This required the introduction of a mapping component to the

M&E System, to enable easy visualisation of program inputs by location.

114

Chapter 5: Challenges of health program M&E in low resource settings

This was the first time I had to set up an M&E system for a large program. I found that there was ample guidance documentation for conducting good M&E for health programs and more broadly for development programs. However, the literature rarely described the processes in sufficient detail and the reality of M&E in the low resource settings poses many practical challenges. This chapter contributes to the M&E literature by describing the M&E system for the CMCA Middle and South Fly Health

Program and the lessons learned.

3. Project overview

During my first Annual Performance Review for the Doctor of Public Health, I was encouraged to document my learnings from my work on M&E. I subsequently presented my learnings at the Population Health Congress in 2015 and went on to develop the learnings into a manuscript for publication.

4. Contribution to the project

I was responsible for establishing and managing the M&E system for the CMCA Middle and South Fly Health Program. I led the development of the manuscript on lessons learned which forms this chapter, in collaboration with the co-authors.

5. Project outputs

Journal manuscript

Emma Field, Mafu Vila, Laina Runk, Fiona Mactaggart, Alexander Rosewell and Sally

Nathan. Lessons for Health Program Monitoring and Evaluation in a low resource setting. Rural and Remote Health, [accepted 7 June 2018]. 115

Chapter 5: Challenges of health program M&E in low resource settings

Declaration

I certify that this manuscript was a direct result of my research towards this DrPH, and that reproduction in this thesis does not breach copyright regulations.

......

Emma Field

Conference Presentation

Field, EJ; Vila, M; Dove, G and Kewa k. Lessons from the field: Monitoring and

Evaluation of Health Programs in Papua New Guinea. Population Health Congress,

Hobart, September 2015.

6. Ethical considerations

Ethical approval was not required for this work as it was conducted as part of routine monitoring and evaluation of the CMCA Middle and South Fly Health Program. The manuscript does not present any data from individuals.

116

Chapter 5: Challenges of health program M&E in low resource settings

7. Manuscript

Title: Lessons for Health Program Monitoring and Evaluation in a low resource setting

Authors: Emma Field1,2,3, Mafu Vila1, Laina Runk1, Fiona Mactaggart1, Alexander

Rosewell2 and Sally Nathan2

Authors affiliations:

1. Abt Associates

2. School of Public Health and Community Medicine, UNSW Australia

3. Menzies School of Health Research, Charles Darwin University, Brisbane,

Australia

Abstract

There are numerous guidelines that outline best practices for health program monitoring and evaluation (M&E). However health programs are often implemented in less than ideal circumstances where these best practices may not be resourced or feasible. This paper describes how M&E has been conducted for a health service delivery improvement program in remote Papua New Guinea and outlines lessons learned. The lessons were: integrate M&E into every aspect of the program; strengthen existing health information data; link primary data collection with existing program activities; conduct regular monitoring and feedback for early identification of implementation issues; involve the program team in evaluation; and communicate M&E data through multiple mediums to stakeholders. These lessons could be applied to other health programs implemented in low resource settings.

Key words

Monitoring and evaluation, low resource setting

117

Chapter 5: Challenges of health program M&E in low resource settings

Introduction

Monitoring and evaluation (M&E) of health programs in low resource settings can be challenging for multiple reasons: limited human resource capacity; weak information systems; inadequate financial and human resources, and limited demand for M&E (1).

However increasingly rigorous M&E is required by governments and donors to transparently report if programs are implemented as planned and achieving the expected outcomes. This paper describes M&E conducted for the Community Mine

Continuation Agreement (CMCA) Middle and South Fly Health Program (the Program) in Papua New Guinea and offers practical solutions as lessons learned from the experiences in this context.

CMCA Middle and South Fly Health Program

The Program is a comprehensive health program that aims to improve health service delivery to remote communities in the Western Province of Papua New Guinea (2). The program coordinates support through a partnership with existing health service providers covering all aspects of health service delivery and primary health care

(Figure 1). Full details of the program are available in an online report of the midline evaluation (3).

The Program activities are implemented by a multi-disciplinary team of about 20 staff, in collaboration with existing health service providers in the program area. There are 22 health facilities supported by the Program covering 50,000 people. The geography in the program area is challenging with transport to villages and health facilities often by boat.

Description of the M&E System

The Program design was based on a program logic (Figure 1). The program design also outlined the following principles for the M&E System: appropriate use of existing

118

Chapter 5: Challenges of health program M&E in low resource settings

data; where available, use the National targets or set realistic targets; focus on the needs of users and encourage use of data; and ensure M&E is integrated into implementation and is not a separate activity. Each year, annual activity plans were developed based on the program logic.

Figure 1: Simplified Program Logic for the CMCA Middle and South Fly Health

Program

The program design also outlined guiding principles for indicator selection: use the minimum number of indicators to track performance as each additional indicator requires additional resources for collection and analysis; the indicators should link to

119

Chapter 5: Challenges of health program M&E in low resource settings

inputs, processes, outputs, outcomes and impacts in the program logic and annual activity plan; and wherever possible use existing data sources for indicators.

To enable regular progress monitoring and reporting of indicators, an M&E system was established, which incorporated both primary and secondary data (Figure 2).

Secondary data was from the National Health Information System (NHIS), a monthly paper-based information system where aggregate health care presentations and health care services provided (e.g. immunisations, antenatal care) are reported monthly. The

NHIS data were used to calculate long-term outcomes (e.g. immunisation coverage).

The program team would use existing NHIS forms to record immunisations given (or any other activity recorded in the NHIS) and provide a copy for the program M&E database and for reporting through the existing NHIS processes. This enabled a direct attribution analysis of the program to the overall indicators (e.g. immunisation coverage) in the program area. Where no data collection for other program indicators existed, program specific M&E forms were developed based on the annual activity plan. These M&E forms were filled in by designated staff on a monthly basis (e.g. the

Infrastructure Officer reported on all infrastructure related activities in the annual activity plan) and entered into the M&E database. The data from the M&E system were used for monthly progress reporting, annual activity planning and periodic evaluation.

120

Chapter 5: Challenges of health program M&E in low resource settings

Figure 2. CMCA Middle and South Fly Health Program M&E System

Lessons learned

Lesson 1: Integrate M&E into every aspect of the program

While it is best practice to incorporate an M&E plan into the program design, these plans need to be sufficiently detailed and feasible to enable M&E to commence with program commencement. However, it is not unusual that M&E plans take substantial time to finalise while program implementation has already started (4). Furthermore,

M&E is often seen as the responsibility of the M&E officer or team, and not of the entire program team.

For the Program, a detailed M&E plan was developed in the program design and was integrated across all program operations. The M&E plan included a program logic, an 121

Chapter 5: Challenges of health program M&E in low resource settings

M&E framework (detailing indicators and their link to the program logic, source of data, and frequency of reporting), and a reporting framework. There was regular monitoring of program progress and a baseline, midline and endline evaluation. M&E was funded in the budget with specific personnel, a part-time M&E Manager, a part-time Data

Manager and a full-time Health Information Officer. A lesson from the Program that enhances M&E best practice knowledge is that M&E specific activities, such as monitoring, were integrated into each team member’s terms of reference. This ensured that M&E was integrated into daily activities and annual performance reviews. Overall this integration of M&E resulted in adequate resourcing of M&E activities, M&E was initiated at the same time as the Program commencement and M&E was not viewed as an activity separate to a staff members duties but rather a core responsibility from management to field staff.

Lesson 2: Strengthen existing health information data

Using secondary data from existing information systems is most cost-effective for M&E.

However health information systems in low resource settings may have issues with timeliness, completeness and accuracy. In Papua New Guinea there is an acknowledgement these issues exist with the NHIS (5). Health information systems are one of the six building blocks of health systems as outlined by the World Health

Organization, forming a critical function in ensuring ‘the production, analysis, dissemination and use of reliable and timely information on health determinants, health system performance and health status’ (6). As with all the health system building blocks, health information systems are not stand alone systems and the ability to generate complete and accurate data has ramifications for all the building blocks (7).

Programs that aim to support or strengthen one or more building blocks of the health system should also invest in strengthening the health information system, although in practice this does not always occur (8).

122

Chapter 5: Challenges of health program M&E in low resource settings

An element of support for strengthening the health information system in Western

Province was integrated into the program design, in the form of a Health Information

Officer. This officer worked with their government counterpart in the Provincial Health

Office, to improve timeliness, completeness and accuracy of the NHIS. This support led to an improvement in both the completeness and quality of the data (completeness went from 91% in 2012 to 99% in 2015). Furthermore, the review of the NHIS data led to the detection of substantial underreporting which was corrected.

Lesson 3: Link primary data collection with existing program activities

While using existing data is preferred, health information systems in developing countries do not always collect the required information for program M&E. However travel to program sites, often in remote locations, for primary data collection adds enormous costs for transport and personnel time. In low resource settings, travel costs can use up scarce funds that would be better used for implementation of activities. The

Program staff regularly travel to program sites to implement activities. This provided an opportunity to incorporate program implementation with primary data collection thereby reducing costs.

The Program was launched with an outreach clinic provided to every village in the program area. This was a huge logistical undertaking, given the remote location of many villages. It was, however, a prime opportunity for primary data collection for baseline evaluation. The Program outreach clinic team were trained in data collection and were able to carry out health facility assessments, interviews with health workers and focus group discussions with community members, along with their outreach clinics. Baseline data collected by the outreach clinic team was extremely valuable in informing specific activities for the first annual activity plan. Additionally, involvement of the program team in the baseline built their capacity for M&E, and provided them with a deeper understanding of the health services and community expectations.

123

Chapter 5: Challenges of health program M&E in low resource settings

Lesson 4: Regular monitoring and feedback is vital for early identification of issues

Process monitoring is the foundation for M&E. Process monitoring enables evaluators to distinguish between a failure in program design or a failure in implementation (9). If what has been done is not sufficiently recorded, it is very difficult to evaluate outcomes and impact of programs. For example, poor outcomes may be attributed to the program when implementation was actually insufficient. Furthermore, for transparency, it is important to communicate to donors and beneficiaries about what the program has done.

For the Program, process monitoring was integrated into all program staff reporting requirements which were linked to the annual activity plan. Each month, progress of the annual activity plan was discussed with the team, identifying where activities were on track or if there were delays. This was critical in terms of timely implementation.

When delayed activities were identified in these monthly meetings a discussion followed as to what could be done to overcome the delay. Often this led to additional resources being allocated and closer monitoring to ensure the activity was completed.

Changes to activities were documented during these discussions. The documentation resulting from these discussions was also used to inform the client and stakeholders about status of program activities and the issues surrounding delayed activities.

Lesson 5: Involve the program team in evaluation

The program team were actively involved not only in monitoring, but also evaluation data collection for the program. Periodic evaluations serve to assess whether the longer-term outcomes and impacts from the program were being achieved and informed alterations to implementation.

There are two options for who conducts the evaluation, the organisation implementing the program (internal evaluators), or an external organisation or consultants (external 124

Chapter 5: Challenges of health program M&E in low resource settings

evaluators). In a review of case studies of influential evaluations of development programs the use of external consultants was seen as being more independent with the ability to explore sensitive political issues (10). On the other hand involving an internal evaluator may provide better access to data and key stakeholders and there are opportunities for fostering program ownership and learning through team involvement (10, 11). This was certainly the case for the Program baseline evaluation, where the team’s involvement in focus group discussions with communities, health workers interviews and health facility assessments allowed them to gain a deep insight into the issues for planning and implementing program activities (Lesson 3). A third approach, not used in any of proposed in the literature is a joint evaluation with internal and external evaluators as a way of ensuring independence and contextual knowledge

(10).

For the midline evaluation of the Program, it was no longer appropriate for the program outreach team members to conduct data collection, given their now established relationship with health workers and community members and their role in program implementation. However, a joint evaluation approach was used with a combination of independent evaluators and the program M&E team, who did not have contact with the health workers or communities.

The role of the independent evaluators was specifically to seek the perspectives of health workers, communities and program partners on changes since program commencement and future directions. The program M&E team designed the overall midline evaluation methodology and data collection tools, based on the evaluation questions developed through a meeting with program stakeholders. Independent evaluators conducted key informant interviews and focus group discussions with program team members, program partners, health workers and community members.

The M&E team collated and analysed quantitative data from the NHIS. The results from the qualitative and quantitative data were synthesised into a report by both the program 125

Chapter 5: Challenges of health program M&E in low resource settings

M&E team and independent evaluators. This combination of internal and external evaluators provided advantages of in-depth knowledge and context of the Program from the program M&E team and the independent evaluators ensured evaluation participants felt comfortable raising concerns about the program and contributed to the transparency of the evaluation findings.

Lesson 6: Communicate M&E data through multiple mediums

There were many audiences for which the results of the program M&E needed to be communicated. The team communicated results in multiple formats: monthly reports to the program team, which served to inform and improve program implementation in real time; quarterly reports to the donor and program partners; quarterly feedback posters and information sessions via the outreach clinic team to beneficiaries, the communities and health workers; and the program website for the wider public.

A key component of the program was equitable distribution of program benefits to the health facilities and communities, which was difficult to present in standard reporting templates (e.g. tables/graphs). Demonstrating this distribution was achieved through mapping program activities by village and health facilities. Data visualisation software,

Tableau, was used for maps which were embedded in the program website. These maps were interactive, allowing users to map different indicators and compare results from baseline and current status. Furthermore, this ensured that stakeholders could access non-confidential program data without having to request data from the program

M&E team or wait for routine reports (Figure 3).

126

Chapter 5: Challenges of health program M&E in low resource settings

Figure 3. Example of data from the CMCA Middle and South Fly Health Program

M&E system displayed on the program website (12)

Data on the health facilities previously were neither available to health service providers in this level or detail nor regularly updated prior to the Program. The program sought to increase demand for data for decision-making through the M&E system, strengthening the NHIS, and regular presentation of data analyses at the program partnership meetings. However, use of data by program partners for decision-making, e.g. annual activity planning, remains limited. Enhancing the utility of information products generated from the M&E system, through seeking feedback from users, may improve data use for decision-making (13).

127

Chapter 5: Challenges of health program M&E in low resource settings

Conclusion

In this paper we have outlined lessons from M&E for the CMCA Middle and South Fly

Health Program in Papua New Guinea. Firstly, integrating M&E into all aspects of the program from program design to implementation assisted in having a solid plan for

M&E, an adequate budget, appropriate human resources and buy-in from the entire team. Furthermore, the program team can contribute to primary data collection while travelling to sites for M&E and improve contextualisation of M&E through participating in joint evaluations with independent evaluators. In the low resource settings, contributing to strengthening of the national health information system, from which the data are often used for M&E indicators, is both beneficial for program M&E and for the national health information system. Regular monitoring and feedback to program team and discussions of M&E data assisted in identifying issues and improved implementation. Finally, we reported results from the M&E system in multiple formats, including using maps, to stakeholders. These lessons may be applicable to health programs in other low resource settings.

128

Chapter 5: Challenges of health program M&E in low resource settings

References

1. Kusek JZ, Rist RC. Ten steps to a results-based monitoring and evaluation system : a handbook for development practitioners. Washington: The World Bank;

2004 [cited 2018 16 May]. Available from: https://openknowledge.worldbank.org/handle/10986/14926.

2. CMCA Middle and South Fly Health Program: Abt Associates; 2013 [cited 2018

16 May]. Available from: http://www.cmsfhp.org/about-us/.

3. Abt JTA. Strengthening Health Services in Western Province, Papua New

NFHSDP-CMCA-MSFHP_DFAT-Print_PA-signed.pdf.

4. Mathis J, Senlet P, Topcuoglu E, Kose R, Tsui A. Best Practices in Monitoring and Evaluation: Lessons from the USAID Turkey Population Program2001 [cited 2017

2 June]. Available from: https://www.measureevaluation.org/resources/publications/sr-

01-11.

5. Ashwell HE, Barclay L. Problems measuring community health status at a local level: Papua New Guinea's health information system. Rural and Remote Health.

2010;10(4):1539.

6. World Health Organization. Everybody’s Business: Strengthening Health

Systems to Improve Health Outcomes: WHO’s Framework for Action. Geneva2007

[cited 2018 16 May]. Available from: http://www.who.int/healthsystems/strategy/everybodys_business.pdf.

7. Systems thinking for health system strengthening. Geneva: Alliance for Health

Policy and Systems Research and World Health Organization; 2009 [cited 2018 16

May]. Available from: http://www.who.int/alliance-hpsr/resources/9789241563895/en/.

129

Chapter 5: Challenges of health program M&E in low resource settings

8. Warren AE, Wyss K, Shakarishvili G, Atun R, de Savigny D. Global health initiative investments and health systems strengthening: a content analysis of global fund investments. Globalization and health. 2013;9(1):30.

9. Bamberger M, Rao V, Woolcock M. Using Mixed Methods in Monitroing and

Evaluation: Experiences from International Development 2010 [cited 2018 16 May].

Available from: https://openknowledge.worldbank.org/handle/10986/3732.

10. World Bank. Influential Evaluations: Evaluations that Improved Performance and Impacts of Development Programs. Washington DC2004 [cited 2018 16 May].

Available from: http://documents.worldbank.org/curated/en/609091468329361924/Influential- evaluations-evaluations-that-improved-performance-and-impacts-of-development- programs.

11. Cole DC, Aslanyan G, Dunn A, Boyd A, Bates I. Dilemmas of evaluation: health research capacity initiatives. Bull World Health Organ. 2014;92(12):920-1.

12. Abt JTA. CMCA Middle and South Fly Health Program: Map of key program indicators [cited 2017 15 February ]. Available from: http://www.cmsfhp.org/health- facilities/key-indicators/

13. Geers E, Nghui P, Ekirapa A, Rop V, Mbuyita S, Patrick J, Kusekwa S, Soti D,

Muthami M, Kumalija C. Information Products to Drive Decision Making: How to

Promote the Use of Routine Data Throughout a Health System2017 [cited 2017 2

June]. Available from: https://www.measureevaluation.org/resources/publications/sr-17-

145-en.

130

CHAPTER 6: CONCLUSION AND RECOMMENDATIONS

131

Chapter 6: Conclusion and recommendations

1. Summary of findings

This thesis includes several publications from the M&E activities of health system support and strengthening in Papua New Guinea that I have undertaken as part of my role as the Monitoring and Evaluation Manager at Abt Associates. In this concluding chapter, I summarize the contribution of this work and the key findings from chapters presented in the thesis, to the knowledge of M&E of health system support and strengthening in Papua New Guinea, how it may relate to similar contexts, and identify possibilities for more rigorous M&E in Papua New Guinea in the future.

Chapter 2: Evaluation of the CMCA Middle and South Fly Health Program

This chapter, including the publication and report, describes the midline evaluation undertaken for the CMCA Middle and South Fly Health Program. The purpose of a midline evaluation is to determine progress to date and identify how implementation can be improved for the remainder of the program. The results of such evaluations are most relevant to the program donors, implementers and stakeholders. More broadly, the lessons from such evaluations can be applied in similar remote, low resource settings.

The CMCA Middle and South Fly Health Program is delivered through a partnership for health service delivery. There were limited peer-reviewed publications about this type of model for improving service delivery. Existing literature from low resource settings had focused on the contracting of private or non-government organisations to provide an agreed level of services (1-4). Evidence suggests that contracting private or non- government organizations can be beneficial in environments where governments are not able to provide services, such as in post-conflict settings and fragile states (3, 5-7).

While it is debatable whether Papua New Guinea is a fragile state, the existing health services providers (government and church) have largely not been able to provide services according to the National Health Service Standards. The midline evaluation of

132

Chapter 6: Conclusion and recommendations

the CMCA Middle and South Fly Health Program was an opportunity to report in the peer-reviewed literature about a novel partnership used for the program and the outcomes in the first two years of implementation.

The CMCA Middle and South Fly Health Program differed to other contracting arrangements in that a private organisation was contracted to work with the existing health service providers to improve service delivery. These existing providers were actively engaged in the program implementation through a partnership, and participated in the development of annual work plans for the CMCA Middle and South

Fly Health Program and in Program implementation with the private organisation. The

Program work plans were aligned with the health service delivery work plans and addressed gaps in resources.

In the introduction I described the difference between health system support and health system strengthening (8). Operating at the sub-province level, the CMCA Middle and

South Fly Health Program largely provided health system support. The model used in this Program has demonstrated that health system support activities can rapidly improve service delivery. Ultimately, when the Program is completed, it is the existing health service providers who will continue providing health services in Western

Province. It is too early to determine if the model used for the CMCA Middle and South

Fly Health Program is better for capacity building and sustainability than direct contracting (1-4). The current nationwide health system constraints, such as the shortage of health workers and medical supplies stock outs, will likely remain after program completion posing challenges to long-term sustainability. To maintain or improve the current level of service delivery it is likely that some level of ongoing external assistance after program completion will be required until these health system constraints are adequately addressed.

133

Chapter 6: Conclusion and recommendations

The CMCA Middle and South Fly Health Program midline evaluation employed a multi- level parallel mixed methods approach. The evaluation, fully described in the report in the Appendix of this thesis, included qualitative methods (key informant interviews, focus group discussions, and the 10 seeds methods) and quantitative methods (before- after analyses of indicators and an assessment of equipment at the health facilities) and aimed to understand the perspective of the program from community members, health workers, health service providers and other stakeholders. This multi-level parallel mixed methods approach, where data for different sources and methodologies for different levels of the program were triangulated to synthesise findings, has been used in complex health program evaluations in other contexts (9, 10).

The quantitative methods used in the evaluation employed a before-after design. This design does not use a control, although we compared the program area to the national level quantitative indicators. A more rigorous study design such as an RCT was not possible. Progressive roll out of an RCT, whereby units of investigation, e.g. health facilities or communities, would progressively receive the interventions, with the period without the intervention serving as the control would have been a more rigorous evaluation methodology (11). However, as the program was funded by the community, and there was both an expectation and an urgent need to rapidly improve access and quality of health services, this approach would not have been acceptable or ethical. I had investigated the possibility of using a quasi-experimental design with proximity score matching, where control health facilities are retrospectively identified based on similar characteristics to the intervention facilities (12), However, the trends in potential control facilities in the province prior to program implementation were too dissimilar to the interventional facilities – meaning they would not be suitable controls. Further, adding control facilities and communities to the primary data collection (including the qualitative data collection and health facility assessments) would have increased the cost of the evaluation by up to 40-50% (13).

134

Chapter 6: Conclusion and recommendations

In this evaluation, mixed methods were particularly useful to explain possible anomalous data from health facilities. For example, a surprising result was the decline in supervised deliveries over the program period. This was in contrast to qualitative data suggesting that many more women were presenting to health facilities for a supervised delivery or having a supervised delivery in the village by the health worker or a VHV. While this discrepancy could not be clarified with the data available, it identified supervised deliveries as an area for attention and the value of mixed methods in ensuring more rigorous evaluation in a context like Papua New Guinea where data quality from health information systems may be unreliable.

Chapter 3: Lessons from the Village Health Volunteer Program and

Considerations for Policy

Lay health workers, often referred to as community health workers, are a cadre of health workers who have received brief training and work in their communities with or without remuneration (14). However in Papua new Guinea, community health workers are paid health workers in the formal health system with a two-year qualification and form the largest proportion of the health workforce (15). Lay health workers have been operating in Papua New Guinea for decades and are more recently referred to as

Village Health Volunteers (16). Training and sustaining a cadre of lay health workers is an important health system strengthening strategy for improving access and coverage of health services and positively influencing risk behaviours in low resource settings

(14).

In this chapter, I employed an exploratory qualitative analysis to describe the perceptions of the Village Health Volunteer Program implemented in the Middle and

South Fly Districts of Western province, Papua New Guinea and discussed the lessons for policy. The only primary data available were qualitative and were collected as part of the larger midline evaluation for the CMCA Middle and South Fly Health Program.

135

Chapter 6: Conclusion and recommendations

Due to time constraints and unavailability of some key informants, only a limited number of interviews had been carried out with VHVs (n=7), VHV Trainers (n=3), health workers (n=8) and male and female focus group discussions which had been carried out in four villages. For the health worker interviews and community focus group discussions, the VHV program was only a focus of a couple of questions as part of midline evaluation. To bolster these data, and increase the usefulness to policy makers, an extensive review of literature on lay health worker programs was undertaken. In this chapter the qualitative data was used to illustrate the key issues that such lay health worker programs may face and identify considerations for policy.

The central themes from the qualitative data highlighted the conflict between designated duties of a VHV and actual duties reported by VHVs, health workers and community members; community support for VHVs and participation in VHV activities; incentives and remuneration; and VHV support from to the health system. There was a plethora of evidence from global literature that these issues occur in other lay health worker programs. I used this literature together with the available data from the evaluation to propose considerations for VHV policy in Papua New Guinea.

The VHV Program was just one of many interventions of the CMCA Middle and South

Fly Health Program. A specific M&E plan for the VHV Program was not developed within the CMCA Middle and South Fly Health Program M&E plan. One of the policy considerations identified in the manuscript was the need for better monitoring and evaluation of the VHV programs in Papua New Guinea. Currently there is no M&E framework or data collection for the VHV program nationally or in Western Province. A

VHV program logic or theory of change would have been beneficial to clearly articulate the inputs, outputs and expected outcomes of the VHV program. Such a program logic or theory of change would have informed the evaluation design and methodology.

Further the addition of quantitative data collection to describe what the VHVs are doing in this study would have allowed for measuring the VHVs’ contribution to health 136

Chapter 6: Conclusion and recommendations

indicators. If a rigorous evaluation of the VHV Program is required in the future, it is recommended that a program logic or theory of change is developed and quantitative data on the VHV activities be collected in addition to qualitative data.

Chapter 4: Contextual factors and health service performance in Papua New

Guinea

The framework for monitoring and evaluation of health system strengthening outlines largely quantitative indicators (17). However Bamberger (13) placed an emphasis on the contextual analysis of economic, political, institutional, environmental and socio cultural factors that may influence each step of a project (design, inputs, processes outputs, outcomes, impacts and sustainability). In Chapter 4 I described work carried out at the commencement of the Rural Primary Health Services Delivery Project for the

Formative Evaluation. The Project interventions covered both health system support and strengthening, with a focus on the latter. Interventions included the review or development of national policies and standards; supporting the establishment of partnerships at provincial level with state and non-state partners in health; development of health worker training curriculum and conducting health worker training; infrastructure development through construction of two Community Health Posts in each district; and community level health promotion activities to improve demand for services. The Project involves two districts each in eight provinces in Papua New

Guinea, with the intention that successful interventions be rolled-out nationally. These provinces varied considerably, thereby making geographical, social and cultural contextual factors potential mediators for Project implementation and impact.

A contextual analysis was undertaken to identify potential mediators for Project impact through a sequential explanatory mixed methods design. This study design involved a quantitative analysis of indicators at the health facility level that was subsequently used to guide key informant interviews with provincial health administrators on factors

137

Chapter 6: Conclusion and recommendations

affecting facility performance. A major finding from this chapter was that there were large variations in performance between districts but also within districts. Reporting on performance at the district, province or national level can mask performance at local levels. If there are facilities performing well in a challenging context, there may be lessons for how success can be translated to other facilities. The factors that contribute to success identified through the interviews with key informants in this chapter were generally modifiable: the number of staff and the quality of service they provide, health service provider (e.g. church or private provider), access to funding and use of data to improve service delivery. The main non-modifiable factor identified was the location of the facility – urban facilities that were easy to access were cited as a reason for better performance than facilities in rural and remote locations. Social and cultural factors also influenced reproductive and maternal health indicators.

For monitoring and evaluation practitioners, understanding the context of the program area at baseline is valuable for measuring progress at midline and endline.

Interventions that are implemented in a variety of settings may achieve very different outcomes due to contextual differences. A contextual analysis is very useful for M&E purposes as it helps explain why a program may be achieving different outcomes in different settings. In this Chapter, the contextual analyses occurred not at midline or endline evaluation but at baseline. Establishing the contextual factors that affect performance of health facilities before Project implementation provided a solid foundation for informing future M&E activities. Such contextual analysis should be conducted at regular intervals throughout project implementation.

Chapter 5: Challenges of health program monitoring and evaluation in low resource setting

While there are guidelines that describe best practices for monitoring and evaluation, there are still many programs where these best practices are not followed. Programs

138

Chapter 6: Conclusion and recommendations

may be designed without a theory of change or program logic. Implementation of programs may commence without a clear, adequately resourced M&E plan. Programs may collect inadequate data to inform M&E. This reflects the reality of program implementation where time and resources are often limited and there are pressures to commence programs before M&E has been established.

There were multiple challenges in conducting the monitoring and evaluation for the

CMCA Middle and South Fly Health Program. The Program operated in a remote area in Papua New Guinea. Transportation to the communities for monitoring and evaluation purposes was a major challenge. Usually team members would need to fly to a nearby urban area and then travel for hours or days by boat to get to communities. There were often delays or cancellation of flights for field staff undertaking data collection. Further, in 2015, when the midline evaluation was carried out, there was a drought affecting many parts of Papua New Guinea, including Western Province. The drought contributed to a worsening of access to some communities as water levels in the river were low and boats could not travel. Additionally, during the midline evaluation of the

CMCA Middle and South Fly Health Program, travel to several communities was cancelled due to conflict in the community and safety concerns for the team.

Communication with team members also proved challenging. My role was based in

Brisbane but required frequent contact with the team in Papua New Guinea. While mobile phone network coverage has improved considerably in Papua New Guinea, there are still communities with poor or no access. Even in urban areas, phone calls can frequently be of poor quality or drop out for extended periods of time.

When data collection was done, there was usually only one opportunity to collect data at each health facility or community due to time and budget constraints. If there were errors or missing data, it was often very difficult to go back to the source for verification.

While training was provided prior to teams going out for data collection, this issue was

139

Chapter 6: Conclusion and recommendations

not reduced to zero. Further for qualitative data collection this meant that only key informants and community members available at the time could participate in the M&E activities. Overall this was not a significant issue, with the exception of Chapter 3 on the VHV Program. Only a small number of interviews and focus group discussions were undertaken, limited the depth of information on the experience of the VHV

Program.

The M&E activities described in this thesis were carried out by M&E staff, other program staff or short-term staff employed specifically for the collection of M&E data.

There was a wide variation in the experience of the staff collecting the data. Some staff had a background in health research but the majority were health workers. As with many specialised areas within health, there is a lack of suitably skilled M&E specialists in Papua New Guinea. Training staff in M&E was an integral part of my role. For me, the opportunity to work with staff to train them in M&E was one of the most enjoyable aspects of my job. In particular the CMCA Middle and South Fly Health Team became very interested in the M&E data and were keen to see their contribution to the improvement in health indicators.

Despite these challenges, the dedicated Program team and short-term evaluation consultants were able to collect the necessary data for M&E that was fit for the purposes of the Program. While there were still issues with data quality, we saw this as a process for learning and improvement for the program team and health service providers.

Chapter 5 described the M&E system for the CMCA Middle and South Fly Health

Program, detailing how the M&E was carried out and the potential lessons from health programs implemented in similar contexts. The lessons were:

140

Chapter 6: Conclusion and recommendations

 Integrating M&E into all aspects of the program from program design to

implementation assisted in having a solid plan for M&E, an adequate budget,

appropriate human resources and buy-in from the entire team.

 The program team can contribute to primary data collection while travelling to

sites for M&E and improve contextualisation of M&E through participating in

joint evaluations with independent evaluators.

 In the developing context, contributing to strengthening of the national health

information system, from which the data are often used for M&E indicators, is

both beneficial for program M&E and for the quality of national health

information system.

 Regular monitoring and feedback to program team and discussions of M&E

data assisted in identifying issues and improved implementation.

 Results from the M&E system need to be reported in multiple formats to

stakeholders, for example reports, website, and maps.

2. Recommendations

Throughout the writing of this thesis I have reflected on what could be done to improve

M&E of health system support and strengthening programs in Papua New Guinea which may be relevant to other low resource contexts.

Not all health programs are designed based on a program logic or theory of change. As learned through Chapter 5, for the CMCA Middle and South Fly health Program, the program logic formed the basis of indicator selection and the M&E framework.

Additionally, the M&E for the CMCA Middle and South Fly Health Program was

141

Chapter 6: Conclusion and recommendations

designed along with the overall program design and was sufficiently resourced. This approach formed a solid foundation for M&E.

The M&E for the CMCA Middle and South Fly Health Program contained detailed monitoring, or process evaluation, which helped inform program implementation. Much of the focus of M&E for health system strengthening is on longer-term outcomes and impacts, i.e. improvements in health services and health status. Given that implementation of policy has been a critical issue in the health sector in Papua New

Guinea (18) it would be beneficial to focus M&E efforts on inputs, processes and short- term outcomes. If program implementers cannot fully describe what was implemented, it is impossible to know if a lack of achievement in outcomes and impact is due to the ineffectiveness of the interventions or poor implementation (19). Further to this point, as learned through Chapter 4, the context in which a program is implemented can greatly influence performance and needs to be considered from the outset and during program implementation.

Papua New Guinea has a history of poor record keeping which creates a barrier for effective M&E. Chapter 2 highlighted the issues with the National Health Information

System data. There are now efforts to improve the low coverage of civil registration and vital statistics (20). Another recent development that may aid the improvement of M&E of health programs in Papua New Guinea is the roll out of a tablet-based reporting platform for the National Health Information System, replacing the paper-based reporting system. Early reports for the malaria data from this new platform suggests improvements in both the quality and timeliness of data (21).

Finally, as with the overall shortage of health workers in Papua New Guinea, there is also a shortage of people with suitable skills in M&E. Those who do work in M&E often come from a clinical or technical background without specific training in M&E. Health

142

Chapter 6: Conclusion and recommendations

programs need to be mindful of the deficit in M&E capacity and plan for building of capacity of their team accordingly.

In summary, the recommendations for improving health system support and strengthening program M&E in Papua New Guinea are:

1. Use a program logic or theory of change for program design as a framework for

program M&E.

2. Ensure M&E is considered at the design phase of any program and is

adequately resourced.

3. Ensure that program implementation is sufficiently monitored, i.e. the inputs,

processes and outputs.

4. Document contextual factors that may influence the program from the design

through to completion to inform data interpretation and implementation.

5. Strengthen information systems to improve the timeliness, completeness and

quality of data used for M&E.

6. Build capacity of the workforce within Papua New Guinea to conduct M&E using

both quantitative and qualitative methods.

3. Conclusion

M&E forms the backbone of program implementation, informing if the program has been implemented as planned and is achieving the stated outcomes. M&E fits within the realm of applied research, where the work is carried out in the real world and therefore often faces logistical and methodological challenges. Nonetheless, there are opportunities to improve M&E of health system support and strengthening in Papua

New Guinea. As the National Health Plan 2011-2020 had a “back to Basics” theme, 143

Chapter 6: Conclusion and recommendations

this approach can also be applied to M&E where the core foundations of M&E are addressed (use of program logic/theory of change, adequate resourcing, contextual analyses, improve data quality and human resource capacity for M&E) and the focus is on determining how well policy is being implemented and what work for whom and in which context.

Overall the Doctor of Public Health has provided me with an opportunity to undertake in-depth review of the literature relevant to my work and detailed analyses of my projects, which is often difficult to do in the fast-paced private sector. I now have a much broader understanding of my work and the methods available for monitoring and evaluation. The emphasis on publication through the Doctor of Public Health meant that work that would only be available in reports has been published in the peer review literature. I hope this contributes to the understanding of what works in Papua New

Guinea.

144

Chapter 6: Conclusion and recommendations

4. References

1. Hushie M. Public-non-governmental organisation partnerships for health: an exploratory study with case studies from recent Ghanaian experience. BMC Public

Health. 2016;16:963.

2. Chirwa ML, Kazanga I, Faedo G, Thomas S. Promoting universal financial protection: contracting faith-based health facilities to expand access--lessons learned from Malawi. Health Research Policy and Systems / BioMed Central. 2013;11:27.

3. Newbrander W, Ickx P, Feroz F, Stanekzai H. Afghanistan's Basic Package of

Health Services: Its development and effects on rebuilding the health system. Global

Public Health. 2014;9(sup1):S6-S28.

4. Ravindran TK. Privatisation in reproductive health services in Pakistan: three case studies. Reproductive Health Matters. 2010;18(36):13-24.

5. Newbrander W. Rebuilding Health Systems and Health Services in Fragile

States 2007 [cited 2017 30 December]. Available from: https://www.msh.org/sites/msh.org/files/rebuilding-health-systems-and-providing- health-services-in-fragile-states.pdf.

6. Hatt L, Johns B, Connor C, Meline M, Kukla M, Moat K. Impact of Health

Systems Strengthening on Health Bethesda, M.D.: Health Finance & Governance

Project, Abt Associates 2015 [cited 2017 31 December]. Available from: https://www.hfgproject.org/impact-hss-health/.

7. National Academies of Sciences Engineering and Medicine. The role of public- private partnerships in health system strengthening: Workshop Summary2016.

8. Chee G, Pielemeier N, Lion A, Connor C. Why differentiating between health system support and health system strengthening is needed. The International Journal of Health Planning and Management. 2013;28(1):85-94.

145

Chapter 6: Conclusion and recommendations

9. Bamberger M, Rao V, Woolcock M. Using Mixed Methods in Monitroing and

Evaluation: Experiences from International Development 2010 [cited 2018 16 May].

Available from: https://openknowledge.worldbank.org/handle/10986/3732.

10. Mookherji S, Meck K. How Can We Better Evaluate Complex Global Health

Initiatives? Reflections From the January 2014 Institute of Medicine Workshop. Global

Health, Science and Practice. 2015;3(2):174-9.

11. White H. An introduction to the use of randomised control trials to evaluate development interventions. Journal of Development Effectiveness. 2013;5(1):30-49.

12. White H, Sabarwal S. Quasi-experimental Design and Menthods Florence:

UNICEF Office of Research, ; 2014 [cited 2017 31 December]. Available from: https://www.unicef-irc.org/publications/pdf/brief_8_quasi- experimental%20design_eng.pdf.

13. Bamberger M, Rugh J, Marby L. Real World Evaluation: Working under budget, time, data and political contraints Sage Publications; 2006.

14. Lehmann U, Sanders D. Community health workers: What do we know about them? Geneva: World Health Organization, 2007.

15. The World Bank. PNG Health Workforce Crisis: A Call to Action 2011 [cited

2018 27 August]. Available from: http://documents.worldbank.org/curated/en/216511468332461651/pdf/NonAsciiFileNa me0.pdf.

16. Minimum Standards for Village Health Volunteers in Papua New Guinea. In:

National Department of Health, editor. 2003.

17. World Health Organization. Monitoring and evaluation of health system strengthening: An operational framework Geneva2009 [cited 31 December 2017].

Available from: http://www.who.int/healthinfo/HSS_MandE_framework_Nov_2009.pdf.

18. The Australian National University. Policy Making and Implementation: Studies from Papua New Guinea: ANU E Press; 2009. Available from: http://press- files.anu.edu.au/downloads/press/p78541/pdf/book.pdf?referer=355. 146

Chapter 6: Conclusion and recommendations

19. Adam T, Hsu J, de Savigny D, Lavis JN, Rottingen JA, Bennett S. Evaluating health systems strengthening interventions in low-income and middle-income countries: are we asking the right questions? Health Policy and Planning. 2012;27

Suppl 4:iv9-19.

20. Bloomberg Philanthropies. Papua New Guinea: CRVS country overview [cited

2017 27 December]. Available from: http://mspgh.unimelb.edu.au/__data/assets/pdf_file/0020/2330408/Overview_PNG_02

1701.pdf.

21. Rosewell A, Makita L, Muscatello D, John LN, Bieb S, Hutton R, Ramamurthy

S, Shearman P. Health information system strengthening and malaria elimination in

Papua New Guinea. Malaria Journal. 2017;16(1):278.

147

Appendix. Strengthening Health services in Western Province, Papua New Guinea: Progress Report of the North Fly Health Services Development Program and the CMCA Middle and South Fly Health Program

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250