Research Data for development in health: a case study and monitoring framework from Kazakhstan Konrad Obermann,1 Tata Chanturidze,2 Erica Richardson,3 Serik Tanirbergenov,4 Marat Shoranov,5 Ali Nurgozhaev6 To cite: Obermann K, ABSTRACT Key questions Chanturidze T, Richardson E, Healthcare reforms are often not coupled with a et al . Data for development in relevant and appropriate monitoring framework, leaving health: a case study and What is already known about this topic? policymakers and the public without evidence about monitoring framework from ▸ Healthcare reforms are always complex and thus Kazakhstan. BMJ Global the implications of such reforms. require feedback about their impact. Health 2016;1:e000003. Kazakhstan has embarked on a large-scale reform of ▸ Healthcare policy initiatives are often not doi:10.1136/bmjgh-2015- its healthcare system in order to achieve Universal coupled with a relevant and applicable monitor- 000003 Health Coverage. The health-related 2020 Strategic ing framework. Development Goals reflect this political ambition. In a case-study approach and on the basis of published and What are the new findings? Received 28 October 2015 unpublished evidence as well as personal involvement ▸ The existing national monitoring concept was Accepted 1 March 2016 and experience (A) the indicators in the 2020 Strategic reviewed and found to have substantial short- Development Goals were assessed and (B) a ‘data- comings in the health policy aspects covered mapping’ exercise was conducted, where the WHO and the usefulness of indicators. health system framework was used to describe the ▸ A simplified framework was presented, coupling data available at present in Kazakhstan and comment national needs and the comprehensive monitor- on the different indicators regarding their usefulness ing framework developed by the WHO, the for monitoring the current health-related 2020 Strategic World Bank and the Global Fund. Development Goals in Kazakhstan. It was concluded that the country’s current Recommendations for policy monitoring framework needs further development to ▸ Initiate a dialogue about the appropriateness of track the progress and outcomes of policy the suggested framework with national counter- implementation. The application of a modified WHO/ parts, develop national capabilities and apply the World Bank/Global Fund health system monitoring simplified framework to feedback results. framework was suggested to examine the implications of recent health sector reforms. Lessons drawn from 2 the Kazakhstan experience on tailoring the suggested evaluating effective policies. The WHO, in framework, collecting the data, and using the generated collaboration with the World Bank and the intelligence in policy development and decision-making Global Fund, published a comprehensive oper- can serve as a useful example for other middle-income ational framework on the monitoring and evalu- countries, potentially enabling them to fast-track ation of health systems strengthening.3 In 2010, developments in the health sector. the WHO presented ‘Monitoring the Building Blocks of Health Systems: A Handbook of Indicators and Their Measurement Strategies’.4 In 2014, the WHO and The World Bank pub- INTRODUCTION lished a framework for monitoring Universal Reforming a healthcare system is always a Health Coverage (UHC) as part of a compre- complex undertaking, characterised by con- hensive framework for monitoring national flicting interests and limited evidence, and it health system performance at national and is often not known whether well-intended international levels with guiding principles initiatives will improve people’s lives.1 and methodological considerations.5 The For numbered affiliations see end of article. Since such complexity and uncertainty can need for encompassing Civil Registration and be found in most development and reform Vital Statistics as a core component of sustain- projects, the United Nations has launched able development has also been emphasised.6 Correspondence to ‘ Professor Konrad Obermann; a large-scale data revolution for sustainable The importance of effective and comprehen- konrad.obermann@medma. development’ in order to provide the right sive measurement of change in health uni-heidelberg.de information for designing, monitoring and systems is increasingly being recognised. Obermann K, et al. BMJ Glob Health 2016;1:e000003. doi:10.1136/bmjgh-2015-000003 1 BMJ Global Health In this paper, the health reform experience in (both level and equity), responsiveness and financial Kazakhstan is used as a case study to explore what data protection (see figure 2).910This framework is simple are needed for the meaningful monitoring of reforms. yet allows one to capture structures, processes and The findings contribute to the global debate on meas- results. urement and data production for development in the Formal progress towards UHC assessed by intermedi- health sector. ate goals will not automatically mean better population The following section (Theoretical background and health,11 so measuring effects at the population level methods) provides a theoretical background both for will be as important as measuring intermediate indica- monitoring the implications of policy changes and the tors like equitable financing and access.12 13 Resonating case study approach taken. Context: healthcare in with its strategy of consolidating and unifying data col- Kazakhstan section then sets the context, giving a brief lection and evaluation for health systems worldwide, the overview of the Kazakh health system and recent reforms. WHO in collaboration with the World Bank and the Next, the findings of the monitoring framework as applic- Global Fund in 2009 published a comprehensive oper- able to Kazakhstan and available data are presented ational framework on the monitoring and evaluation of (section Findings). The discussion (section Discussion) health systems strengthening based on the International presents the development of a comprehensive yet man- Health Partnership (IHP+) common evaluation frame- ageable monitoring and assessment framework. Finally, in work (figure 3).35 the Conclusion section, we conclude what could be learnt This concept was taken up again in 2010 with a hand- from the Kazakh experience and provide the outlook for book on suitable indicators and measurement strategies further developments. ‘for monitoring performance and evaluating progress in the scale-up for better health’.4 The recent UN initiative on the ‘data revolution’ has THEORETICAL BACKGROUND AND METHODS put forward the importance of data for sustainable dev- Monitoring the implications of policy changes—a elopment. ‘Data are the lifeblood of decision-making and framework the raw material for accountability. Without high-quality Given that health policy is frequently not evidence-driven,7 data providing the right information on the right things policymaking regarding the provision and financing of at the right time; designing, monitoring and evaluating healthcare is often as much based on the perceptions of effective policies becomes almost impossible’.2 politicians, the influence of advisors and personal empir- ical evidence from healthcare systems elsewhere as it is on Method: the case study approach rigorous analysis and systematic identification of strengths We use the case study approach14 primarily to describe and weaknesses of the status quo. Furthermore, the ‘meas- the current context and need for policy monitoring in urement of success’ of health financing and delivery Kazakhstan. On the basis of our personal involvement arrangements is still in its infancy.1 Figure 1 depicts the and experience, we (A) assessed the indicators in relation most general approach to looking at a healthcare system, to national policy in Kazakhstan and (B) conducted a starting from ideas and taking outcomes as the final goal. data-mapping exercise, where we used the WHO health Several approaches have been developed to establish a system framework to describe the data available in conceptual framework with a clear distinction between Kazakhstan and to comment on their usefulness in aug- objectives and instruments, that is, between what the menting the health-related 2020 Strategic Development system does and what the system is trying to achieve.8 Goals in Kazakhstan. One of the most widely used frameworks is based on (A) ‘The case study approach allows in-depth, multi- the WHO’s six building blocks of any healthcare system, a faceted explorations of complex issues in their real-life mix of inputs, processes and management issues4 and settings.’14 It has only recently become more widely used (B) the core functions of a health system: Improved health in health service research. Early examples of the case Figure 1 Ideas, interests and institutions in healthcare. Source: Modified after.1 2 Obermann K, et al. BMJ Glob Health 2016;1:e000003. doi:10.1136/bmjgh-2015-000003 BMJ Global Health Figure 2 The WHO Health System Framework with building blocks and goals outcomes. Source: Adopted from.4 study approach in health policy have been more descrip- be prone to discussion and criticism; yellow indicated tive with little theory15 or have emphasised the personal that some element of verifiable assessment was possible, involvement and experience of the authors.16 Later whereas green indicated clear (and often easy) measure- research has tried to provide a more nuanced theoret- ment. To the best of our knowledge, such an assessment ical grounding and combined methods from