Part I: Policy and economic issues Part II: Pharmaceutical management Part III: Management support systems Planning and administration Organization and management Information management 48 Monitoring and evaluation 49 Pharmaceutical management information systems 50 Computers in pharmaceutical management Human resources management chapter 48 Monitoring and evaluation Summary 48.2 illustrations 48.1 Definitions of monitoring and evaluation 48.2 Figure 48-1 The health management process and illustrative indicators 48.9 48.2 Monitoring issues 48.2 Table 48-1 Examples of performance indicators and 48.3 Monitoring methods 48.3 performance targets 48.10 Supervisory visits • Routine reporting • Sentinel reporting Table 48-2 Example of a list of indicator pharmaceuticals and systems • Special studies supplies 48.12 48.4 Designing the monitoring system 48.6 Table 48-3 First-year progress review for promoting rational 48.5 Using indicators for monitoring and medicine use 48.14 Table 48-4 Types of evaluation 48.15 evaluation 48.8 Table 48-5 Sample assessment budget template 48.18 Applications of indicators • Types of indicators • Selecting indicators • Setting performance targets • Formulation of country studies indicators • Indicator pharmaceuticals and supplies CS 48-1 Developing a monitoring and evaluation 48.6 Using the monitoring system to improve component for a new ART program in Mombasa, performance 48.12 Kenya 48.4 Communicating plans and targets • Reviewing progress • CS 48-2 Monitoring the introduction of medicine fees in Giving feedback • Taking action Kenya 48.7 48.7 Evaluation 48.15 CS 48-3 Using supervisory visits to improve medicine Evaluation questions • Conducting an evaluation • management in Zimbabwe 48.8 Evaluation methods and tools • Who should evaluate? • CS 48-4 Monitoring the use of neuroleptic medicines in a Resources required psychiatric hospital in Tatarstan. Russua 48.13 CS 48-5 Use of indicators for self-monitoring in rural 48.8 Some common pitfalls in monitoring and Java 48.16 evaluation 48.17 References and further readings 48.19 Assessment guide 48.20 copyright © management sciences for health 2012 48.2 INFOrmatiON MANAGEMENT SUMMARY Monitoring refers to the ongoing review of the progress clarity, usefulness, measurability, reliability, and validity, toward completing program activities and achieving as well as acceptance by key stakeholders. objectives. It allows corrective action during program To be effective in improving program performance, implementation. Monitoring systems focus on inputs monitoring requires— and short-term outputs and should be an integral part of day-to-day management. • Clear communication of plans and targets • Regular review and sharing of monitoring results Fully developed monitoring systems, which may be • Follow-up to provide feedback and take corrective established in phases, typically consist of a combination action of four methods— Evaluation is commonly discussed along with monitoring 1. Supervisory visits for continual, informal moni- as part of an overall strategy. It refers to the periodic anal- toring of workplan implementation and progress ysis of a program’s progress toward meeting established toward program plans objectives and goals. Evaluations fall into three categories, 2. Routine reporting of selected data through the phar- which differ in timing and purpose— maceutical management information system (PMIS) 3. Sentinel sites for more detailed reporting when new • Needs assessment (situation analysis, see Chapter 36) initiatives or rapid expansion requires more inten- • Formative evaluation (midterm review) sive monitoring • Summative evaluation (final evaluation) 4. Special studies whenever an implementation prob- Evaluations use data collected through the ongoing lem or planning question requires specific additional monitoring system, supplemented by document review, information interviews, additional data collection, and field surveys Performance indicators can facilitate tracking a pro- using standard pharmaceutical assessment indicators. gram’s progress toward established performance targets Strategies for monitoring and evaluation are normally or milestones and help compare this progress to that of developed in parallel to ensure a comprehensive, unified other programs. Indicators should meet the criteria of evaluation strategy. 48.1 Definitions of monitoring and evaluation tive and concentrates on the strengths and weaknesses of the program strategies. Monitoring refers to reviewing, on a continuous basis, the Country Study 48-1 illustrates the monitoring and evalu- degree to which program activities are completed and per- ation system put into place for pharmacy and laboratory formance targets or milestones are being met. Typically, services in a new antiretroviral therapy (ART) program in monitoring focuses on tracking program inputs such as Kenya. funding, staff, facilities, supplies, and training. As such, This chapter is concerned mostly with monitoring phar- monitoring is part of the operational management of a pro- maceutical supply systems. Monitoring is closely linked gram. Monitoring also tracks outputs such as availability of with the pharmaceutical management information system, medicines and supplies, number or percentage of trained described in Chapter 49. Evaluation, which is closely linked staff, and quality of services. Systematic monitoring of inputs with systematic assessment (Chapter 36), is discussed briefly and outputs can help identify potential problems and cor- at the end of this chapter. rective actions to be taken during program implementation. Evaluation refers to analyzing progress toward meeting established objectives, goals, or results. It provides feedback 48.2 Monitoring issues on the outcomes of activities, such as changes in prescrib- ing behavior and health care–seeking behavior, whether Systematic and ongoing monitoring is essential for ensur- plans have been met, and the reasons for success or failure. ing that program performance is on track, for improving Evaluation should also provide direction for future pro- performance, and for achieving long-term program goals grammatic plans. Evaluation methods may be used to carry and results. Unfortunately, during program implementa- out a situation analysis or a needs assessment as the first step tion, attention can easily get focused on specific technical in designing an appropriate intervention to improve pro- activities at the expense of monitoring activities. Too often, gram performance. Evaluation takes a longer-term perspec- monitoring is done casually, without a clear plan, without a 48 / Monitoring and evaluation 48.3 clear link to program objectives and targets, and without any of four formal and informal methods: supervisory visits, effort to use monitoring results to improve program perfor- routine reporting, sentinel reporting systems, and special mance. studies. Monitoring should be an integral part of the day-to-day management of pharmaceutical supply systems. Managers Supervisory visits of pharmaceutical supply programs are concerned with getting the most out of scarce resources. Meeting this goal Supervisory visits support the performance of individual means making the program as efficient and as effective as staff or health care workers, provide some on-site, in-service possible. Therefore, managers need to generate current, reli- training, and represent an important method for informal able information to use in making decisions on program but direct monitoring of program implementation (see performance and operations. Chapter 51). The monitoring system should center on key program Supervisory visits should reinforce routine reporting activities and objectives. To operate efficiently, it must focus requirements. Such visits may include checking the qual- on a small number of specific, clearly formulated monitor- ity of entries on standard reporting forms, such as inven- ing issues that are directly related to performance and are tory management forms (for example, stock cards). Visits generally taken from program plans, objectives, and targets. sometimes involve the collection of information, such as the Monitoring is intended to— availability of specific medicines, for special studies. • Determine whether activities are being carried out as Routine reporting planned • Measure achievement of targets The core of a monitoring system for pharmaceutical sup- • Identify implementation problems to initiate correc- ply programs is the routine reporting that is accomplished tive action through the pharmaceutical management information sys- • Identify and reinforce good performance tem. A PMIS consists of record-keeping documents; data • Identify and strengthen weak performance reporting forms; feedback reports; and procedures that • Help target supervision toward problem areas govern the availability, use, and flow of information up and • Assess whether activities are having their expected down the system, including tracking the availability and use effect of medicines. • Assess long-term trends Chapter 49 provides a practical overview and specific • Contribute to reviewing and revising program priori- guidance on designing an effective PMIS. It emphasizes the ties and plans need to build on existing recording and reporting systems, to involve users in developing the system, to use appropriate Funding agencies and donors may impose their par- data collection methods, to integrate the PMIS with other ticular reporting requirements for their own monitoring