Part I: Policy and economic issues Part II: Pharmaceutical management Part III: Management support systems Planning and administration Organization and management Information management 48 Monitoring and evaluation 49 Pharmaceutical management information systems 50 Computers in pharmaceutical management Human resources management
chapter 48 Monitoring and evaluation
Summary 48.2 illustrations 48.1 Definitions of monitoring and evaluation 48.2 Figure 48-1 The health management process and illustrative indicators 48.9 48.2 Monitoring issues 48.2 Table 48-1 Examples of performance indicators and 48.3 Monitoring methods 48.3 performance targets 48.10 Supervisory visits • Routine reporting • Sentinel reporting Table 48-2 Example of a list of indicator pharmaceuticals and systems • Special studies supplies 48.12 48.4 Designing the monitoring system 48.6 Table 48-3 First-year progress review for promoting rational 48.5 Using indicators for monitoring and medicine use 48.14 Table 48-4 Types of evaluation 48.15 evaluation 48.8 Table 48-5 Sample assessment budget template 48.18 Applications of indicators • Types of indicators • Selecting indicators • Setting performance targets • Formulation of country studies indicators • Indicator pharmaceuticals and supplies CS 48-1 Developing a monitoring and evaluation 48.6 Using the monitoring system to improve component for a new ART program in Mombasa, performance 48.12 Kenya 48.4 Communicating plans and targets • Reviewing progress • CS 48-2 Monitoring the introduction of medicine fees in Giving feedback • Taking action Kenya 48.7 48.7 Evaluation 48.15 CS 48-3 Using supervisory visits to improve medicine Evaluation questions • Conducting an evaluation • management in Zimbabwe 48.8 Evaluation methods and tools • Who should evaluate? • CS 48-4 Monitoring the use of neuroleptic medicines in a Resources required psychiatric hospital in Tatarstan. Russua 48.13 CS 48-5 Use of indicators for self-monitoring in rural 48.8 Some common pitfalls in monitoring and Java 48.16 evaluation 48.17 References and further readings 48.19 Assessment guide 48.20
copyright © management sciences for health 2012 48.2 Information management
summary
Monitoring refers to the ongoing review of the progress clarity, usefulness, measurability, reliability, and validity, toward completing program activities and achieving as well as acceptance by key stakeholders. objectives. It allows corrective action during program To be effective in improving program performance, implementation. Monitoring systems focus on inputs monitoring requires— and short-term outputs and should be an integral part of day-to-day management. • Clear communication of plans and targets • Regular review and sharing of monitoring results Fully developed monitoring systems, which may be • Follow-up to provide feedback and take corrective established in phases, typically consist of a combination action of four methods— Evaluation is commonly discussed along with monitoring 1. Supervisory visits for continual, informal moni- as part of an overall strategy. It refers to the periodic anal- toring of workplan implementation and progress ysis of a program’s progress toward meeting established toward program plans objectives and goals. Evaluations fall into three categories, 2. Routine reporting of selected data through the phar- which differ in timing and purpose— maceutical management information system (PMIS) 3. Sentinel sites for more detailed reporting when new • Needs assessment (situation analysis, see Chapter 36) initiatives or rapid expansion requires more inten- • Formative evaluation (midterm review) sive monitoring • Summative evaluation (final evaluation) 4. Special studies whenever an implementation prob- Evaluations use data collected through the ongoing lem or planning question requires specific additional monitoring system, supplemented by document review, information interviews, additional data collection, and field surveys Performance indicators can facilitate tracking a pro- using standard pharmaceutical assessment indicators. gram’s progress toward established performance targets Strategies for monitoring and evaluation are normally or milestones and help compare this progress to that of developed in parallel to ensure a comprehensive, unified other programs. Indicators should meet the criteria of evaluation strategy.
48.1 Definitions of monitoring and evaluation tive and concentrates on the strengths and weaknesses of the program strategies. Monitoring refers to reviewing, on a continuous basis, the Country Study 48-1 illustrates the monitoring and evalu- degree to which program activities are completed and per- ation system put into place for pharmacy and laboratory formance targets or milestones are being met. Typically, services in a new antiretroviral therapy (ART) program in monitoring focuses on tracking program inputs such as Kenya. funding, staff, facilities, supplies, and training. As such, This chapter is concerned mostly with monitoring phar- monitoring is part of the operational management of a pro- maceutical supply systems. Monitoring is closely linked gram. Monitoring also tracks outputs such as availability of with the pharmaceutical management information system, medicines and supplies, number or percentage of trained described in Chapter 49. Evaluation, which is closely linked staff, and quality of services. Systematic monitoring of inputs with systematic assessment (Chapter 36), is discussed briefly and outputs can help identify potential problems and cor- at the end of this chapter. rective actions to be taken during program implementation. Evaluation refers to analyzing progress toward meeting established objectives, goals, or results. It provides feedback 48.2 Monitoring issues on the outcomes of activities, such as changes in prescrib- ing behavior and health care–seeking behavior, whether Systematic and ongoing monitoring is essential for ensur- plans have been met, and the reasons for success or failure. ing that program performance is on track, for improving Evaluation should also provide direction for future pro- performance, and for achieving long-term program goals grammatic plans. Evaluation methods may be used to carry and results. Unfortunately, during program implementa- out a situation analysis or a needs assessment as the first step tion, attention can easily get focused on specific technical in designing an appropriate intervention to improve pro- activities at the expense of monitoring activities. Too often, gram performance. Evaluation takes a longer-term perspec- monitoring is done casually, without a clear plan, without a 48 / Monitoring and evaluation 48.3 clear link to program objectives and targets, and without any of four formal and informal methods: supervisory visits, effort to use monitoring results to improve program perfor- routine reporting, sentinel reporting systems, and special mance. studies. Monitoring should be an integral part of the day-to-day management of pharmaceutical supply systems. Managers Supervisory visits of pharmaceutical supply programs are concerned with getting the most out of scarce resources. Meeting this goal Supervisory visits support the performance of individual means making the program as efficient and as effective as staff or health care workers, provide some on-site, in-service possible. Therefore, managers need to generate current, reli- training, and represent an important method for informal able information to use in making decisions on program but direct monitoring of program implementation (see performance and operations. Chapter 51). The monitoring system should center on key program Supervisory visits should reinforce routine reporting activities and objectives. To operate efficiently, it must focus requirements. Such visits may include checking the qual- on a small number of specific, clearly formulated monitor- ity of entries on standard reporting forms, such as inven- ing issues that are directly related to performance and are tory management forms (for example, stock cards). Visits generally taken from program plans, objectives, and targets. sometimes involve the collection of information, such as the Monitoring is intended to— availability of specific medicines, for special studies.
• Determine whether activities are being carried out as Routine reporting planned • Measure achievement of targets The core of a monitoring system for pharmaceutical sup- • Identify implementation problems to initiate correc- ply programs is the routine reporting that is accomplished tive action through the pharmaceutical management information sys- • Identify and reinforce good performance tem. A PMIS consists of record-keeping documents; data • Identify and strengthen weak performance reporting forms; feedback reports; and procedures that • Help target supervision toward problem areas govern the availability, use, and flow of information up and • Assess whether activities are having their expected down the system, including tracking the availability and use effect of medicines. • Assess long-term trends Chapter 49 provides a practical overview and specific • Contribute to reviewing and revising program priori- guidance on designing an effective PMIS. It emphasizes the ties and plans need to build on existing recording and reporting systems, to involve users in developing the system, to use appropriate Funding agencies and donors may impose their par- data collection methods, to integrate the PMIS with other ticular reporting requirements for their own monitoring information systems, to take advantage of practical analysis and reporting purposes. For the most part, these reporting methods, and to communicate information promptly and requirements do not deviate much from standard report- clearly. ing needs. The advent of international funding initiatives, Depending on the program’s objectives, routine report- such as the Global Fund to Fight AIDS, Tuberculosis and ing focuses on the availability of supplies at different levels Malaria, has been accompanied by increased attention to in the system, finances, procurement and supplier perfor- the issue of ensuring accountability in the use of funds and mance, training, quality assurance, and medicine use. Clear achievement of program goals (see References and Further presentation of reported information, feedback to those Readings). Unfortunately, specific reporting requirements, providing the information, and follow-up action contribute performance indicators, and targets may differ significantly to the effectiveness and usefulness of the reporting system. between donors and funders, thereby increasing the moni- Reports should include both quantitative information and toring and reporting burden of funding recipients. brief descriptions of processes, key problems, and proposed Ultimately, monitoring is meant to improve the long-term follow-up actions. Some countries are now using mobile performance of the program and individual staff members. telephones to transmit routine monitoring data to a central location for timely analysis or even analyzing data at remote locations. For example, the National Malaria Control 48.3 Monitoring methods Program in Malawi is using a mobile phone–based tool to collect and analyze data on malaria commodity availability How are monitoring activities organized, and where does and case management indicators during supervisory visits monitoring information come from? Information required to facilities. This method allows supervisors to identify and for monitoring can be obtained through a combination address problems quickly. 48.4 Information management
Country Study 48-1 Developing a monitoring and evaluation component for a new ART program in Mombasa, Kenya
In countries that are introducing and scaling up ART • Monitor all aspects (input, process, output, and programs, a challenge is improving ART access quickly quality) of the program, including support systems while at the same time working to strengthen systems (for example, availability of human resources and that support long-term quality care. In 2002, the govern- enhancement of their capacities). ment of Kenya, with technical assistance from various • Build on the existing information system to mini- partners, initiated an ART program in four health facili- mize the additional burden on service providers. ties in Mombasa. The overall goal of the Mombasa ART • Integrate feedback mechanisms at all levels. program was to reduce HIV/AIDS-related morbidity and • Ensure system simplicity to facilitate staff training mortality and to improve the quality of life of people liv- on its modification and use. ing with HIV/AIDS and their families in the Mombasa • Make sure the system is functioning manually before district. The program’s specific objectives were to— computerizing it. • Improve the capacity of HIV/AIDS clinics, labora- The diagram opposite illustrates the processes and timing tory, and pharmacy services in selected public health of the monitoring and evaluation system throughout the facilities in Mombasa to provide HIV/AIDS com- life of the program (five years). prehensive care, including ART Baseline status: Result of the pre-implementation assess- • Provide ART to 300 patients over a period of five ment formed the baseline status. years in accordance with eligibility criteria • Sensitize communities and strengthen support Activity monitoring: Pharmacist and laboratory staff groups’ knowledge of comprehensive HIV/AIDS responsible for the ART program monitored the activi- care, including ART ties during the week and reported to their supervisors at the end of each week. A monitoring and evaluation system was incorporated into the program design to provide quality information Supportive auditing: At the end of each month, the to help decision makers take timely corrective actions supervisor, together with the respective responsible and ensure that the program is achieving its goal and staff members, performed an audit of the activities and objectives. resources and discusses problems and solutions. The key features of the system were to— Review with pharmacy/laboratory technical partner: At the end of each quarter, all staff related to ART (phar- • Encourage analysis and use of information by using macy and laboratory) and technical partner staff jointly well-defined indicators rather than only reporting. reviewed the activities and progress and discussed prob- The system stresses the use of data at the point where lems and solutions. Quarterly indicators were used as a data are collected. basis for discussion.
The biggest failures in routine reporting systems are ceutical supply cycle. The simple rule is: do not collect what overdesign and underimplementation. Collecting too much you cannot use. data usually results in too little analysis. Also, reporting sys- tems that are overly complex result in poor compliance with Sentinel reporting systems reporting requirements. Implementing an information sys- tem takes time and money. Moreover, the more complex the When routine reporting systems are properly collecting only system is, the greater is the need for qualified and trained the minimal essential information, a great deal of potentially staff, time, and money to make it function. useful information is not included. To supplement routine Therefore, the content of routine reporting systems should reporting, sentinel reporting systems can be useful. be limited to the minimum amount of information that the A sentinel reporting system consists of a carefully selected typical reporting unit can reasonably be expected to provide sample of health facilities or dispensaries that are given and that can routinely be analyzed for decision making and greater recording and reporting responsibilities. For exam- feedback purposes. Identification of key data requirements ple, in a country with forty districts, six districts may be is the result of a prioritization exercise that considers the selected as sentinel districts. Within each district, a sample critical information needs of all components of the pharma- of health facilities from each level may be selected. Sentinel 48 / Monitoring and evaluation 48.5
Review with all technical partners: All technical Kenyan Ministry of Health (MOH). Quarterly indicators assistance partners participated in the biannual review. were used as a basis for discussion. Quarterly indicators were used as a basis for discussion. Final evaluation: The pharmacy/laboratory partner Midterm evaluation: The technical partner conducted conducted a program-end review of related activities and a midterm review of activities related to pharmacy and participated in the final evaluation jointly conducted by laboratory and also participated in the midterm evalua- the other technical assistance partners and the MOH. tion jointly conducted by all the partners with the Source: Bhattarai and Walkowiak 2005.
Weeks 1 2 3 4 1 2 3 4 1 2 3 4 … 1 2 3 4 … 1 2 3 4 … 1 2 3 4 Processes Months 1 2 3 … 6 … 30 … 60 Baseline status § Activity monitoring § § § § § § § § § § § § § § § § § § § § § § § Supportive auditing § § § § § § Review with pharmacy/laboratory § § § § technical partner Review with all technical partners § § Midterm evaluation § Final evaluation §
reporting differs from routine reporting in the amount of pected or undesirable issues related to program implemen- information collected, the frequency or promptness of tation, and information on the availability of antiretrovirals, report submission, and the level of accuracy demanded. patient adherence to medication regimens, and other key Sentinel reporting is common in disease control pro- variables may be collected and rapidly reported. grams such as HIV/AIDS prevention, control of diarrheal Whereas the routine reporting system answers only the disease, and treatment of sexually transmitted diseases. questions it asks, a sentinel reporting system can be useful Sentinel sites provide a relatively economical means of col- to detect unexpected or unintended outcomes. However, lecting up-to-date, detailed information on disease inci- recording and reporting formats developed for sentinel sites dence, antimicrobial resistance, and responses to program sometimes prove sufficiently useful and convenient to be interventions. incorporated into routine reporting. Need for sentinel reporting. A sentinel reporting sys- Selection of sentinel sites. The level and number of tem is most useful when a system is undergoing rapid or health units in a sentinel system depend on the organization substantial change, such as when a new treatment or public of the health and pharmaceutical management systems, the health intervention is being introduced. Sentinel reporting monitoring objectives, and the overall diversity within the helps assess the implementation and short-term effect of country. If, for example, a country has only one central med- changes or interventions such as— ical store and a handful of regional stores, they would all be included in sentinel reporting requirements. Generally, • Introduction of a major new computerized stock man- at least six units should be included within each level. For agement system example, a sentinel system could include six districts, six • Transition from a central medical stores system to a district hospitals, and six health centers within each district decentralized system (thirty-six health centers total). Larger numbers of units are • Introduction or scale-up of a treatment program, such needed in more populous, more organizationally complex, as ART or artemisinin-based combination therapy or more diverse countries. Selection of specific districts, locations within districts, Sentinel reporting is particularly important for new or or health facilities is generally purposeful, not random. expanded HIV/AIDS treatment programs where donors Selection is aimed at achieving diversity rather than statis- may have many new record-keeping requirements and tical representativeness. The selected sites should represent where many unknowns may exist regarding how the patients the range of facilities that serve culturally, linguistically, and will be integrated into the health system. Policy makers and geographically diverse groups; the distribution of ministry managers can recognize and react more quickly to unex- and nongovernmental facilities; and accessibility. 48.6 Information management
Staffing levels at sentinel sites should be sufficient to han- • Level of patient adherence to prescribed treatment dle additional recording and reporting requirements, which • Level of prescriber adherence to standard treatment may mean hiring extra staff at some sites. Extra incentives protocols are sometimes given for prompt, complete, accurate report- ing, although this practice should be avoided. Prompt Facility or population-based survey and research methods feedback of information can be sufficient to motivate staff are typically used to obtain this type of information. These to respond quickly and reliably. Because cost is a consider- methods may involve expenses that are not considered in ation, relatively inaccessible sites are usually chosen as senti- regular recurrent operating costs and may require specialists nel sites, even at the price of omitting cultural or geographic to design and carry out, but many questions can be answered diversity; however, if these inaccessible sites are particularly with relatively simple methods and at little additional cost. likely to suffer shortages of medicines and delayed delivery, For example, Chapter 40 describes how ABC analysis, lead- including some of them in the sentinel system may be worth time analysis, expiry-date analysis, and pipeline analysis can the effort, especially if there is an intent to make changes in help improve performance and reduce costs. the system to address these deficiencies. Both managers and staff should be involved in the design Information required from sentinel sites. Sentinel sites and implementation of these studies, as well as in the analy- may differ in the amount of information recorded, the sis and interpretation of results. However, it is also the man- amount of information routinely reported, or the speed at ager’s role to identify when additional information is needed which the information is processed. Additional recording and to use experts to help design and conduct these special and reporting requirements should be based on the reasons investigations, so that resources are not wasted on a useless that the sentinel system was established, such as monitoring effort. a new stock management system, assessing transition to a In-depth interviews, structured observation, focus group supply agency system, or evaluating the effect of a user-fee discussions, or other qualitative methods (see Chapter 28) program. can be used to explore behavior, attitudes, practices, and Most of the principles used in designing a PMIS apply causal factors. to sentinel reporting: being selective in deciding which Rapid assessments are small-scale studies that include a indicators to collect, choosing appropriate data collection survey of sample facilities, use of selected core indicators, methods, and building on existing recording and reporting and interviews with key informants. Rapid assessments typ- systems whenever possible. In some instances, putting in ically are designed and conducted within one month (see computers at sentinel sites may be appropriate (or choosing MSH/RPM et al. 1995). sites that already have computer capability), even if national Country Study 48-2 describes how these four approaches computerization is not currently foreseen. were combined and how the system evolved for monitoring Working with sentinel sites. Establishing a system of medicine and treatment fees in one East African country. sentinel sites nearly always requires training key staff at the sites and making frequent supervisory visits, especially in the beginning. Sites may require additional forms, registers, 48.4 Designing the monitoring system stock records, and other materials. Finally, a system of sentinel sites is generally maintained The principles for designing a monitoring system are to for a specified period of time, perhaps for the first several focus on key monitoring questions and indicators; keep years of a transitional program. After that, these sites may data collection to a minimum; develop practical procedures revert to normal reporting status. for managing monitoring information; and consider the need to make comparisons between performance of simi- Special studies lar programs, performance of different facilities within the program, and performance over time. Design should also Sometimes managers and planners need to gather informa- consider how information will be available for timely feed- tion that is not available from routine or sentinel report- back and follow-up action. The “KISS” concept should be ing. Examples of topics that may warrant a special study applied: keep it simple and straightforward (or keep it short include— and simple). When time pressures, financial resources, and staff mem- • Names, dosage forms, and values of medicines pur- bers’ inexperience with monitoring methods are limiting chased by individual health units factors, phasing the design and implementation may make • Names, dosage forms, and value of recently expired sense. Initial monitoring efforts can focus on issues of selec- medicines tion and procurement. Medical stores data, for example, are • Reasons for expiration of pharmaceuticals often easy to collect. As the program develops, the moni- • Average percentage of essential medicines available toring system and core indicators can be expanded to cover 48 / Monitoring and evaluation 48.7 medicine distribution and use. Regardless of the approach ous health units or facilities. When a facility is introducing taken, the design of the monitoring system must be based or phasing in a new activity, practice, or program, it can be on a sound understanding of how the system works, includ- matched to a control site that has not yet been introduced ing the relationships between system inputs, outputs, and to the new practice or program. The control site should desired outcomes. be as similar to the program site as possible, so that any Managers can often benefit by designing a monitoring changes can be more reliably attributed to the new pro- system that allows comparison of the performance of vari- gram. If no control site is available to use as a comparison,
Country Study 48-2 Monitoring the introduction of medicine fees in Kenya
The Kenyan Ministry of Health introduced a new user- of fees; (c) outpatient and inpatient surveys to assess fee program. After early implementation problems, the patients’ perceptions of the fee system and quality ministry initiated a program of management improve- changes; and (d) a quality-of-care checklist to assess the ment and regular fee adjustments. An outpatient medi- availability of critical patient care inputs. cine and treatment fee was introduced in phases. To Special studies: During implementation, questions assess the effect of the program—in particular the medi- arose that could not be answered through either routine cine treatment fee—a comprehensive monitoring system reporting or the indicator district system. Special studies was implemented. The system consisted of targeted field were, therefore, conducted on the planning and expen- supervision, routine reporting, a sentinel system of indi- diture process for the use of revenue; on fee preferences, cator districts, and special studies. to assist in expanding the fee schedule; on exemptions, Targeted field supervision: Supervisory staff from the to assist in adjusting exemptions to balance equity and MOH health financing program made regular supervi- revenue needs; and on revenue losses caused by non sory visits to each part of the country. Information from collection of inpatient fees. routine reporting and other sources was used to identify Uses of monitoring information: Supervision and problem districts and problem facilities. These were vis- routine reporting information were used to identify ited more often. Gradually, the role of headquarters staff districts and facilities that were performing poorly and evolved from primary supervision (directly visiting dis- to strengthen their performance. Supervisory visits tricts and facilities themselves) to secondary supervision: corrected misunderstandings of the new management teaching supervisory skills to provincial and district staff systems. In some cases, staff members were intentionally by making visits with counterparts. flouting new rules for personal gain, and disciplinary Routine reporting: Routine reporting used the financial action was taken. Districts and facilities that were per- information system (FIS) and the health information forming well were identified and publicly recognized. system (HIS). The FIS was developed specifically for the Data from the indicator districts were used to guide user-fee program and consisted of reports from districts decisions about the type, level, and timing of fee changes. and hospitals covering collections, expenditures, insur- Results from special studies were used to correct man- ance claims, exemptions, and bank balances. The exist- agement problems and to revise management systems. ing HIS was adapted to support additional information Evolution of the monitoring system: As the new user- needs of the user-fee system. fee system matured over a five-year period, the nature of Indicator districts: The routine reporting system was the monitoring system evolved. Supervision was decen- kept to a minimum for reasons of feasibility, cost, and tralized to the provincial and district levels. Routine staff availability. Therefore, a system of six indicator reporting requirements were simplified to focus on districts was developed to provide additional details on the few critical FIS and HIS reports that most facilities the implementation of user fees. Districts were selected could generate regularly. Data from indicator districts to achieve rural/urban and socioeconomic diversity. were important for monitoring major fee changes, but Specialized information gathering in the indicator dis- subsequent fee adjustments were having less and less tricts included (a) outpatient use data from all MOH effect, so little information was being gained from the levels as well as selected mission and private facilities; indicator districts. Efforts were then concentrated pri- (b) rapid household surveys before and after major fee marily on effective local use of targeted supervision and changes to assess care-seeking patterns and knowledge routine reports. 48.8 Information management
mance of the overall system. A well-defined indicator is Country Study 48-3 clearly linked to an important input, process, or outcome. A Using supervisory visits to improve medicine well-selected indicator will help managers quickly identify management in Zimbabwe potential problems in critical areas. Indicators are extremely helpful to communicate important performance gains and In Zimbabwe, training pharmaceutical technicians and losses to other stakeholders of the pharmaceutical supply health workers in medicine management resulted in system. Indicators can be developed for different levels of significant improvements in inventory management the supply system. and rational medicine use, but the achievements were Managers should be aware that well-established indicators not sustained. A monitoring program based on two exist for measuring the performance of the different com- supervisory visits every three months was instituted ponents of the pharmaceutical supply system (for example, in district-level health facilities to determine whether WHO 2007). However, because systems can be organized in progress in managing medicines could be maintained. different ways, managers should adapt or modify interna- The study compared three different groups— tionally recognized indicators to reflect the realities of their own system if necessary. 1. Twenty-three facilities receiving supervision on using standard treatment guidelines Applications of indicators 2. Twenty-one facilities receiving supervision on inventory management When used to make measurements at one point in time, 3. A control group of eighteen facilities receiving no indicators allow a manager to compare a program’s perfor- supervision in either area mance with a target level of performance (or with another The evaluation measured performance using a range program’s performance) and to identify areas of relative of indicators relating to medicine availability, use of strength and weakness. Applied over time, such indicators stock cards and books, and monthly ordering, as well can be used to set and monitor performance improvement as adherence to the standard treatment guidelines. targets, such as— The results showed that after supervision, overall inventory management and adherence to standard • Monitoring implementation of program plans and treatment guidelines improved significantly when workplans compared with the control and comparison groups. • Evaluating achievement of long-term goals In addition, supervisory visits had a positive effect on • Assessing the performance of individual units improving staff performance in other areas besides just • Identifying relative strengths and weaknesses in cur- inventory management and rational medicine use. rent policies and systems • Measuring the effect of new policies or management Source: Trap et al. 2004. systems • Self-monitoring to improve performance • Demonstrating needs to treasury, donors, or other baseline data can be collected to use as a basis for assess- funders ing development after the program is started at the site. If • Reporting on progress to senior officials, donors, or more than one facility is initiating a program, the different other interested parties facilities can serve as comparison groups to monitor how well each group is progressing. Country Study 48-3 details For program management purposes, whether at the the effects of supervisory visits using both comparison and national or facility level, performance indicators should control groups. tie directly to program plans and annual workplans and to a general performance improvement process. One way to view the management process is that it takes inputs (for 48.5 Using indicators for monitoring and example, human and financial resources, equipment, poli- evaluation cies) and the implementation of certain management and clinical processes (meetings, trainings, development of mate- Indicators are variables that measure change. They may be rials) to create specific service or activity outputs (number numerical and expressed in terms of numbers, percentages, of staff trained or clients served, new management sys- or averages. They may also be expressed as binomials such tem), which have immediate outcomes (change in practices, as “yes” and “no.” Indicators are useful tools for managers improved services), which lead to a desirable long-term to track the performance of particular aspects or activities impact on health status (change in disease rates, change in of the pharmaceutical supply system as well as the perfor- birth rates). This process is illustrated in Figure 48-1. 48 / Monitoring and evaluation 48.9
Figure 48-1 The health management process and policies and standard operating procedures. Examples of illustrative indicators corresponding indicators include—
Management Management Clinical process example example • Existence of a national medicine policy updated within the last three years Inputs USD 250,000 for Three clinicians • Percentage of total health program budget dedicated to Funds, sta , regional stores experienced in procuring pharmaceuticals vehicles, other resources acute respiratory infection (ARI) needed to carry out Managers are also concerned with activities or processes the program treatment such as procurement, stock keeping, and prescribing and their outputs. Examples of related indicators include—
Plan for Training course Processes • Percentage of the pharmaceutical budget spent on construction of developed on essential medicines Management activities, regional stores ARI treatment clinical protocols • Average supplier lead time from order to delivery (in days) • Average inventory turnover • Number of prescribers trained on standard treatment guidelines Outputs Three regional Twelve hundred stores health workers Number of services or constructed trained Important outcome indicators for pharmaceutical man- activities completed, agement focus on aspects of availability and affordability medicines delivered, sta trained of key medicines, quality issues, and the appropriate use of medicines. Indeed, these indicators are typically the most visible and most commonly cited in evaluating how success- fully a supply system is functioning. Examples of outcome Outcomes Drug availability Eighty percent indicators include— Immediate changes in increased to of ARI patients medicine availability, 80 percent correctly medicine quality, treated • Average number of days of stockouts (of key medicines rational medicine use and supplies) • Average number of items out of stock at a given point in time • Average number of medicines prescribed to a patient Reduced ARI Reduced ARI for a given condition Impact mortality mortality Long-term changes in health status Selecting indicators
Five necessary criteria for selecting appropriate perfor- mance indicators are—
Types of indicators • Clarity: the indicator is easily understood and calcu- lated. Indicators can be developed for each point in the health • Usefulness: the indicator reflects an important aspect management process to monitor specific inputs, processes, of performance. and outputs associated with both management and clinical • Measurability: the indicator can be defined in quan- activities. Similarly, pharmaceutical supply system manag- titative terms and used within existing constraints on ers can identify input, output, and outcome measures for information quality and availability. each major component of the pharmaceutical supply cycle. • Reliability: the indicator permits consistent assessment Table 48-1 lists commonly used indicators that correspond over time and among different observers. to the components of selection, procurement, distribution, • Validity: the indicator is a true measure of what it is use, policy, and management support. meant to measure (see Chapter 36). Validity must also Managers of pharmaceutical supply systems will note be based on the indicator’s acceptance by key stake- that many of their management activities focus on inputs holders and the consistency of interpretation among such as financing, human resources, and the existence of different stakeholders. 48.10 Information management
Table 48-1 Examples of performance indicators and performance targets
Objective and performance indicator Performance target Overall performance Central stores: indicator pharmaceuticals available (unexpired) 90% District stores: indicator pharmaceuticals available (unexpired) 90% Health units: indicator pharmaceuticals available (unexpired) 90% Health units: average stockout duration for indicator pharmaceuticals 10 days Financing—ensure that financial resources are adequate to meet basic pharmaceutical needs Per capita government pharmaceutical expenditure USD 1.20 MOH budget allocation for pharmaceuticals 15% Actual pharmaceutical expenditures as a percentage of budgeted allocation for pharmaceuticals 85% Medicine costs covered by user fees 60% Procurement—obtain a regular supply of pharmaceuticals at favorable prices Total value of MOH medicines purchased through competitive tender 95% Total value of MOH purchases on essential medicines list 98% Average ratio of unit prices of indicator pharmaceuticals to international prices 0.9 to 1.1 Average lead time for external suppliers 50 days Average lead time for local suppliers 20 days Quality assurance—ensure that procured pharmaceuticals meet recognized standards of quality and that quality is maintained throughout the distribution chain Number of medicines/batches tested of number of medicines/batches procured 1 of 4 Number of medicines/batches that failed quality control testing, of number of medicines/batches tested 0 Central storage—ensure that medications are properly stored, with minimal expiration and other losses Medicines for which stock records and physical counts agree (all medicines) 100% Average difference between stock records and physical count (all medicines ) Less than 5% Drugs for which stock records, procurement records, and physical counts agree (indicator pharmaceuticals) 100% Value of expired medicines as a percentage of total pharmaceutical purchases last year Less than 3% Delivery—ensure timely delivery of pharmaceuticals to health units Average lead time from central medical stores to health units, routine monthly orders 30 days Average lead time from central medical stores to health units, emergency orders 5 days Storage at health units—ensure that medicines are properly stored, with minimal expiration or other losses Health units using stock cards correctly 75% Stock records correspond with physical counts 90% Health units with expired items 25% Health units practicing first-expiry/first-out (FEFO) 90%
In selecting indicators, considering how data will be col- of health units using stock cards correctly). Some indicators lected is also important. Indicators may rely on generally represent annual summary measures (per capita govern- available information from routine reporting, or they may ment pharmaceutical expenditure or percentage by total require special surveys or other sources of information. The value of pharmaceuticals purchased through competitive sources and the cost of collecting and processing these data tender). must be carefully considered in selecting indicators. It may be useful to distinguish core performance indica- Some indicators may be routinely available from stan- tors, for which data are routinely reported and monitored, dard recording and reporting systems (such as percentage of from complementary performance indicators, for which indicator pharmaceuticals available), whereas other indica- data may be collected only at sentinel sites or in special tors may require a special survey (for example, percentage studies or that may be used only for periodic evaluation of pharmaceutical costs covered by user fees or percentage purposes. 48 / Monitoring and evaluation 48.11
The indicator examples in this chapter draw on five Formulation of indicators useful sets of internationally recognized indicators, some of which are described in greater detail in Chapter 36 If indicators are variables that measure change, they can (Brudon, Rainhorn, and Reich 1999; CPM 2003b; MSH/ be counted. Indicators can be in the form of counts (400 RPM et al. 1995; WHO 2007). By using internationally health workers trained), rates (two workshops per year), defined indicators for national pharmaceutical policy ratios (USD .90 per tin versus USD .60 per tin equals a ratio (Brudon, Rainhorn, and Reich 1999), pharmaceutical of 3:2), proportions (400 of 1,200 health workers trained management (MSH/RPM et al. 1995; WHO 2007), and equals one-third), or percentages (400/1,200 health work- medicine use (Nachbar et al. 2003; WHO 2007; WHO/ ers trained equals 33 percent). Indicators can also require a DAP 1993), countries can compare their performance “Yes” or “No” response (“Yes, there is a national pharmaceu- against that of other countries. tical policy,” or “No, standard treatment guidelines have not been updated in the last three years”). Setting performance targets When defining a numerical indicator, considering the availability of information is important. Proportions and A performance target is a desirable and—in principle— percentages require that the size of the whole (the denomi- attainable standard of performance. For example, the impact nator) be reliably known; if not, then actual counts may be indicator may be the percentage of ten indicator pharma- more useful. For example, if the number of health workers ceutical products in stock, and the performance target may in a district is not precisely known, specifying a count (400 be 80 percent availability at each level for this list of indica- health workers trained) is preferable to a percentage that tor pharmaceuticals. Performance targets should be set for cannot be verified. each indicator. Table 48-1 presents illustrative targets for the indicators listed. Indicator pharmaceuticals and supplies A widely used guide to developing performance targets is SMART (MSH/LMS 2008). A SMART result is— Several of the performance indicators listed in Table 48-1, mentioned in Chapter 36, or included in some of the chapter Specific (S): The performance target is clearly written to assessment guides are based on the availability, prices, and avoid different interpretations. accuracy of stock records for a list of indicator pharmaceu- Measurable (M): The target allows a team to monitor and ticals and supplies. evaluate progress toward achieving its result. Indicator pharmaceuticals and supplies are a small number Appropriate (A): It is in line with the scope of a team’s pro- of representative items, which are also known as tracer or gram or work activities, so that it can have influence or index medicines. Economists use a market basket of com- make changes. mon goods and services to measure inflation through the Realistic (R): The target is achievable within the time consumer price index. Similarly, the list of indicator medi- allowed. cines is sometimes called a basket of medicines (WHO Time-bound (T): The performance target has a specific time 2006). The advantage of using a list or basket of items for period for completion. indicators is that data collection and analysis are simplified. Selecting which items to include on a tracer list is based Performance indicators can be compared with agreed- on preestablished criteria and should include input from upon performance standards over time within the same important stakeholders. The items should be on the national health unit, among health units of the same level, or across essential medicines list or national medicine formulary, countries. Initially, comparisons among health units may therapeutically important, widely used, appropriate for the help set realistic performance standards, which can be level of care where measurements will be made, and com- adjusted upward as the system develops. monly available internationally (if the study includes mul- When setting targets, managers should keep in mind that tinational comparisons). In principle, these items should be costs are associated with reaching their targets, and there- available at all times (target 100 percent). For this reason, fore, they should consider budget and staff limitations. For inclusion of medicines that are not commonly used, that are example, attaining a 100 percent service level (in essence, difficult to obtain, or that have other unusual characteristics completely filling every order for medicines in one deliv- influencing how they should be managed is not practical. ery) may not be possible or practical, even though it may be Table 48-2 provides a list of medicines, grouped by thera- theoretically desirable. Therefore, the target should be set at peutic category, that was used to assess the pharmaceutical an appropriate level, and the manager’s challenge is to have a supply system in Tanzania. Depending on the purpose and strategy to ensure that shortfalls in the pharmaceutical ser- uses of the indicators, distinguishing a list of fifteen core vice level do not jeopardize the facility’s ability to provide indicator medicines to use as indicators for all levels of the quality care. health system and a supplementary list of ten additional 48.12 Information management
Table 48-2 Example of a list of indicator pharmaceuticals of ten or twenty medicines usually suffices. For calculating and supplies rates and percentages, using a number that divides easily into 100 (for example, 10, 20, or 25) is convenient. Larger Pharmaceutical/supply Form, dosage or more diverse lists of indicator pharmaceuticals may be Analgesics/antipyretics needed for specific purposes. When first introducing perfor- Acetylsalicylic acid (aspirin) Tablet, 300 mg mance indicators and indicator items, however, it is probably Paracetamol Tablet, 500 mg wise to keep to a core list such as that shown in Table 48-2. Antihelminthics Mebendazole Chewable tablet, 100 mg Anesthetic 48.6 Using the monitoring system to improve Ketamine Vial, 50 mg/mL performance Antibacterials A monitoring system gives managers a way to identify Amoxicilline Tablet, 250 mg potential problems with program and staff performance Metronidazole Tablet, 450 mg and to improve performance. A formalized monitoring Benzylpenicillin sodium Vial, 5 MU system facilitates the development of improvement plans Sulfamethoxazole/trimethoprim Tablet, 400 mg + 80 mg and performance targets, all of which must be clearly com- (co-trimoxazole) municated at all levels of the pharmaceutical management Ciprofloxacin Tablet, 500 mg system. For the monitoring system to be useful, managers Doxycycline Tablet, 100 mg should review and share the results regularly and take timely Erythromycin Tablet, 250 mg action to follow up. For example, Country Study 48-4 shows Gentamycin Ampoule, 40 mg/mL how a psychiatric hospital instituted a monitoring system Rifampicin + isoniazid Tablet, 150 mg/100 mg that included regular feedback to psychiatrists to improve prescribing habits. The ongoing monitoring activities will Antimalarials determine if follow-up actions achieved the desired results. Sulfadoxine/pyrimethamine Tablet, 500 mg/25 mg Quinine dihydrochloride Ampoule, 300 mg/mL Communicating plans and targets Cardiovascular medicines Propranolol Tablet, 40 mg Program plans and workplans are sometimes viewed simply Hydroclorothiazide Tablet, 25 mg as documents for fund-raising or for “the people above.” To Gastrointestinal medicines the contrary, they should be blueprints to guide the day-to- Oral rehydration salts Sachet day work of staff at all levels, who need to be aware of the Minerals plans as well as of output and performance targets. Managers should communicate plans and targets both in writing and Ferrous sulfate + folic acid Tablet, 200 mg/0.25 mg face-to-face: memoranda or circular letters alone are usu- Ophthalmological preparations ally insufficient. Often, staff members need some education Oxytetracycline eye ointment 1% Tube, 5 mg or training to understand why indicators were selected, how Vaccines they are measured, and how results will be used. Polio vaccine Vial Source: CPM 2003a. Reviewing progress
A schedule is needed to review program progress, with the medicines to use at central storage facilities and hospitals review period depending on the nature of the plan and the may be helpful. The supplementary list of items might specific indicators. Managers should review implementa- include other dosage forms, such as injections and topical tion progress for a five-year program plan, for example, at preparations, and additional therapeutic categories, such least once a year. During the review, key staff members meet as cardiovascular medications and contraceptives. Items to to systematically assess each objective, each activity, and consider for the core list include oral rehydration solutions, each output target. For an annual workplan, reviewing prog- procaine penicillin injections, paracetamol tablets, tetracy- ress each month or at least each quarter is important. New cline eye ointment, iodine, gentian violet or a local alter- programs should also be reviewed often as they are getting native, benzoic acid and salicylic acid ointment, or retinol off the ground. (vitamin A) (WHO 2007). In a progress review, actual outputs and performance are To facilitate measurement calculations and consistency, compared against the established targets; when discrepan- the number of items on the list should be kept small: a list cies exist between the expected results and actual perfor- 48 / Monitoring and evaluation 48.13 mance, further discussion or investigation is needed. When mation is. Feedback also demonstrates the value and impor- progress has exceeded expectations, managers should ask tance of the reports. As such, it represents one of the most whether positive lessons can be learned and whether spe- powerful tools for motivating staff. Feedback also improves cific people should be recognized for their work. Table 48-3 the quality of data by breaking the “bad data cycle” (see provides an example of a progress review based on a work- Chapter 49). plan to promote rational medicine use. Direct, action-oriented feedback involves presenting staff When progress is less than expected, it is even more impor- with some of the problems and successes identified in the tant for managers to ask the reasons why. For instance— monitoring or evaluation reports—
• Were necessary funds, materials, or other inputs lack- • Discuss the achievement of specific performance tar- ing? gets. • Were two units unable to communicate their expecta- • Identify weak performers for more intensive supervi- tions to each other or to coordinate their actions? sion or training. • Did key staff simply fail to take the plans and perfor- • Identify and congratulate successful districts and mance targets seriously? facilities. • Is failure to achieve a specific planned output a reflec- • Identify policy or program weaknesses and how to tion of overly ambitious plans? strengthen them.
A well-designed, well-implemented monitoring system Always check to see whether the information has been can usually provide information on what happened or did used, how it has been used, and what action has been taken. not happen. Information about why things happened—or did not happen—may come from the monitoring system or Taking action may require management follow-up. Effective use of the monitoring system requires prompt Giving feedback follow-up action. A manager can take at least five types of action— No monitoring system is complete without feedback. Giving feedback to individual units or staff members tells them how • Provide positive feedback to high-performing units or well they have done the reporting and how useful the infor- staff to encourage continued good performance.
Country Study 48-4 Monitoring the use of neuroleptic medicines in a psychiatric hospital in Tatarstan, Russia
Neuroleptic drugs, such as chlorpromazine, can help The hospital used standardized medicine-use indicators control symptoms related to schizophrenia, but overuse, developed by the World Health Organization (WHO) to polypharmacy, and medicine interactions can contribute track the outcomes related to the monitoring program. to dangerous adverse effects, such as irreversible move- Results showed that implementing medicine-use moni- ment disorders. The Clinical Psychiatric Hospital in the toring helped psychiatrists improve their prescribing Republic of Tatarstan, Russia, implemented a monitoring habits for schizophrenic patients, although adding an system to measure the use of neuroleptic drugs and related educational intervention component to regular monitor- adverse effects and to institute regular feedback to hospital ing might result in additional gains. psychiatrists aimed at decreasing the use of neuroleptic Source: Ziganshina et al. 2004. medicines and improving overall prescribing habits.
WHO medicine use indicatorsa Percentage of cases Percentage of Percentage of Number of prescribed generic Percentage of cases medicines on cases prescribed Year medicines/patient medicines including antibiotics formulary injections 1b 8.3 56.7 32.2 63.5 40.8 2 5.8 69.1 10.6 99.0 38.8 a WHO/DAP 1993. b Before monitoring system was implemented. 48.14 Information management Required Required additional to workshop finalize Materials placed order late too Comments 0 0 850 2,330 8,140 3,000 ( US D) 37,170 Balance 49,790 First year year First 0 0 0 2,670 4,830 2,850 39,860 50,210 First-year First-year actual ( US D) Expenditures 0 0 5,000 3,000 2,000 ( US D) 48,000 42,000 budget First-year First-year 100,000 Budgets 5,000 4,000 ( US D) 96,000 84,000 15,000 budget Five-year Five-year 204,000 See above See above ctual Initial workshop held, Initial held, workshop curricula not completed and Manual completed not distributed printed, first designed, Training held workshop not Materials ordered, received See above See above Done A Outputs Curricula designed for for designed Curricula nursing, pharmacy, medicine, training paramedical edition of STGs First and distributed completed of Sixty-five percent (1,300 of clinicians trained 2,000) estimated and labeling Packaging sufficient provided, materials of patients 80 percent for with essential Combined for medicines training prescribing with essential Combined for medicines training prescribing Revision of EML completed Target
First-year progress review for promoting rational medicine use rational promoting for review progress First-year Undergraduate training: Ensure that all that Ensure training: Undergraduate medicines in essential trained are students medicine use in basic and rational concept health curricula of time depends (amount category). on professional guidelines (STGs): treatment Standard standard and distribute Develop/revise least every years. manual at four treatment all clinicians Provide education: Continuing of in-service least one week with at training medicine use every years. on rational three least at that and labeling: Ensure Packaging medications receive of patients 80 percent or dispensing in clearly labeled containers envelopes. all that Ensure training: Undergraduate in communications trained are students methods in basic education and patient health curricula of time depends on (amount category). professional Include education: skills Continuing for in-service at education patient training workshops. Essential medicines list (EML): Revise Essential and least every EML at national distribute two years. election Prescribing Dispensing use Patient Total Objectives/activities S Table 48-3 Table 48 / Monitoring and evaluation 48.15
• Provide corrective feedback to staff or units that have Table 48-4 Types of evaluation not met expectations, but that should be able to take Type Project stage Purpose specific steps to improve their performance; many Needs Design • Assess current situation problems can be corrected through supervision and assessment • Develop program/project retraining. (initial plan • Reallocate resources or reassign staff to achieve a bet- analysis) • Acquire baseline for comparison ter fit between the task to be accomplished and the Formative Implementation • Focus on implementation resources or staff available. evaluation process • Make plans and targets more realistic based on actual (midterm • Assess progress toward goals experience. review) and objectives • Improve program/project • Request additional information to further define a implementation specific performance problem and the reasons for the Summative Follow-up • Assess program/project problem. evaluation outputs (final • Measure impact of program/ evaluation) project Country Study 48-5 illustrates how indicators can be used • Demonstrate program/ to improve performance through self-monitoring. project impact to donors Whatever actions are taken after a progress review, the • Recommend future actions effects of these actions should be considered at the next Source: Adapted from García-Núñez 1992. progress review.
from the program’s monitoring system reflect the 48.7 Evaluation actual situation? • Is the program efficient? Are the effects of the program Whereas monitoring focuses on program activities, evalua- being achieved at an acceptable cost compared with tion focuses primarily on assessing progress toward achiev- alternative approaches to providing the same services? ing goals by fulfilling program objectives—taking a step • Is the program sustainable? Financially and institu- back to look at the program as a whole. An evaluation is car- tionally, can the program continue with present levels ried out at a specific time and should have a clear purpose. of local inputs? If external financial and technical assis- tance is involved, can the program continue after it has Evaluation questions stopped? • Is the program having the intended impact? Does the Depending on the timing and purpose, evaluations may be program appear to be achieving or will it achieve its of three types (Table 48-4)— intended long-term health care benefits? • What future changes should be made? What recom- 1. Needs assessment (situation analysis) mendations can be made for program development, 2. Formative evaluation (midterm review) new plans, or project assistance? Are new goals or 3. Summative evaluation (final evaluation) objectives needed?
A needs assessment or situation analysis is meant to In practice, needs assessments, formative evaluations, and appraise the pharmaceutical system or essential medicines summative evaluations sometimes overlap. For example, the program and to identify areas of strength and weakness (see final evaluation for one project may also serve as the needs Chapter 36). The purpose is to design a project to address assessment for the next project. major weaknesses in the system. Formative and summative evaluations are more program- Conducting an evaluation matic and concerned with answering some or all of the following questions— Chapter 36 describes three approaches to assessment: self- assessment, limited assessment, and structured assess- • Is the program relevant? Are its goals and objectives ment. Structured assessment is the most comprehensive appropriate to the present circumstances of the coun- but requires the most resources. The choice of assess- try and the pharmaceutical system? ment approach depends on available financial and human • Is the program effective? Is it achieving satisfactory resources, timing, sponsorship, and intended uses of the progress toward its stated goals and objectives? What results. are the reasons for success or failure? Health program and health project evaluations are • Are monitoring results representative? Do the results sometimes limited assessments, consisting primarily of 48.16 Information management
Country Study 48-5 Use of indicators for self-monitoring in rural Java
The twenty-nine health centers and 109 subcenters of from the four subcenters under Widoyo Health Center Gunungkidul district in Java, Indonesia, suffer from peri- are compared with those from the previous month (see odic shortages of medicines, as do other health facilities. the accompanying figure). Using this form, health center These shortages are caused by limited resources; increas- staff can easily determine whether each indicator has ing patient demand; and overprescribing of injections, increased or decreased in each facility. antibiotics, and other medicines. To address this prob- The continuity of the self-monitoring process is sustained lem, the district health team in Gunungkidul undertook by— a series of activities to improve and control medicine use. The team first surveyed medicine use with three prescrib- • Weekly staff meetings of the district team at which ing indicators. Results showed extensive polypharmacy results are discussed (4.2 medicines per case), a high percentage of patients • Monthly district-level meetings for the heads of all receiving antibiotics (63 percent), and a very high per- health centers centage receiving injections (76 percent). The team • Regular feedback and occasional supervisory visits explored these issues in randomly selected health centers to health centers by members of the district team using in-depth interviews, observation, focus group dis- After two years, an evaluation showed substantial cussions, and questionnaires (see Chapter 28). From this improvements in all three indicators. Despite these information, a self-monitoring system was developed, changes in practice, the number of attendances at health pilot-tested, and implemented in all health centers in the centers had remained constant. Interviews with health district. workers showed improved attitudes toward the use of Each health center completes a monitoring form each standard treatments, willingness to improve skills, and month, based on a survey of thirty cases in each health increased communication among health workers and the center and subcenter. Results are discussed locally and district team. forwarded to the District Health Office for review. Data Source: Sunartono and Darminto 1995.
Local area monitoring form
Health center Widoyo Month/Year June 2010 Number of cases sampled 30 Medicines per case Injections
7 7 100%100% 6 6 80%80% 5 5 60% 4 4 60%
3 3 40%40% 2 2 20%20% 1 1 0 0 0%0% Current month 3.1 3.0 3.3 3.1 2.9 Current month 36.7 24.8 40.0 10.0 33.4 Previous month 3.0 2.8 3.5 3.2 3.0 Previous month 33.0 29.0 40.0 16.7 35.1 > > > > > > > Trend > > Trend — Subcenter I II III IV Avg. Subcenter I II III IV Avg. 48 / Monitoring and evaluation 48.17 interviews, secondary analysis of existing monitoring and The budget should be prepared early in the planning other data, and review of available documents. Increasingly, process. A template presented in Table 48-5 can be used to however, evaluations are expected to contain performance develop a budget. It should be updated as additional infor- data indicating what services and outcomes are resulting mation becomes available, such as personnel daily rates and from the program. The starting point for an evaluation the cost of translators, if needed. Some key considerations should be the data collected through the monitoring sys- for the budget include— tem, but field surveys using standard pharmaceutical sec- tor assessment indicators (WHO 2007) will contribute to a • Team member time much more objective, credible, and useful evaluation. –– Planning time—technical lead and administrative/ logistics support Evaluation methods and tools –– Team member time—preparatory, fieldwork, and report preparation An evaluation is much like a research project. The usual • Travel costs (as needed) considerations of research design apply, including defining –– Airfare the scope and questions for the evaluation, choosing evalua- –– Per diem tion methods, developing and testing data collection instru- –– Visa costs ments, managing data collection, collating and analyzing –– Telecommunications costs (phone/Internet access) data, interpreting results, and presenting the findings. • Contracted services (as needed) Methods of obtaining information include document –– Local consultant review, key informant interviews, data collection from exist- –– Translator(s) ing records, and prospective surveys. These methods and –– Driver(s) and car(s) the steps involved in a structured assessment are discussed –– Conference room facilities for the stakeholder in detail in Chapter 36. workshop (room charge, food costs, and equipment Knowledge, skill, and experience are required to design rental) and execute a credible, comprehensive evaluation. See • Miscellaneous charges References and Further Readings for several sources that • Photocopies—reference materials, reports, etc. provide practical guidance on evaluation methods. • Postage (mailing documents prior to visit, for example) Who should evaluate?
When individuals involved in a program carry out an evalu- 48.8 Some common pitfalls in monitoring and ation, they have the advantage of understanding its aims and evaluation design; however, such evaluations can be subjective and may miss important lessons. Therefore, outsiders are frequently Although monitoring and evaluation are two distinct activi- asked to perform evaluations. Because they are disinter- ties, they share certain common pitfalls— ested, outsiders may be more objective and may bring fresh ideas. Outside evaluators might come from local universi- Failure to identify the basic questions: All monitoring and ties, local nongovernmental organizations, international evaluation activities should start with a clear statement organizations, or organizations in other countries. of the questions they intend to answer. Without this, the Combined insider and outsider evaluations have many information gathering has no focus. advantages. The outsider, being ignorant of the local situ- Overambitiousness: Collecting too much information is ation, can ask the difficult questions, and the insider often perhaps the most common failing. A basic rule is that knows whether the true answer is being given. Having the more information means greater expense, less accuracy, insider view the program or project through the eyes of the more time spent managing the data, less time spent inter- outsider is also useful. The outsider benefits because the preting the data and providing feedback, and less time for insider knows where the answers are to be found. using the data. Complexity: Monitoring systems should be as practical and Resources required streamlined as possible. Cumbersome systems have often been designed from the top down, with insufficient test- Evaluation is an activity that goes beyond routine monitor- ing and input from staff involved in generating and using ing and requires staff, time, and funds to be earmarked for it. monitoring information. In the case of a specific program, funds for evaluation should Lack of integration with planning and implementation: be built into the original proposal. Careful costing should be Monitoring and evaluation activities should follow performed, taking into account all proposed activities. directly from program plans, link closely with ongoing 48.18 Information management Total ( US D) Total S ubtotal S ubtotal S ubtotal S ubtotal ( S um of subtotals) Quantity # days # days # days # days # days # fares at that rate that at # fares # fares at that rate that at # fares # fares at that rate that at # fares # days # days # days # # # # days # people # days # days # days U nit /day /day /day /day /day /trip /trip /trip /days /days /days /trip /trip /trip /day /person /day /day /day ate ( US D) ate R Title Title Title Title Title Destination Destination Destination Destination Destination Destination Destination Stakeholder workshop Stakeholder Stakeholder workshop Stakeholder Stakeholder workshop Stakeholder Sample budget template assessment . S Labor Name Name Name Name Name ubtotal U S ubtotal – airfare Travel Travel – airfare Travel Travel – airfare Travel Per diem Per Per diem Per Per diem Per Other – local travel costs Other – visa costs Other – misc. costs travel S ubtotal room Conference Coffee service Coffee Audiovisual equipment Audiovisual Driver and car Driver Translators subcontracts S ubtotal Postage Communications Other other S ubtotal Line Item people as needed) as many Labor (add lines for Travel ubcontracts/outside services S ubcontracts/outside Other costs budget assessment Total Table 48-5 Table 48 / Monitoring and evaluation 48.19
implementation activities, and lead logically into the next for Enhancing Access to Medicines Program. Arlington, Va.: round of planning. Management Sciences for Health.
assessment guide
Monitoring methods and systems Do they include process, output, and impact indica- • Does a program or unit exist to monitor and evalu- tors? Are they both qualitative and quantitative? ate pharmaceutical services? Who is responsible for Have performance targets been set? monitoring and evaluation? • Have indicator pharmaceuticals been identified for • Is monitoring based on program objectives and indicators requiring medicine-specific informa- linked to specific activities? Are these objectives, tion? along with output targets and performance targets, Evaluation clearly communicated to concerned staff? • Which of the following methods are used as sources • Is progress toward achievement of program goals of monitoring information: supervisory visits, rou- and objectives periodically assessed through formal tine reporting, sentinel reporting, special studies? evaluation? • How often are data collected? How are they used? • Does the evaluation begin with a clear statement of • Are monitoring results regularly reviewed and its basic question? Is this question used as a focus for shared? How and with whom? Is feedback provided information gathering? to concerned staff? • Which of the following methods are used to obtain • What actions are taken when problems are detected information—review of existing monitoring reports, through monitoring and evaluation? additional data collection, document review, inter- views, field surveys using standard pharmaceutical Indicators assessment indicators? • Have performance indicators been established? If so, • How is the evaluation team determined? Does it how were they established, and who was involved? include insiders, outsiders, or both?
Sunartono, Y., and B. Darminto. 1995. From Research to Action: The Coordinators and Data Collectors. Geneva: WHO.