Contents Full Report Case Studies Summary + Findings Stakeholder Map View Alphabetically P Timeline View Geographically Process Diagram View by Discipline Evidence in Appendices Practice Print

MEXICO | ECONOMIC DEVELOPMENT Progresa | Oportunidades A pioneering national program incentiving school attendance and healthcare through conditional cash transfers. By Victor Cervantes and María del Mar Gutiérrez Contents Full Report Case Studies Page 02 of 27 Summary + Findings Stakeholder Map View Alphabetically P Timeline View Geographically Process Diagram View by Discipline Evidence in Appendices Practice Print

Progresa | Oportunidades Evidence in Practice Introduction 03 Table of Contents Part I: The Progresa Story 04 The Story of Progresa: The Program and its Objectives 04 Initial Design, Pilot Project + Poverty Measurement 05 Launching the Program 08 Implementing the Program 10 Evaluating the Program 10 The New Federal Administration: 2000-2006 13

Part II: Evidence in Practice: Key Themes + Insights 15 Evidence Definition, Creation, + Use 15 Definition of Evidence 15 Evidence Creation 15 Uses of Evidence 16 Incentive Alignment Among Actors 17 The Role of Timing in the Incorporation of Evidence into Practice 18 Need to Devote Exclusive Time and Resources to Learn About and Operationalize Evidence 18 Importance of Building Trust and Forging Relationships Among Stakeholders 19 The Role of Funders | Shielding the Program 20 Conclusion 20 Image Credits Cover: Bud Ellison Page 5: Enrique Saldivar Part III: Resources Page 7: Eli Duke Stakeholder Map 21 Page 9: Son of Groucho Timeline 22 Page 11: Darij + Ana Page 12: Doris Hausen Process Diagram 23 Page 18: Kashfi Halford Appendices 24

evidenceinpractice.yale.edu Contents Full Report Case Studies Page 03 of 27 Summary + Findings Stakeholder Map View Alphabetically P Timeline View Geographically Process Diagram View by Discipline Evidence in Appendices Practice Print

The Evidence In Practice directors. Second, we conducted a matched the complex relationship between “evidence” comparison of eight cases of development and “practice.” These representative research project at the Yale programs or interventions where rigorous stakeholder groups are: Researchers, School of Management, evidence was integrated with varying degrees Funders, Influencers 2, Intermediaries, funded by the William and of effectiveness. This case study is one of Policymakers, Implementers, and Flora Hewlett Foundation, the eight produced by the project. The third Beneficiaries3. was conducted from January component, conducted in parallel to the eight case studies, consisted of interviews Each of the cases thus contains a map of the See the relationship 2016 to January 2018 in with prototypical representatives of each of specific organizations (and individuals) that between the the stakeholder groups, or individuals who defined its evolution, their structural affiliation stakeholders in order to better understand this project in the the conditions under which could clearly describe the typical experience to a stakeholder category (in some cases, stakeholder map. of enacting a particular stakeholder role. organizations played more than one formal, rigorous evidence can be Our synthesis analysis is presented in the structural role), the informal roles that certain effectively integrated into public accompanying report.1 individual actors played, as well as the key policies and non-governmental relationships between these individuals and Stakeholder Characterization organization (NGO) practices organizations. Based on our research, we have found it in the field of international useful to think of the flow of evidence into development. policy and practice as an “ecosystem” in 1 Please see the appendex for a detailed description on Data and Methods. which a set of archetypical stakeholder The Evidence in Practice project followed a groups interact. This set of stakeholder 2 While some of our interviewees identified rigorous methodology comprised of three categories was described and reinforced “Influencers” (such as the media, the general public, lobbyists, and influential individuals) as playing an broad elements: First we conducted an initial by our interviewees throughout the project. important role in the evidence-to-practice eco-system, round of expert interviews with individuals While this is not a perfect description (e.g., this group did not play an explicit role in the narrative who have spent a significant portion of their some organizations fall within more than one of any of the case studies. So we have included the category here, though it does not appear in the professional lives attempting, researching, stakeholder group and individuals often shift stakeholder maps of the individual case studies. or promoting the integration of evidence into across stakeholder groups or play roles that development practice, including academics, effectively span categories), it can help frame 3 We use the term “beneficiaries” to indicate those whom a specific policy or program is intended to government officials, foundation program the conversation to identify the critical roles, help. Different analytic frameworks use various terms to officers, NGO practitioners, and think-tank incentives, and relationships that animate describe this group, including clients, users, recipients, etc. Contents Full Report Case Studies Page 04 of 27 Summary + Findings Stakeholder Map View Alphabetically P Timeline View Geographically Process Diagram View by Discipline Evidence in Appendices Practice Print

Part I: The Progresa Story In 1994, presidential elections were held in Mexico. That year, Mexico The Story of Progresa: The Program and its Objectives experienced its highest level of income inequality in the past 30 years.4 Shortly after the winner of the elections, Dr. Ernesto Zedillo, assumed office, The program’s main objective was to disrupt the intergenerational transmission of poverty a profound economic crisis struck the Mexican economy. The crisis had an by encouraging and supporting household especially large impact on the poor. The extreme poverty rate grew from investments in child education. In particular, 21.4 percent to 37.4 percent between 1994 and 1996.5 the program combined health, education, and nutrition elements into a single intervention, At that time, most poverty alleviation programs in which barriers that tend to inhibit the Dr. Ernesto Zedillo creating reinforcing effects between the three. consisted of food subsidies. In 1996, roughly integration of evidence into practice can be In essence, the program consisted of a cash two thirds of the subsidies were untargeted, overcome. First, it demonstrates the ways in transfer to eligible households conditioned generalized price subsidies and more than which embedding evidence into the fabric on continued and demonstrable school three fourths of the subsidies went to urban of a program from the beginning can align attendance by the family’s children, coupled areas, leaving rural areas—where extreme the (often conflicting) incentives of different with routine health and nutrition checkups for poverty was more prevalent—underserved.6 stakeholders and overcome issues related all household members. During the checkups, Food subsidies were expensive, did not to discordant decision-making timelines. nutritional supplements were provided for reach the poor, and were routinely abused as Second, it shows that developing deep children under the age of five and pregnant or tools of political manipulation—for example, relationships with, and engaging a diverse nursing women.7 The cash transfer was given exchanging subsidies for votes. The crisis group of stakeholders can be critical to to the female head of the household, seeking prompted the urgency of supporting the poor developing trust—necessary to implement a to improve the allocation of resources within more effectively within the added constraints rigorous evidence-based program. the family. of a reduced budget. A body of academic literature had been generated before 1995 advocating for a paradigm shift in anti-poverty policy. Building on this literature, Progresa 4 Esquivel, Gerardo, Nora Lustig, and John Scott, 6 Levy, Santiago. “Progress against poverty.” (“Prosperous”) marked a notable departure (2010). “A decade of falling inequality in Mexico: Washington DC: Brookings Institute (2006). from existing social policy in Mexico, market forces or state action?.” Declining inequality in developing a groundbreaking program based Latin America: A decade of progress. 7 Ibid. on targeted, conditional cash transfers. 5 Cortés, Fernando, and Rosa María Rubalcava, (2012) Progresa (later renamed Oportunidades, or El PROGRESA como Respuesta a la Crisis de 1994. “Opportunities”) illustrates several key ways International Policy Centre for Inclusive Growth, No. 169. Contents Full Report Case Studies Page 05 of 27 Summary + Findings Stakeholder Map View Alphabetically P Timeline View Geographically Process Diagram View by Discipline Evidence in Appendices Practice Print

Progresa marked a dramatic departure from trade, and development. In the early 90’s, he existing social policy in Mexico. It was the was hired as a consultant for the World Bank first program in Mexico to explicitly combine to evaluate anti-poverty programs in Mexico. the three elements of healthcare, nutrition, In his work, he questioned the effectiveness and education. Progresa was targeted to of traditional programs, such as the tortilla specific families according to their needs subsidy.10 In 1992, he joined the Mexican and characteristics, which were clearly government as the first President of the articulated in advance through clear rules that Federal Commission for Competition. Gómez were defined by rigorous statistical models. de León started his career at the Mexican That meant not only that the beneficiaries Central Bank, where he and Mr. Zedillo were likely to be those most in need of met. He obtained a Ph.D. in Demography the subsidy, but also that once the criteria and was a faculty member at El Colegio de were established, eligibility would not be Mexico, researching the intergenerational manipulated by policymakers, implementers, transmission of poverty before joining the or beneficiaries. This shielded the program Mexican government as the Chief Advisor from political abuse, as discussed below. to Zedillo at the Ministry of Budget and

While the program’s main objective was to Initial Design, Pilot Project, + Poverty limit the inter-generational transfer of poverty, Measurement 8 Scott et all., 2010 the intervention also alleviated current poverty. Initially, the larger component of After assuming Mexico’s Presidency, Mr. 9 In 1974, President Zedillo pursued a Master’s degree the cash transfer was calibrated to offset Zedillo9 appointed Dr. Santiago Levy to be the and a PhD at Yale University. He worked at the Mexican Deputy Minister of Finance, whose portfolio Central Bank and served in two Ministries (Finance the household opportunity costs of keeping and Education). children in school, according to their age and would include preparing the budget and gender (accordingly, the amount transferred approving funding for federal programs. Dr. 10 One of the programs started in 1986 as a varied by family size and composition). As the José Gómez de León was appointed to chair generalized subsidy in urban areas that gave the right the National Council of Population (CONAPO). to buy tortillas at a subsidized price. In 1990, Fidelist program evolved, the alleviation of current emerged, a new directed scheme that granted the poverty became an increasingly important At that time, Levy and Gómez de León were right to obtain a free kilo of tortilla every day. These objective. Based on its impact, the program public officials with considerable academic two programs coexisted until October of 1998, when has been the most effective instrument backgrounds. Levy had obtained a Ph. D. in the generalized subsidy was eliminated. In November of 2000 Fidelist was phased out. [Levy, Santiago; in Mexican social policy for redistributing Economics in 1980 and had been a faculty Evelyne Rodríguez. Sin Herencia de Pobreza. Banco income to the poor.8 member at Boston University. His research Interamericano de Desarrollo – Planeta. 2005. Pages focused on macroeconomics, international 119-121.] Contents Full Report Case Studies Page 06 of 27 Summary + Findings Stakeholder Map View Alphabetically P Timeline View Geographically Process Diagram View by Discipline Evidence in Appendices Practice Print

would allow the federal government to create “Within Progresa, we never thought a more rigorous measurement of household- level poverty to effectively target eligible about the development of the program recipients. The survey sampled communities where poverty was expected to structural aspects of the program be more prevalent according to the general marginalization indexes. The instrument, without always thinking of the named “ENCASEH” (Survey of Socioeconomic operational aspects as well.” Characteristics of the Households) sought to collect granular, household-level baseline RESEARCHER data on health, education, and consumption habits. These data were designed to allow for the precise targeting of an intervention and would become instrumental for the randomized controlled trial that later served as the initial pilot for the program. The team Programming. He later served as Director of problems, and benefits that different types of also gathered information on the school drop- the National Institute of Public Health. Gómez policies had shown, and different alternatives out rate by gender. The survey was designed de León was very close to Zedillo, as they for fighting poverty. to estimate the size of the subsidy required to were longtime collaborators. offset a family’s opportunity costs of keeping Gómez de León and his team had been children in school, a cost that increased with In late 1994, Zedillo, based on a set of recent researchers, and they collected additional a child’s age. Additionally, information on the publications that extensively and rigorously quantitative data to inform the design local supply of education and health services analyzed the shortcomings of existing of the program. In particular, a critical had to be gathered, as it was necessary to paradigms for poverty alleviation, asked initial challenge was identifying the target ensure local capacity to absorb the increased Gómez de León to design a program to fight population, as Mexico lacked an official demand for health services that would be poverty based on conditional cash transfers. measure of poverty, and existing data was generated by the program. Early on, Gómez de León and his team11 insufficient to target at the household proposed a rough design of the program, level. In 1995, using their previous work on based on the evidence available to them, marginalization indexes at the municipal including a rich body of literature. The team level, they created tools to estimate poverty paid particular attention to recent work by the at a more granular level. In particular, the 11 Mr. Gómez de León’s team included Rodolfo Tuiran, World Bank discussing the diverse qualities, team began to roll out a national survey that Daniel Hernández, and Monica Orozco. Contents Full Report Case Studies Page 07 of 27 Summary + Findings Stakeholder Map View Alphabetically P Timeline View Geographically Process Diagram View by Discipline Evidence in Appendices Practice Print

In parallel, the Deputy Ministry of Finance, One of the programs Mr. Levy, and his team12 had been working started in 1986 as a independently on the design of a similar generalized subsidy in urban areas that program, seeking to address the inter- provided the right generational transmission of poverty. An initial to buy tortillas at a design of the program­—one that captured the subsidized price. 1990 essence of what became the final design— saw the emergence of Fidelist—a new was presented in March of 1995. Levy and his directed scheme that team were strong proponents of the program, granted the right to had control over the budget, and understood obtain a free kilo the broader political and financial situation of tortilla every day. These two programs of the Federal government. To test the initial coexisted until program design, Levy’s team conducted a October of 1998, pilot of 31,000 households in the state of when the generalized subsidy was finally Campeche along with the Ministry of Social eliminated. Development during the first half of 1996. The pilot was designed to generate evidence to analyze the conditional cash transfer scheme. The name of the pilot was “Basic that a targeted cash transfer was much After two of his trusted collaborators Food Basket Program for the Wellbeing of the more efficient and effective than traditional independently developed similar program Family”. The pilot was externally evaluated untargeted subsidies. It also revealed that proposals, President Zedillo instructed by the Autonomous Technology Institute of building some sort of conditionality into the them to join efforts and became particularly Mexico (ITAM) and by the National Institute of program, as the program designers had long invested in the success of the unified Nutrition Salvador Zubirán. believed, was important to further amplify program, turning it into his flagship social positive effects by, for example, linking cash policy. The teams divided the next phase of The initial pilot showed that, as hypothesized transfers to useful behavioral changes within work: Levy and his team created technical by the program’s designers, directing the the household. Finally, the pilot showed that papers documenting the Campeche pilot to subsidies towards women enhanced the household members preferred cash to in-kind support the claims made; Gómez de León’s effects of the program, as it led to a more transfers and that they valued the healthcare judicious use of the transfers within the component.13 households and did not create feared 12 Mr. Levy’s team included Evelyn Rodriguez, Antonio Alvarado and Enrique Davila. secondary consequences, such as domestic violence. The pilot thus demonstrated 13 Levy, 2006 Contents Full Report Case Studies Page 08 of 27 Summary + Findings Stakeholder Map View Alphabetically P Timeline View Geographically Process Diagram View by Discipline Evidence in Appendices Practice Print

team continued collecting rigorous, granular as he was directly involved with Gómez de data to target households for the next Against government León and Levy reporting directly to him. iteration of the program. Several elements tradition, Zedillo The accompaniment, encouragement, and of the pilot in Campeche were modified in ownership by Zedillo was key. The leadership further refinements of the program, including decided to wait for a that Gómez de León, Levy, and Zedillo showed, the addition of a conditionality on child thorough and tested as well as their background as researchers education, which was not present in the pilot, who had become policymakers, proved to be and providing the transfers in cash instead design, instead of very powerful. of doing it through electronic cards. The launching—with electronic cards were dropped as the media Against government tradition, Zedillo decided and program detractors had criticized them, typical fanfare—an to wait for a thorough and tested design, making them politically toxic. The pilot had ambitious program at instead of launching—with typical fanfare— enabled the team to convince the President an ambitious program at the beginning of and most of the Cabinet that the program was the beginning of his his administration. In August of 1996, two both viable and effective, which supported administration. years into an administration under enormous them in the face of stiff political opposition: political strain from the 1995 crisis, but not based on longstanding tradition, the Ministry until the results of the pilot project had been of Social Development opposed targeted presented, the President authorized an or conditioned cash transfers and strongly expanded intervention named Program of favored generalized subsidies. Nutrition, Health and Education (PASE).

Levy was respected among academics, egos come in the way of the program. Rather, but he also demonstrated managerial and contemporaries describe them as recognizing Launching the Program political skills. In parallel, Gómez de León each other’s complementary skills and Before August of 1996, the participation of was in charge of demographic information establishing a productive partnership. other government agencies in the program and policy, and had the skills and a team Gómez de León is regarded as leading had been minimal. At that point, given capable of gathering, creating and analyzing the implementation and having a great the decision to incorporate a behavioral a wide range of data. Both Levy and Gómez deal of influence on program design. Mr. conditionality into the program—school de León were technocrats but showed Levy, also influential in program design, is attendance and visits to physicians—the sensitivity towards political and operational regarded as an agile leader who oversaw the involvement of the Ministry of Health and constraints. They were technically strong and necessary political negotiations. President the Ministry of Education became crucial. committed to sound policymaking. They both Zedillo’s leadership was also a necessary While in the pilot, for example, the health had strong personalities, but did not let their condition for the success of the program, interventions were monitored by a group of Contents Full Report Case Studies Page 09 of 27 Summary + Findings Stakeholder Map View Alphabetically P Timeline View Geographically Process Diagram View by Discipline Evidence in Appendices Practice Print

doctors hired specifically for this purpose, a government stakeholders showed the will larger roll-out would require using the public to collaborate and commit to the program health system. Given its federal structure, as a result of the leadership, support, and health and education services in Mexico are pressure from President Zedillo. decentralized and managed by each state government. Thus, certifying the fulfillment During that time, weekly discussions were of the behavioral commitments on education held among the inter-ministry group14 on and healthcare by the beneficiaries required the operating and technical aspects of the coordination with state governments, program. Questions ranged from the amount including negotiations on how to launch of the transfers to the specific formula of child and monitor operations in each state. This nutritional supplements. Gómez de León was not a trivial issue, as the new program and his team built a database with the geo- design identified target households ex-ante location of the communities and households based on technical factors, removing the intended for the program in order to draw the local government’s long-held control over the routes to go to the field and disburse the cash. distribution of government subsidies. Also, negotiations with the local authorities to help with the logistics started. This level Levy took the leadership in engaging of inter-governmental coordination proved these two key ministries as well as state instrumental to the success of the program. governments. Zedillo provided top-level Mr. Levy and his team played a central role in support (and often pressure) to create conducting negotiations and securing federal political goodwill among other government funding for the program, most of which came stakeholders. An inter-ministry group, with the from shifting funds previously used for the Chair of CONAPO and the Deputy Ministers generalized subsidies. The pushback from of Finance, Education, and Health, was the proponents of generalized subsidies was created to refine the operational aspects relentless, partly because this had been the of the program. Zedillo´s involvement prevailing paradigm, but also because of the was particularly important to enable the room this approach provided for political coordination between ministries and with manipulation. Such opposition only increased state governments. Although the path to the team’s commitment to rigorously and 14 Group with the Deputy Ministers of Finance, implementation was not without resistance, transparently evaluate the new program’s impact. Education, and Health, and the Chair of CONAPO. Contents Full Report Case Studies Page 10 of 27 Summary + Findings Stakeholder Map View Alphabetically P Timeline View Geographically Process Diagram View by Discipline Evidence in Appendices Practice Print

The program was ready to launch in 1997, Gómez de León and his team moved in the program, its neighbors would also December of 1996— and its funding was from CONAPO to the National Coordination be in the program. That eased suspicions approved in the Federal budget of 1997. of Progresa. At that time, Progresa’s National that some households would be arbitrarily However, Mr. Zedillo’s administration decided Coordination was a young and small (or politically) excluded from the program. It against launching before the midterm organization of no more than 30 people. also simplified program operations. While elections to be held in July of 1997, to avoid The team was led by Gómez de León, and choosing the community as the unit instead of suspicion of political use of the program and received support from Levy and his team. the household reduced the inference power again departing from traditional electoral of the RCT, the program was big enough to norms in Mexico. The name of the program The roll-out of the program, which had to cover a large number of communities and was changed to Program of Education, Health, be gradual given resource and operational draw statistically reliable conclusions. Second, and Nutrition (Progresa). It was the first social constraints, was designed as a randomized all the households in the control group would program to carefully and transparently define control trial (RCT) to create rigorous evidence eventually added be to the program. More operating rules and principles. The rules for subsequent program refinement and importantly, a large number of household were intended to increase transparency and growth. Out of the group of potential would have to be excluded in the beginning, shield the program from political interference. beneficiary communities, a total of 300,000 even in the absence of the RCT, for purely (Based on this program’s experience, explicit households were selected as the treatment operational and financial considerations. rules of operation subsequently became a group and the rest were assigned to the Random assignment to the treatment or the standard practice for all social programs in control group, producing an experimental control groups was, in fact, an arguably fairer Mexico.) design that allowed for the rigorous testing decision than one made based on political or of several hypotheses about the program´s operational considerations. Implementing the Program effects. Zedillo delegated program implementation to Gómez de León and created the This experimental approach had its critics. Evaluating the Program National Coordination of Progresa as a There was a vocal group, including some in During 1997, Levy and Gómez de León decentralized body inside the Ministry the press, who thought it was outrageous had explored alternatives for a rigorous, of Social Development (which was the for the government to conduct experiments transparent, and external evaluation of ministry that had traditionally managed the with social programs, as they thought it Progresa. Although there were costs generalized subsidies and thus was the most was unethical to arbitrarily deny access associated with having a robust evaluation, ardent opponent of Progresa). To maintain to some households. Two factors made the benefits far outweighed them. Progresa appropriate inter-ministry coordination, an the randomized control trial socially and was a departure from traditional policy oversight board was created, with permanent politically viable. First, the randomization was in several respects. First, it was targeted seats for the Ministers of Health, Education, made with the community, not the household, and based on objective, verifiable rules Finance, and Social Development. In June of as the treatment unit. If a household was of operation. Second, the program was Contents Full Report Case Studies Page 11 of 27 Summary + Findings Stakeholder Map View Alphabetically P Timeline View Geographically Process Diagram View by Discipline Evidence in Appendices Practice Print

explicitly forbidden from being used for political objectives, and beneficiaries were systematically informed of this. Third, the program explicitly forbade adding beneficiaries during political campaign seasons. Finally, the program was organizationally shielded from political influence: program officers were, by mandate, selected based solely on their technical merit and were non-partisan. Furthermore, no career politician was allowed to participate. As a result, there was enormous political pressure to change or scrap the program. Only evidence of its effectiveness, validated by a legitimate, external institution could provide political cover. The evaluation would thus have to be performed by an institution that was viewed as independent from the Mexican government or other political actors in Mexico, and that had reputable evaluation experience.

The program designers did not want a visible organization, such as the Inter-American economists at the World Bank encouraged it would be desirable to have some senior Development Bank (IDB) or the World Bank, Evelyn Rodriguez, who worked with Levy, to participants beyond IFPRI staff to give it to be involved in the initial evaluation in consider IFPRI during a visit to Washington greater visibility and credibility. order to avoid suspicions of foreign financial D.C. in 1997. Levy and Daniel Hernández institutions influencing the program. Nora Franco, who was part of Gómez de León’s Gómez de León asked Lustig, who had been Lustig, who was then a researcher at the team, were aware of the work by IFPRI on the recently appointed at a senior position at the Brookings Institution, had mentioned the effect of nutrition programs on poverty. Levy IDB, to organize a small workshop, and gave International Food Policy Research Institute and Gómez de León decided to hire IFPRI to her a set of names of people whom the team (IFPRI) to Levy in 1995. Additionally, some conduct the evaluation, but they thought that wanted to get involved. The workshop took Contents Full Report Case Studies Page 12 of 27 Summary + Findings Stakeholder Map View Alphabetically P Timeline View Geographically Process Diagram View by Discipline Evidence in Appendices Practice Print

place in December of 1997 at the IDB, with The original program provided cash transfers IFPRI representatives and some potential to mothers who could demonstrate continued senior external advisors, including Jere enrollment of their children in school up to 9th Behrman from University of Pennsylvania, grade. The evaluation, in particular the work Paul Schultz from Yale, and James Heckman by Paul Schultz, found that the conditional from University of Chicago (who could not cash transfer had the strongest impact on attend but invited Petra Todd instead). Paul the enrollment in secondary school (grades Gertler, from UC Berkeley, later became part 7 to 9). Based on this, the leadership of the of the team, as he learned from Hernández program decided to extend program eligibility about the project and asked to be involved. for households with children enrolled up to As the program was to be expanded very 12th grade. In contrast, the impact evaluation rapidly from the initial experimental rollout, showed that the nutritional supplements data collection and evaluation was integrated had a negligible impact on anemia, due to a into program operations from the beginning. mistake in the iron formula that caused low The IFPRI team, the four external advisors, absorption. Some operational evaluations and the Progresa staff worked out the details showed that this was compounded by a of designing the questionnaires used in the lack of household education and monitoring, evaluation. The contract with IFPRI lasted for which translated to low intake of the three years, after which it was thought that nutritional supplement, and that appropriate the credibility of the evaluation would have “consumption declined as indicators of poverty been established and a strong domestic increased.”16 Even though the evaluations institution could continue the evaluation. detected the problem as early as 2001, the There was a contract with the National iron formula was not changed until 2005. This Institute of Public Health15 (INSP) to continue was partly because, while the supplements the evaluation. did not have a measurable impact, they certainly posed no harm and were popular The IFPRI evaluation found that Progresa among beneficiaries, so they helped improve had positive effects on the majority of the compliance with other program conditions. intermediate targets. There was no evidence of the effects on cognitive skills, but what was clear was that the behavioral conditions in the 15 Mr. Gómez de León was the director of the INSP before being invited to lead the CONAPO. program had significantly impacted household habits around nutrition and child education. 16 Levy, 2006, p. 55 Contents Full Report Case Studies Page 13 of 27 Summary + Findings Stakeholder Map View Alphabetically P Timeline View Geographically Process Diagram View by Discipline Evidence in Appendices Practice Print

In addition, there were other, more critical, 2002. President Fox decided to increase the adjustments to the program that were As the program coverage from 3 million families to 5 million prioritized, such as expanding the program to was to be in the first years of his administration. That urban populations (discussed below). required extending the coverage to urban expanded areas, as almost total coverage of the poor in After launching, the National Coordination of very rapidly rural areas had been already achieved. Progresa committed to a fast scale-up, with the support of Levy and Zedillo. Beneficiary from the initial The first director of Progresa to be appointed Vicente Fox households increased five-fold by the end experimental by the new administration, in January of 2001, of 1998 and by the end of the year 2000 was Vicente Arredondo. He was replaced the program reached almost 2.5 million rollout, data by Rogelio Gómez Hermosillo in August of households. collection and 2001. Mr. Gómez Hermosillo came from the NGO sector and did not have experience The New Federal Administration: evaluation was with conditional cash transfers. However, he 2000-2006 integrated maintained key people in operations who had In April of 2000, José Gómez de León passed worked from the beginning and reappointed away and Daniel Hernández assumed the into program others who had left during Mr. Arredondo’s leadership of the National Coordination by operations from brief tenure. He later became an expert on presidential appointment. In July of 2000, Mr. conditional cash transfer programs. The Vicente Fox won the Presidential election. the beginning. program was rigorously audited by the Fox That was the first time in more than seven administration under the suspicion that the decades that a political party other than the program could have been misused or that Institutional Revolutionary Party (PRI) had during that process. The President Elect money could have been diverted from its won a presidential election. asked specifically about the results and rigor intended purposes. This placed a burden of the evaluations. Because of the program’s on the operational teams and created stress Levy shared the program’s impact evidence performance evidence and IDB’s and the for the public officials who had been in the with Mr. Fox as President Elect. Also, World Bank´s positive opinions of the program, program during the previous presidential term. Hernández talked to Fox and Vázquez President Fox decided to maintain the The audit found that no misuse had taken Mota, who was appointed as the Social program. Under the new administration, the place, which strengthened the legitimacy Development Secretary, about keeping and program’s core design and components were of the program and the Fox administration’s strengthening the program. The body of maintained, with some adjustments, including respect for what had been done. evidence that had been produced was key changing its name to “Oportunidades” in Contents Full Report Case Studies Page 14 of 27 Summary + Findings Stakeholder Map View Alphabetically P Timeline View Geographically Process Diagram View by Discipline Evidence in Appendices Practice Print

The National Coordination faced the Oportunidades experience with rigorous budget, the loan was large relative to the IDB’s challenge to develop the technical tools evaluation, the National Council of Evaluation lending history. Rather, the partnership with to increase the scale and scope of the (CONEVAL) was created with the mandate to the IDB, which included technical assistance program. During the initial years of Fox’s evaluate the government’s social programs. and monitoring, as well as contractual tenure, Progresa/Oportunidades had clear The systems for monitoring and evaluation obligations that contained automatic operational rules and was well established fostered by CONEVAL took important lessons repayment triggers following alterations to in the budgeting process, so the Fox from Progresa, and in fact all the members the program’s operating rules, provided firm administration simply kept the program of CONEVAL’s first advisory board had been technical and political guardrails. running based on its previous design and evaluators of that program. specifications. In order to scale the program, however, Mr. Gómez Hermosillo and his team In 2001, the Office of Public Debt at the had to redesign the targeting systems and Ministry of Finance brought a team from the to recalibrate the parameters of the cash IDB to explore giving a loan to Mexico, tied to transfer, as the mechanisms designed for the maintenance of Progresa. The program rural areas were not directly applicable to was fully funded through the federal budget, urban areas. They spent 2001 and 2002 on so it did not require additional resources. this endeavor, and in 2003, the strategy for Rather, the loan was negotiated as a strategic the urban expansion was finalized. Unlike protection so the program would continue the rollout in rural communities, there was under its core operating rules and guardrails. no pilot or RCT for the urban expansion. The In fact, the loan agreement gave the federal operation of the transfers, control systems, government ample discretion on the use planning systems, communication with of the loan proceeds. Changes in core families, and co-responsibility review systems elements of the Progresa program, however, were improved, but important opportunities to would trigger the immediate payment of strengthen evaluation were missed, including the principal plus some termination fees. designing an evaluation for the urban rollout. The goal of the loan, then, was to shield the At the end of Mr. Fox’s term in 2006, the program from political cycles or temptations program reached 5 million households. at manipulation. In 2002, the IDB granted a US $1 billion loan to the Mexican government, During Ms. Vázquez Mota’s tenure as which was renewed in 2005 for US $1.2 Secretary of Social Development under billion. While this was not sizable compared President Fox, and based on the Progresa/ to the program’s, let alone to Mexico’s, overall Contents Full Report Case Studies Page 15 of 27 Summary + Findings Stakeholder Map View Alphabetically P Timeline View Geographically Process Diagram View by Discipline Evidence in Appendices Practice Print

Part II: Evidence in Practice: Key Themes + Insights

This section discusses the Evidence in Practice themes as they pertain to Council of Evaluation (CONEVAL), which Progresa | Oportunidades and summarizes key insights and implications now brings in rigorous methodologies to the for thinking about the translation of evidence to policy and practice more evaluation of all social programs in Mexico, illustrating how the effective integration of generally. evidence into practice can translate to new structures for routinely integrating evidence Evidence Definition, Creation, + Use importance and use of rigorous evidence into practice in other areas. There are varying definitions and both to make programmatic decisions and understandings of what constitutes to provide strong political support, rooted in Evidence Creation “evidence,” dependent especially on the the randomized control trial (RCT) that was perspectives of each stakeholder group. For embedded in the rollout of the program. The Progresa/Oportunidades was a departure example, the framing, language, and limited conceptualization of evidence, the definition from the traditional design of social policy accessibility of academic evidence can of program objectives, and the philosophy in Mexico. The case narrative shows render it less useful to other stakeholders. for policy design brought in by Progresa the important role evidence plays: from These diverging views of evidence create were diametrically opposed to the traditional committing to generating it, to collecting barriers across stakeholder groups, as what incentives and philosophies for policymaking and using evidence; while also being flexible constitutes valid evidence for each exists in in Mexico. While Progresa was always defined enough to take the risk of a new idea for different realms and in different forms that are by the demonstrable impact it could have which the evidence was not all in hand. The challenging to reconcile. on its beneficiaries, policymaking in Mexico conditional cash transfer model had not been had traditionally focused solely on political empirically tested at the time. What had been dimensions and implications. Progresa, demonstrated, however, was that generalized Definition of Evidence therefore, brought in a radically different and in-kind subsidies—which had dictated As described in the case study narrative, definition both of what category of evidence the paradigm of social policy in Mexico up Progresa was conceived based on the most was most relevant for policymaking and until that moment—were grossly ineffective. rigorous academic evidence available at the what actually constituted reliable, actionable The decision to launch the program came time. Designers centered their work on a rich evidence. before there was a randomized controlled body of literature, paying particular attention trial to prove the effectiveness of the model. to the work of the World Bank on social This new approach to integrating evidence The team, partly because of its background in policy and on the discussion of in-kind vs. into policymaking was so influential that it academia and also because of the humbling monetary transfers. As Progresa developed, not only informed the evolution of Progresa wave of evidence that had just been produced the continued common thread was the but also led to the creation of the National regarding existing social programs, knew Contents Full Report Case Studies Page 16 of 27 Summary + Findings Stakeholder Map View Alphabetically P Timeline View Geographically Process Diagram View by Discipline Evidence in Appendices Practice Print

the importance of designing policy based on evidence, and the importance of generating The Progresa team made it a priority new evidence both to evaluate progress and to inform future policy. The Progresa team not only to research the best available thus made it a priority not only to research the best available evidence, but also to generate evidence, but also to generate new new data and evidence when necessary to inform their policy decisions. data and evidence when necessary to inform their policy decisions. The decision to pilot and evaluate before fully launching the program was crucial, allowing policymakers to use the data to understand the impacts of the program and the components that needed to be improved. Also, precisely because of the rigor and transparency with which they were generated, these data provided critical political strength to carry on with the rollout and scale-up evidence created by the program. It is also transfer policy. Ironically, this lesson was often of the program against enormous political worth considering that, throughout the lost, even on those with experience with the resistance. The information produced design and rollout, the program retained program who decided to scale it into urban was essential not only for conceptual significant flexibility to incorporate lessons areas (where it proved less successful). policymaking, but for successfully translating from any new findings that emerged from the theory of change into a feasible and program operations. Evidence and practice Uses of Evidence actionable large-scale program. where thus integrated fully throughout The paradigm shift sought by Progresa design and ongoing implementation—with implied not only a different conceptualization Another key factor the case study illuminates a natural evolution on both the scale and of social policy, but also and more importantly, is the importance of not limiting the use of type of decisions that were made and the the loss of a powerful and effective tool evidence to the design phase, but integrating most relevant evidence to be generated. Put for political mobilization. Accordingly, one the generation and use of evidence into differently, the best lesson to learn from of the key motivations to document and every phase of the program so its quality Progresa as a policy success is the process produce evidence from the very beginning and effectiveness can be assessed and that led to its creation much more than the was to provide powerful elements to face adjusted based on the findings of the ongoing specific characteristics of its conditional cash and overcome relentless political pushback Contents Full Report Case Studies Page 17 of 27 Summary + Findings Stakeholder Map View Alphabetically P Timeline View Geographically Process Diagram View by Discipline Evidence in Appendices Practice Print

from different actors, including from the Bank published a volume on conditional As researchers who had become proponents of generalized subsidies, cash transfers, counting around 30 programs policymakers, President Zedillo and the members of Congress, and influential similar to Progresa around the world. team of designers and implementers shared members of the governing political party. Informing policymaking was a two-way the vision of integrating rigorous evidence street for the program and its implementers, into policy and practice. Although they had Another key lesson from Progresa is how where the delegations the World Bank sent all been researchers, they also had plenty a program that generates and adapts to helped the Mexican team learn the strengths of experience as public officials, and were rigorous ongoing evidence can increase its and weaknesses of the program through well aware of the political landscape and See the relationship own probability of survival across political the questions and discussions with the operational realities. They did not produce between the cycles. The year 2000 marked a watershed international visitors. original research to inform the program, but stakeholders in moment in Mexico’s democracy. For rather used the best available research to this project in the the first time in 70 years the PRI lost the Incentive Alignment Among Actors design the first pilot of the program, and then stakeholder map. presidential elections to the National Action Throughout the ecosystem, within and across generated the evidence needed to evaluate Party (PAN). As mentioned above, designers stakeholder groups, formal and informal its implementation and design the program and implementers of Progresa shared the incentive structures are frequently not at scale. No decision was made without program’s impact evidence with Vicente Fox conducive – and are often in contradiction— gathering data and documenting it. Second, as President Elect and Vazquez Mota, who to the integration of evidence into practice. the idea of a robust external evaluation was was appointed as the Social Development Typically, organizational incentives are defined of crucial importance. The program not Secretary. The program’s positive results, around an insular view of the organization only generated quantitative and qualitative the rigor of the evaluations, the fact that (e.g., academics publish in academic journals, information, but also made it public. Levy the evaluations were performed by external policymakers must exercise their budgets and Gómez the León were pragmatic about trustworthy institutions, and the large body according to program and budgetary rules, the quality of the information, especially the of evidence that was produced and widely NGOs must operationalize their programs need for rigor and objectivity. This resulted disseminated were key to what seemed as stated in their budgets and proposals in spending time and resources finding the impossible: maintaining the flagship social to funders). Usually, these organizational best available external partners, as well as program of the previous administration and its incentives have no mandate or room for the engraining evidence generation deep into now maligned party. explicit search of external evidence, much less the program’s structural design. Finally, the for the generation of internal evidence that commitment to ongoing learning within the The evidence generated by Progresa also would then lead to continuous adaptation of design and implementation of Progresa can shaped policy making in other countries programs and policies as new learning emerges. be seen throughout the decisions made and around the world that adapted the conditional the organizational structure for operating cash transfer program. In 2009, the World the program. Learning from the evidence Contents Full Report Case Studies Page 18 of 27 Summary + Findings Stakeholder Map View Alphabetically P Timeline View Geographically Process Diagram View by Discipline Evidence in Appendices Practice Print

generated was central to Progresa, providing an operational structure with the capacity to make decisions and incorporate emerging evidence so the program could have more impact.

The Role of Timing in the Incorporation of Evidence into Practice The different and often discordant timeframes within which researchers, policymakers, and implementers operate often hobble efforts to coordinate, let alone collaborate, on evidence- informed approaches. Electoral cycles and political windows differ from NGO funding cycles and from academic publishing rhythms. Yet each actor is bound by the timeframes of her formal stakeholder group.

The three main actors behind the design of the program (Zedillo, Levy and Gómez political trade-off in the integration of While Progresa ultimately became Mexico’s de León) had a good understanding of the evidence as demonstrated by Progresa. From most successful social policy, it is important time horizons of research. Also, their public a traditional policymaking perspective, the not to lose sight of the enormous trade-offs policy experience provided them with the rollout of Progresa was unusually slow. The involved in its design. knowledge of the “traditional” timeframe of program was officially launched two years public policy design and implementation both into the Zedillo administration—and even Need to Devote Exclusive Time in Mexico and in global best practices. Due to then the program’s initial scale was modest and Resources to Learn About and this shared experience, they were able to go by the standards of its predecessors. This Operationalize Evidence against the political pressure of quick timing was an administration under enormous Few organizations provide incentives or carve to maximize public “splash” with constituents, political pressure to have a visible policy out explicit time for managers to devote to sacrificing the launch of the program at accomplishment in the wake of the 1995 learning about emerging evidence in their the beginning Zedillo’s presidential tenure, financial crisis. The design team nonetheless field of endeavor, and even fewer have staff and develop and test a solid program. Put chose a gradual, evidence-informed design explicitly devoted to learning about evidence differently, there is a clear and substantial and rollout of its flagship social program. and translating it into forms relevant for the organization. Even organizations with strong Contents Full Report Case Studies Page 19 of 27 Summary + Findings Stakeholder Map View Alphabetically P Timeline View Geographically Process Diagram View by Discipline Evidence in Appendices Practice Print

monitoring and evaluation departments often tradeoffs associated with the commitment do not transform the operational data into The designers to a robust evaluation, the benefits far formats that could be widely used within the of Progresa outweighed these costs. Every evaluation organization, or beyond, to expand actors’ was done by reputable external actors that understanding about what has been learned incorporated a ensured the objectivity, transparency and rigor from past or existing programs. Data is thus long-term focus of the program. This delayed the launch of used to evaluate retrospective operations, but the program for the Zedillo administration— not to improve the prospective design of new on demonstrable much in need of an early policy success—but initiatives. Discovering, incorporating and impacts for program created a powerful political platform, built translating evidence requires time, energy on legitimate evidence, that armored the and funding. beneficiaries, which program against opposition throughout translated to an its incorporation and allowed it to survive President Zedillo, Santiago Levy and José through four different presidential terms and Gómez de León were well positioned to understanding that two transitions between opposing political become champions of evidence uptake. gathering, analyzing, parties. Progresa helps us understand five successful paths to guide evidence uptake. These and evaluating Finally, weekly discussions were held among paths are: 1) recognizing the crucial value its impact was the key players, in this case an inter-ministry of gathering, analyzing, and integrating group that met to discuss the operation and emerging evidence, 2) commitment of not a luxury, but a technical aspects of the program, which resources to constant learning, 3) structural necessity. illustrates the strong commitment to fostering flexibility so a program can adapt to learning, ongoing collaboration among the key players. 4) transparency, objectivity, and rigor in Progresa integrated school attendance evidence generation, including through narrative we observe a constant commitment and visits to physicians, which meant the external partnerships, and 5) building to ongoing learning, from designing participation of other government agencies decision structures for ongoing collaboration better questionnaires, to obtaining better like the Ministry of Health and the Ministry and negotiation among the key players. information about the needs of intended of Education. Cooperation was crucial to the First, the designers of Progresa incorporated beneficiaries through the pilot, to constantly success of the program and fostering open a long-term focus on demonstrable impacts evaluating results to learn and modify the dialogue and inclusion could make or break it. for program beneficiaries, which translated program and its theory of change based on to an understanding that gathering, analyzing, emerging evidence. Third, the realization that, and evaluating its impact was not a luxury, although there were costs and enormous but a necessity. Second, throughout the case Contents Full Report Case Studies Page 20 of 27 Summary + Findings Stakeholder Map View Alphabetically P Timeline View Geographically Process Diagram View by Discipline Evidence in Appendices Practice Print

Importance of Building Trust Once all the design details were fleshed out, purposes. The guardrails created by the IDB and Forging Relationships Among the inter-ministry group (with the Deputy loan indeed protected the core of the program Stakeholders Ministers of Finance, Education, and Health, and strengthened its structural commitment The cross-stakeholder collaborations and the Chair of CONAPO) was created to evidence integration. required for evidence-informed policies to finalize the operational aspects of the and practices are often difficult to initiate, program. This shows the commitment to Conclusion develop, and sustain. Particularly when ongoing collaboration, trust-building and Progresa established a new standard for institutional incentives are lacking, personal negotiation among the key players from social programs in Mexico. Its insistence on trust, respect, and buy-in between individuals early on. A key element to enable the weekly generating and using rigorous evidence to across stakeholder groups become critical to discussions to develop the operating and inform its design and ongoing implementation fostering the effective flow of evidence into technical aspects was President Zedillo´s challenged the prevailing political norms, in practice. involvement to enable the coordination terms of both content of social programs between ministries. Government stakeholders (previously centered on unsuccessful price Progresa’s narrative highlights two very showed the will to collaborate and commit subsidies) and process (where programs important relationships based on trust, to the program as a result of the leadership, tended to be subject to political manipulation). without which success would have been support, and pressure from Mr. Zedillo. Progresa’s evidence-driven approach led impossible. The first component of forged Fostering inter-governmental coordination to the creation of an extremely successful relationships was the trust among Zedillo, through strong leadership was key to the program on multiple levels: Progresa has Gómez de León, and Levy. President Zedillo success of the program. reached massive scale in Mexico; the concept appointed them based on the personal of conditional cash transfer programs has affinity and trust he had for them, as he had spread to dozens of countries around the The Role of Funders | Shielding the Program an academic background himself, but not world; and the establishment of CONEVAL, to explicitly because he was trying to staff the Funders have levers to support evidence- assess the effectiveness of all the country’s government with researchers. However, that informed policy and practice. Progresa sheds social programs based on rigorous evaluation affinity put the right people in the right places. light on a very interesting angle, using an of evidence, has transformed the structure Gómez de León and Levy trusted each other international loan agreement to maintain of how Mexico integrates evidence-based based on their shared goals, their common the continuity of the program under its core practice into public policy. understanding of the overarching objective operating rules based on rigorous evidence. (demonstrable impact for beneficiaries), and The goal of the loan agreement was to the professional respect they had for each shield the program from political cycles or other. This trust was also infused into their teams. temptations to manipulate it for electoral Contents Full Report Case Studies Page 21 of 27 Summary + Findings Stakeholder Map View Alphabetically P Timeline View Geographically Process Diagram View by Discipline Evidence in Appendices Practice Print

Stakeholder Map Intermediaries Funders

Researchers This stakeholder map is a visual represen- tation of the major stakeholders involved Policy Makers Implementers with this project. The importance of each of IDB the actors is defined by their relative size, Primary Relationship and their proximity to the center of the Secondary Relationship project. Their role is defined by the color; multiple colors indicate multiple roles. Primary relationships, denoted by solid lines, indicate the most directly significant relationships while secondary relationships, Ministry of Finance denoted by dashed lines, indicate indirect, IFPRI but influential relationships. Actors not connected by lines are still involved with the project, but less directly. National Coordination INSP of Progresa

CONAPO

Other government agencies President Zedillo’s Office World Bank

Past studies on subsidies Contents Full Report Case Studies Page 22 of 27 Summary + Findings Stakeholder Map View Alphabetically P Timeline View Geographically Process Diagram View by Discipline Evidence in Appendices Practice Print

Timeline

Basic Food- basket Progresa Oportunidades PILOT 3 million households 5 million households Pilot evaluation Pilot National survey National survey National

1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2014

CASH Change in Urban expansion program Progresa strategy formalized designed leadership National Progresa organization founded

Engage Pilot results Administration change, Ministry of presented to new president elected Finance the President

Ministries of Health and Education engaged Evidence generated by the project

A change in policy or significant policy decision influenced the project Contents Full Report Case Studies Page 23 of 27 Summary + Findings Stakeholder Map View Alphabetically P Timeline View Geographically Process Diagram View by Discipline Evidence in Appendices Practice Print

Process Diagram Household survey Growing income inequality Need-based targeting PROBLEM FRAMING PROBLEM FRAMING Progresa was rolled out nationally and scaled during subsequent political administrations to full adoption. At the same time the program became a model Conditional cash transfers for the spread of government conditional Direct subsidies to women cash transfer programs around the world, SOLUTION FRAMING Link subsidy to and changed the structure of assessing child education development interventions by the Mexican SOLUTION RE-FRAMING Government. ITAM + National Basic Foodbasket Institute of Nutrition Program EVALUATION + RCT PILOT

National survey EVALUATION

Progresa

3 million households

The roll out was a Implementation incorporated series of RCTs and ROLL OUT learning from staged roll out evaluations using RCTs and collaboration EVALUATION with academics SOLUTION RE-FRAMING 5 million households

Oportunidades Contents Full Report Case Studies Page 24 of 27 Summary + Findings Stakeholder Map View Alphabetically P Timeline View Geographically Process Diagram View by Discipline Evidence in Appendices Practice Print

Appendix: Data + Methods The research design for the Evidence in Practice project consisted of three broad components. First, we conducted expert interviews (31) with individuals who had spent a significant portion of their professional lives attempting, researching, or promoting the integration of evidence into development practice.A1 This included academics, government officials, foundation program officers, NGO practitioners, and think-tank directors. To identify these experts, we first contacted individuals who had either published extensively and prominently on the topic or who had actively funded research or programs with the explicit goal of integrating evidence into practice. From this first set of experts we conducted snowball sampling until we reached a saturation point.A2 This initial set of interviews informed and directed the next two components, as they resulted in an initial map of the relevant stakeholders in the “evidence-to-practice ecosystem” and the hypothesized and actual paths that seemed to link them together.

Second, we conducted a matched comparison of eight cases of development A1 By development practice, we mean the work of government actors, NGOs, and others who are programs or interventions where rigorous responsible for designing and executing development evidence was integrated with varying degrees projects and programs. of effectiveness. These cases were matched on structural, geographic, and programmatic A2 Data saturation is difficult to define and is dependent on the field of study. In this case, we characteristics—as well as on the extent to defined saturation as the moment when, in a sequence which evidence had informed practices—to of several expert interviews, no interviewee gave us better identify the critical factors that allowed information that we had not encountered before. actors in certain cases, and not others, to A3 George, A. L., & Bennett, A. (2005). Case studies A3 integrate rigorous evidence into practice. and theory development in the social sciences. MIT This matching process led us to identify Press, Cambridge, MA. Chapter 5. pairs of cases across four different countries, leveraging temporal and cross-sectional variation between them as seen in table a2. Contents Full Report Case Studies Page 25 of 27 Summary + Findings Stakeholder Map View Alphabetically P Timeline View Geographically Process Diagram View by Discipline Evidence in Appendices Practice Print

Table A1. Expert Interviews the stakeholder groups, or individuals who Researchers Funders Intermediaries Policymakers Implementers Total would clearly describe the typical experience of enacting a particular stakeholder role. Phase 1: Using the stakeholder map and initial February 2015 – 10 8 6 2 5 31 hypotheses as starting points, this stage May 2016 focused on the dynamics that shape the Phase 2: interactions between stakeholder categories. September 2016 – The work consisted of 34 in-depth interviews 10 7 6 3 8 34 June 2017 with representative actors from each stakeholder group. The interviews focused on each individual’s needs, assumptions, For each case, we first identified, through tactics they employed throughout the operational constraints, main concerns, existing literature and interviews with subject process, to better understand the rationale professional and ideological backgrounds, experts, a series of key informants who had behind their decisions as well as the factors timelines, and aspirations—especially detailed knowledge of the case’s history and that led them to succeed or fail. In total, we concerning the development, dissemination, protagonists. These initial interviews with conducted 161 interviews across the eight and use of novel evidence in development case experts led to the creation of a detailed cases. Interviews were complemented with a practice. This in-depth analysis resulted in a actor/stakeholder map for each case, where wealth of archival information including media more nuanced and detailed stakeholder and we identified the key stakeholder groups that articles, private documents (donor reports, system map that more clearly identified both either participated in or were affected by the internal presentations and communications, breakdown points and paths of connection program, as well as the specific individuals etc.), and public documents (announcements, that hinder and facilitate the exchange who played an active role in the program’s academic articles, editorial pieces). These of knowledge and information across evolution.A4 These stakeholder maps were data were used to trace the chronological stakeholder groups, as well as a refined validated with several informants for each list of events for the overall development of of the cases. We then conducted interviews each case. Each storyline was developed in A4 See Canales, R. (2016). From ideals to institutions: with each of the key individuals across an extensive document that established the Institutional entrepreneurship and the growth of stakeholder groups. Interviewees were asked causal links described by the subjects and Mexican small business finance. Organization Science, 27(6), 1548-1573. to relate chronologies of objective events, ensuring a balanced consideration of different A6 behaviors, choices at critical junctures, stakeholders. A5 Davis, J. P., & Eisenhardt, K. M. (2011). Rotating and facts of the processes described.A5 In leadership and collaborative innovation: Recombination every instance, the goal was to identify the The third component, conducted in parallel to processes in symbiotic relationships. Administrative Science Quarterly, 56(2), 159-201. individuals responsible for the particular the eight case studies, consisted of interviews evolution of a case, as well as the specific with prototypical representatives of each of A6 Ibid. Contents Full Report Case Studies Page 26 of 27 Summary + Findings Stakeholder Map View Alphabetically P Timeline View Geographically Process Diagram View by Discipline Evidence in Appendices Practice Print

Table A2: Case Studies Dates of Number of Primary Country/Program Description Intervention Interviews Stakeholders

South Africa Collaborative Analysis of Labor Employment program introducing new elements to 2011 – 2016 42 Government, Intervention Effectiveness vocational training Researchers

FUEL: Feed, Uplift, Educate, Love School nutrition program 2007 – present NGO

Ghana Teacher Community Assistant Initiative Remedial education program for primary school 2010 – 2013 30 Researchers, children in reading and math through teaching Government assistants from local communities

Graduating the Ultra Poor Poverty alleviation program integrating elements 2010 – 2013 Researchers, NGO of social protection, livelihoods development, and financial services

India Teaching at the Right Level Remedial education program for primary school 2001 – present 51 NGO, Researchers, children in reading and math Government

AQUA+ Water purification drops for retail sale 2010 – present NGO

Mexico

Progresa | Oportunidades Poverty alleviation program using conditional 1997 – present Government cash transfers 38 Government Programa Primer Empleo Employment program using government 2007 – 2012 incentives for the private sector Contents Full Report Case Studies Page 27 of 27 Summary + Findings Stakeholder Map View Alphabetically P Timeline View Geographically Process Diagram View by Discipline Evidence in Appendices Practice Print

set of hypotheses about the breakdown three, we mapped the codes to each of our workshops, we discussed our emerging of communication and about possible case narratives to detect patterns of activities, findings and we gathered additional, interventions to solve it. constraints, and decisions that defined the essential insights from participants. The evolution of each case at critical junctures. workshops served to validate and deepen our Across the three components, we conducted This allowed us to identify similarities and understanding of emerging insights. a total of 226 interviews. All interviews were discrepancies across cases, as well as to in-depth and semi-structured, with an average create comparable counterfactuals that could length of around 90 minutes (minimum of 60, account for differing outcomes.A8 maximum of over 120). Around two-thirds of them were done in person and the rest In stage four, we created process maps, were conducted remotely. All interviews were concept maps, data tables, and detailed recorded and transcribed verbatim. case synopses that linked key challenges, events, and decisions to the specific Data analysis was conducted in several alternative tactics employed by actors and stages. Each of the 226 interview transcripts then to their subsequent consequences for was coded extensively to identify first- the development program or intervention in order concepts related to the integration of question. This final set of analyses revealed evidence into development practice. First- a somewhat consistent set of factors faced order concepts include “concerns about at comparable stages by actors across our reputation” or “short-term decision-making”. different settings. Throughout our analysis, A7 Glaser BG, Strauss AL (1980) The Discovery of This required multiple readings of interview we iterated between emerging insights, Grounded Theory: Strategies for Qualitative Research transcripts, field notes, and archival data to existing theory, and matched comparisons (Aldine Publishers, Hawthorne, NY). associate nearly every passage of text with across cases to identify the mechanisms that one or more codes. These codes were then operated at critical junctures. A8 We ensured consistency in coding across the different cases and authors through several grouped into second-order themes,A7 always mechanisms, including: a) a selection of interviews contrasting them with current research on It is worth mentioning that, at two moments of was coded by two or more coders, after which they the integration of evidence into practice. the project (the first after our first set of expert reviewed discrepancies and agreed on their resolution, Second order themes included “incentive interviews was over and the second after the b) a common project book where all the codes were structures” or “timing misalignments”, each of completion of our initial case narratives) we collectively kept, aggregated, and analyzed, c) a weekly meeting to review coding process and to develop which was developed extensively in a memo hosted a workshop with two different groups a joint coding standard, d) memos were developed that explored the characteristics, tensions, of highly experienced representatives from jointly, with contribution from and verification by the and contradictions of each theme. In stage each of the stakeholder groups. During these different team members, among others. Access here.