Assessment of Impact Evaluation Approaches for Application to Forest Conservation Initiatives in the Brazilian Amazon
Total Page:16
File Type:pdf, Size:1020Kb
Assessment of impact evaluation approaches for application to forest conservation initiatives in the Brazilian Amazon Phase 1 Report delivered to Mercy Corps Authors: Erin Sills, Subhrendu K. Pattanayak, Alexander Pfaff, Justin Kirkpatrick, Luisa Young, Simon Hall, David Shoch and Rebecca Dickson Date: 4 April 2014 Version 2.0 (edited) Phase 1 Report to Mercy Corps Table of Contents Terms and acronyms .................................................................................................................................... 3 Acknowledgements ....................................................................................................................................... 4 Executive Summary ...................................................................................................................................... 5 1.0 Introduction ........................................................................................................................................ 7 2.0 Process and data ............................................................................................................................... 7 3.0 Framing the research question ................................................................................................... 9 3.1 Background and characterization of interventions (“treatments”).......................... 9 3.2 Expected impacts (“outcomes”) ........................................................................................... 10 4.0 Preliminary examination of relationship between outcome and interventions ... 11 5.0 Challenges in constructing robust comparisons................................................................ 12 Selection bias in the Imazon and PMV interventions ............................................................... 14 6.0 Methods to reduce bias ............................................................................................................... 16 6.1 Quasi-experimental methods: SCM, matching, and DID ............................................. 16 6.2 Alternative quasi-experimental methods ........................................................................ 17 7.0 Recommendations ......................................................................................................................... 18 7.1 Proposed research questions ............................................................................................... 18 7.2 Donor pools ................................................................................................................................. 20 7.3 Proposed methods .................................................................................................................... 25 7.4 Summarizing considerations in constructing the analyses ...................................... 27 7.5 Test runs of SCM ........................................................................................................................ 28 7.6 Cost-effectiveness...................................................................................................................... 29 7.7 Pending data issues ....................................................... Error! Bookmark not defined. 7.8 Timeline ............................................................................. Error! Bookmark not defined. Appendix A. Deforestation trends in municipalities partnered with IMAZON, compared to potential control pools ........................................................................................................................ 31 Appendix B. Comparison of covariate distributions ..................................................................... 38 Appendix C. Synthetic control trial runs for Imazon10 developed from all other municipalities in Pará................................................................................................................................ 40 Appendix D. Synthetic control trial runs for Tailândia (example of Imazon10) developed from alternative donor pools ........................................................................................... 43 TerraCarbon, Version 1.0 / 17 February 2014 2 Phase 1 Report to Mercy Corps Terms and acronyms APP: Área de Preservação Permanente, or Area of Permanent Preservation, including riparian zones, steep slopes, and hilltops designated for forest protection per the Brazilian Forest Code Black list: federal priority list, established in 2007, identifying municipalities with high levels of deforestation, administered by the MMA CAR: Cadastro Ambiental Rural, or Rural Environmental Cadaster DID: differences in differences Gross deforestation: conversion of original mature forest to a non-forest land-use, not to be confused with net deforestation which is total change in forest cover, including all transitions IBAMA: Brazilian Institute of Environment and Renewable Natural Resources, federal environmental agency IBGE: Instituto Brasileiro de Geografia e Estatística, or the Brazilian Institute of Geography and Statistics Imazon10: Ten municipalities receiving technical support from Imazon under the USAID/Skoll program, including Don Eliseu, Tailândia, and Ulianópolis; Brasil Novo, Monte Alegre, Novo Progresso, Novo Repartimento, Santana do Araguaia and Santarém; and Moju (all are also participants in the PMV) MMA: Ministério do Meio Ambiente, or Ministry of the Environment MPF: Ministerio Público Federal, or Public Prosecutor’s Office PMV100: One hundred municipalities in Pará that had joined the Programa Municípios Verdes (GMP in English) by signing agreements with the federal prosecutor in Pará (MPF, or Ministério Público Federal) and/or directly with the PMV, by the end of 2013 PA: State of Pará Prefeitura/prefeito: municipal government/ mayor PSDB: Partido da Social Democracia Brasileira, political party of current governor of Pará SCM: Synthetic control methodology TerraCarbon, Version 1.0 / 17 February 2014 3 Phase 1 Report to Mercy Corps Acknowledgements We are grateful for the time and insight generously provided by the following people, which was integral to ensuring the proper focus of this effort: Andreia Pinto, Amintas Brandao, Adalberto Verissimo, Mariana Vedoveto, Carlos Souza, Paulo Amaral, Heron Martins and Paulo Barreto of Imazon, Pércio Barros de Lima of Sindicato Rural in Paragominas, Adriano Gomes Pascoa (BEA consultoria), Armindo Felipe Zagalo Neto of SEMMA in Paragominas, and Justiano de Queiroz Netto and Marussia Whately of Programa Municípios Verdes. In particular we are indebted to Amintas Brandao and Mariana Vedoveto for accompanying us on our visit to Paragominas and for graciously responding to repeated inquiries and data requests. 4 Phase 1 Report to Mercy Corps Executive Summary This report assesses options and makes specific recommendations for how to conduct a robust impact evaluation of two programs to reduce deforestation in the Brazilian Amazon state of Pará: (i) the Programa Municípios Verdes (PMV) and (ii) a supporting program by Imazon that provides additional technical assistance to 10 municipalities in the PMV (Imazon10). Following additional direction received from Mercy Corps, USAID and Skoll, our recommended approach is designed to inform the scaling up of these types of programs to other states in the Brazilian Amazon. Given the obvious challenge inherent in assessing the impacts of actions taken in relatively few locations, we began by reviewing 35 published impact evaluations using the synthetic control methodology (SCM), which has been developed for evaluating the impacts of programs on a few units. We also met with Imazon, PMV, and the municipal government of Paragominas during a week-long visit (28 October – 2 November) to Pará. Following our visit we compiled and processed data on interventions, impacts and covariates (i.e. characteristics that influence participation and deforestation) for municipalities in the Brazilian Amazon. Consistent with the short-term goals of the PMV and Imazon10, impacts were defined as reductions in gross deforestation (as either a level or a percent of forest area) at the municipal level. We assembled a database on impacts, implementation of the two programs, and over a hundred covariates for 550 municipalities located at least 50% inside the Amazon biome. This database provides the basis for conducting the impact evaluation in phase II of this study. With these data, we conducted mock runs of various evaluation approaches by (i) examining the distributions of covariate and outcome variables; (ii) estimating a two-way fixed effects model with federal blacklisting as the only additional covariate; (iii) constructing trial synthetic controls for the municipality that collaborated with Imazon before the current project; and (iv) constructing trial synthetic controls for the Imazon10 from a donor pool limited to municipalities in Pará. This allowed us to assess both the required computing power and the sensitivity of each approach to methodological choices such as which covariates and donor pools to include in Phase II. Our focus was on controlling for the selection bias inherent to programs such as the PMV and Imazon10 that are intentionally targeted, i.e., not randomly distributed across space. Trials included: (1) Two-way fixed effects panel regression model, estimated with data from the 550 Brazilian municipalities in the Amazon biome; (2) Synthetic controls for municipalities partnering with Imazon, examining sensitivity to specification of