Assessment of Impact Evaluation Approaches for Application to Forest Conservation Initiatives in the Brazilian Amazon

Assessment of impact evaluation approaches for application to forest conservation initiatives in the Brazilian Amazon

Phase 1 Report delivered to Mercy Corps

Authors: Erin Sills, Subhrendu K. Pattanayak, Alexander Pfaff, Justin Kirkpatrick, Luisa Young, Simon Hall, David Shoch and Rebecca Dickson Date: 4 April 2014 Version 2.0 (edited)

Phase 1 Report to Mercy Corps

Table of Contents Terms and acronyms ...... 3 Acknowledgements ...... 4 Executive Summary ...... 5 1.0 Introduction ...... 7 2.0 Process and data ...... 7 3.0 Framing the research question ...... 9 3.1 Background and characterization of interventions (“treatments”)...... 9 3.2 Expected impacts (“outcomes”) ...... 10 4.0 Preliminary examination of relationship between outcome and interventions ... 11 5.0 Challenges in constructing robust comparisons...... 12 Selection bias in the Imazon and PMV interventions ...... 14 6.0 Methods to reduce bias ...... 16 6.1 Quasi-experimental methods: SCM, matching, and DID ...... 16 6.2 Alternative quasi-experimental methods ...... 17 7.0 Recommendations ...... 18 7.1 Proposed research questions ...... 18 7.2 Donor pools ...... 20 7.3 Proposed methods ...... 25 7.4 Summarizing considerations in constructing the analyses ...... 27 7.5 Test runs of SCM ...... 28 7.6 Cost-effectiveness...... 29 7.7 Pending data issues ...... Error! Bookmark not defined. 7.8 Timeline ...... Error! Bookmark not defined. Appendix A. Deforestation trends in municipalities partnered with IMAZON, compared to potential control pools ...... 31 Appendix B. Comparison of covariate distributions ...... 38 Appendix C. Synthetic control trial runs for Imazon10 developed from all other municipalities in Pará...... 40 Appendix D. Synthetic control trial runs for Tailândia (example of Imazon10) developed from alternative donor pools ...... 43

TerraCarbon, Version 1.0 / 17 February 2014 2 Phase 1 Report to Mercy Corps

Terms and acronyms

APP: Área de Preservação Permanente, or Area of Permanent Preservation, including riparian zones, steep slopes, and hilltops designated for forest protection per the Brazilian Forest Code Black list: federal priority list, established in 2007, identifying municipalities with high levels of deforestation, administered by the MMA CAR: Cadastro Ambiental Rural, or Rural Environmental Cadaster DID: differences in differences Gross deforestation: conversion of original mature forest to a non-forest land-use, not to be confused with net deforestation which is total change in forest cover, including all transitions IBAMA: Brazilian Institute of Environment and Renewable Natural Resources, federal environmental agency IBGE: Instituto Brasileiro de Geografia e Estatística, or the Brazilian Institute of Geography and Statistics Imazon10: Ten municipalities receiving technical support from Imazon under the USAID/Skoll program, including Don Eliseu, Tailândia, and Ulianópolis; Brasil Novo, Monte Alegre, Novo Progresso, Novo Repartimento, Santana do Araguaia and Santarém; and Moju (all are also participants in the PMV) MMA: Ministério do Meio Ambiente, or Ministry of the Environment MPF: Ministerio Público Federal, or Public Prosecutor’s Office PMV100: One hundred municipalities in Pará that had joined the Programa Municípios Verdes (GMP in English) by signing agreements with the federal prosecutor in Pará (MPF, or Ministério Público Federal) and/or directly with the PMV, by the end of 2013 PA: State of Pará Prefeitura/prefeito: municipal government/ mayor PSDB: Partido da Social Democracia Brasileira, political party of current governor of Pará SCM: Synthetic control methodology

TerraCarbon, Version 1.0 / 17 February 2014 3 Phase 1 Report to Mercy Corps

Acknowledgements

We are grateful for the time and insight generously provided by the following people, which was integral to ensuring the proper focus of this effort: Andreia Pinto, Amintas Brandao, Adalberto Verissimo, Mariana Vedoveto, Carlos Souza, Paulo Amaral, Heron Martins and Paulo Barreto of Imazon, Pércio Barros de Lima of Sindicato Rural in Paragominas, Adriano Gomes Pascoa (BEA consultoria), Armindo Felipe Zagalo Neto of SEMMA in Paragominas, and Justiano de Queiroz Netto and Marussia Whately of Programa Municípios Verdes. In particular we are indebted to Amintas Brandao and Mariana Vedoveto for accompanying us on our visit to Paragominas and for graciously responding to repeated inquiries and data requests.

Phase 1 Report to Mercy Corps

Executive Summary

This report assesses options and makes specific recommendations for how to conduct a robust impact evaluation of two programs to reduce deforestation in the Brazilian Amazon state of Pará: (i) the Programa Municípios Verdes (PMV) and (ii) a supporting program by Imazon that provides additional technical assistance to 10 municipalities in the PMV (Imazon10). Following additional direction received from Mercy Corps, USAID and Skoll, our recommended approach is designed to inform the scaling up of these types of programs to other states in the Brazilian Amazon.

Given the obvious challenge inherent in assessing the impacts of actions taken in relatively few locations, we began by reviewing 35 published impact evaluations using the synthetic control methodology (SCM), which has been developed for evaluating the impacts of programs on a few units. We also met with Imazon, PMV, and the municipal government of Paragominas during a week-long visit (28 October – 2 November) to Pará. Following our visit we compiled and processed data on interventions, impacts and covariates (i.e. characteristics that influence participation and deforestation) for municipalities in the Brazilian Amazon. Consistent with the short-term goals of the PMV and Imazon10, impacts were defined as reductions in gross deforestation (as either a level or a percent of forest area) at the municipal level. We assembled a database on impacts, implementation of the two programs, and over a hundred covariates for 550 municipalities located at least 50% inside the Amazon biome. This database provides the basis for conducting the impact evaluation in phase II of this study.

With these data, we conducted mock runs of various evaluation approaches by (i) examining the distributions of covariate and outcome variables; (ii) estimating a two-way fixed effects model with federal blacklisting as the only additional covariate; (iii) constructing trial synthetic controls for the municipality that collaborated with Imazon before the current project; and (iv) constructing trial synthetic controls for the Imazon10 from a donor pool limited to municipalities in Pará. This allowed us to assess both the required computing power and the sensitivity of each approach to methodological choices such as which covariates and donor pools to include in Phase II. Our focus was on controlling for the selection bias inherent to programs such as the PMV and Imazon10 that are intentionally targeted, i.e., not randomly distributed across space. Trials included: (1) Two-way fixed effects panel regression model, estimated with data from the 550 Brazilian municipalities in the Amazon biome; (2) Synthetic controls for municipalities partnering with Imazon, examining sensitivity to specification of the covariate set, demonstrating feasibility of constructing synthetic controls from donor pool of municipalities in the state of Pará, and examining differences in the synthetic control constructed to answer different evaluation questions (e.g. impact of Imazon10 on municipalities in Pará vs. the Amazon biome vs. municipalities included in the federal blacklist).

These initial empirical mock runs were not intended to provide impact estimates but rather to provide ‘proof-of-concept’ of applying these methodologies to the specific impact evaluation questions of interest, given the availability and distribution of data and the statistical modules available in readily accessible software (STATA and R). As a result, we have in place an analytical framework for organizing, processing, and analyzing data to estimate causal effects of the PMV and Imazon10 in Phase II.

Phase 1 Report to Mercy Corps

Based on consultations with Imazon, PMV, Mercy Corps, Skoll and USAID, we propose focusing the evaluation on the following questions that we believe are most relevant to scaling the program to other states in the Brazilian Amazon and that acknowledge the inter-dependence of the Imazon and PMV programs: (1) What has been the impact on gross deforestation (2011-2015) in a municipality of the local government joining the PMV through a signed commitment no later than July 2011? In more general terms, what has been the effect on deforestation of getting a local government to commit to a low-deforestation development path by signing agreements with state and federal governments? Throughout our discussions, we will refer to this intervention as “PMV on paper.” (2) What has been the impact on gross deforestation (2011-2015) in a municipality of the local government implementing a core set ('core' to be defined in consultation with the PMV and others) of PMV-required actions? In order to allow enough time to observe impacts, we will define these municipalities based on actions undertaken before 2012 (January or July depending on timing of actions and precision of PMV records). We refer to this intervention as “PMV implementation.” (3) What has been the impact on gross deforestation (2011-2015) of a municipal government joining the PMV with additional technical cooperation and support from Imazon (or in general terms, from a NGO of similar capacity)? We refer to this intervention as “Imazon10.” Credible impact evaluations should present estimates based on the preferred methodology and most plausible assumptions, as well as robustness or sensitivity tests. Thus, we propose to generate and thoroughly compare estimation results from a number of approaches including fixed effects panel regression, synthetic control matching (for Imazon10 and “PMV on paper”) and differences-in-differences covariate matching (for “PMV on paper” and “PMV implementation”). Within each evaluation approach, we will assess statistical significance, e.g. through the use of multiple model iterations and placebo analysis of un-treated municipalities in the case of SCM, and sensitivity to assumptions, e.g. by calculating Rosenbaum bounds for robustness to unobserved selection bias in the case of covariate matching. In each case, we will draw our controls from a donor pool of all municipalities in the Brazilian Amazon biome that are not in the PMV and where there was significant deforestation in the decade of the 2000s, and we will check for robustness to dropping municipalities contiguous with the treated group. In each case, we will also ask a slightly different evaluation question: what is the impact of the intervention on municipalities that have been black-listed by the federal government? To answer that question, we will limit our sample of treated and potential controls to municipalities on the black list. Finally, if there is interest, we will also examine the impact of the third intervention on municipalities in Pará (using a donor pool of all municipalities in Pará not in the PMV) and the impact of IMAZON’s support on municipalities in the PMV (using a donor pool of all municipalities in the PMV). We further propose to compute cost-effectiveness ratios for Imazon’s intervention in the 10 target municipalities that relate avoided deforestation achieved in the municipalities to the implementation costs of the Imazon and PMV programs within those specific municipalities. We will collect data on intervention costs and calculate ratios for both the bundled impact of the Imazon and PMV intervention and for the incremental impact of Imazon’s intervention.

Phase 1 Report to Mercy Corps

1.0 Introduction

The Programa Municípios Verdes (PMV) in the Brazilian state of Pará (PA) seeks to engage municipal governments in reducing deforestation and promoting economic development that does not require deforestation, thereby helping municipalities to comply with the environmental goals set by national climate change mitigation policies. As part of its support for this state-wide program, Imazon is providing additional technical assistance to 10 municipalities in the program (Imazon10), selected to represent the range of conditions influencing deforestation in PA. To date, PA is the only state with a program like the PMV. In order to assess whether and how to scale up this type of intervention to other states in the Brazilian Amazon, we require a better understanding of its causal effects, or impacts. While Imazon and the PMV administration ultimately aim to generate sustainable economic development and stabilize forest cover, their immediate objective is to reduce gross deforestation rates. Thus, this report focuses on how to estimate program impacts on gross deforestation. The key challenges are that (1) municipalities were selected into these programs based on factors that also influence their future deforestation rates, (2) the number of treated units in the Imazon10 is very small, limiting the statistical power of most quasi- experimental methods, and (3) the number of treated units in the PMV is large relative to the rest of the state of Pará, requiring identification of similar municipalities in other states in order to model the counterfactual.

2.0 Process and data

In order to develop a plan for impact evaluation of the PMV and Imazon10, we first reviewed the literature and compiled an annotated bibliography of 35 impact evaluations that use the synthetic control methodology (SCM) to estimate the impacts of interventions in a few aggregate units (typically countries, sometimes provinces or states). Second, Sills and Shoch visited Pará for a week (28 October – 2 November) of meetings with Imazon, PMV, and the municipal government of Paragominas. We received further direction regarding framing the research questions and target audience from Mercy Corps, USAID and Skoll via conference calls held on 21 November 2013 and 2 January 2014. Third, we compiled data on municipalities in the Brazilian Amazon, including their participation in the PMV, their deforestation rates, and their characteristics that influence participation and/or deforestation. We obtained these data from Imazon, PMV, IBGE and other internet sources, and previously compiled data that we had in hand (e.g. from 2000 population census and 2006 agricultural census). We gathered data on 550 municipalities, because out of 778 municipalities in the Legal Amazon, 550 have at least 50% of their area in the Amazon biome (originally covered predominantly by dense moist tropical forest). Of those 550, 48 are located partly in other biomes, i.e., 51-99% of their area is in the Amazon biome (Figure 1). Using shape files of deforestation polygons provided by Imazon, we calculated gross deforestation in each municipality for the years 2000 through 2012, referencing a base year of 2000 to define ‘original’ forest cover. The purpose of re- calculating deforestation (rather than using INPE’s estimates) was to ensure that we have measures of deforestation corresponding to current municipal boundaries, and to allow us to calculate deforestation in sub-areas of municipalities (e.g. CAR-eligible zones) using a consistent methodology.

Phase 1 Report to Mercy Corps

Figure 1. Municipalities in the Brazilian Legal Amazon (n=778), Amazon biome, and state of Para.

After compiling this dataset, we examined the distribution of treatment (program participation), covariates, and outcomes (level and percent of gross deforestation) by comparing Paragominas (the original “green municipality”) and the Imazon10 to the broader treated set of the PMV100, and several sets of potential control municipalities (“donor pools”). We then (i) tested the synthetic control methodology for Paragominas, focusing on sensitivity to specification of the covariate set, (ii) assessed the feasibility of constructing synthetic controls for the Imazon10 using the donor pool of all other municipalities in Pará, and (iii) assessed differences in the synthetic control constructed to answer different evaluation questions about one of the Imazon10, as a test case. This process led us to propose the impact evaluation approach below.

Phase 1 Report to Mercy Corps

3.0 Framing the research question

3.1 Background and characterization of interventions (“treatments”)

In 2008, Imazon began collaborating with the municipality of Paragominas to support their efforts to improve municipal environmental governance, develop economic alternatives, and reduce deforestation in order to get off the federal “black list” or priority list of the MMA, which had been established by the federal government in 2007 in order to focus enforcement efforts on municipalities where the most deforestation was occurring. In 2010, Paragominas became the first municipality to be taken off the black list. Their success captured the attention of other municipalities in the same region and of the state government. As a result, Imazon sought funding to work with additional municipalities, obtaining a grant from the Amazon Fund that allowed them to expand collaboration to other municipalities in the same region in 2011. Subsequently, the governor established a special secretariat for a state-wide green municipalities program (Programa Municípios Verdes, hereafter “PMV”), also in 2011.

In March of 2011, the state government formally launched the PMV with the stated goals of promoting sustainable economic activities and reducing deforestation in the state. Imazon closely collaborates with the state government on this program, participating in the management council and the so-called “environmental observatory” that tracks progress reducing deforestation and expanding registration in the CAR (rural environmental cadaster), and providing deforestation alerts that the PMV passes on to the municipal governments. Municipalities can join the PMV by signing an agreement with the MPF (specifically, the federal prosecutor from the Ministério Público Federal assigned to Pará) and/or signing an agreement with the PMV. When the PMV was established, it apparently immediately incorporated 39 municipalities that had already signed agreements with the MPF in 2010. By the end of 2011, 90 municipalities in total had signed agreements with the MPF and thereby officially joined the PMV. Starting in 2012, the PMV also directly signed agreements with other municipalities, and another 10 have joined by signing agreements with the MPF, the PMV, or both. Muncipalities with significant deforestation typically sign agreements with both the MPF and the PMV; municipalities without significant deforestation may sign agreements with the PMV in order to demonstrate support for the program, e.g. through initiatives to improve their urban environment. For our analysis, we propose to consider the municipalities that joined the program by 1 August 2011 (the date that INPE uses to divide years of deforestation data) as the “PMV on paper treatment,” defined as a municipal government (prefeitura) signing up for the PMV (based on government records of the dates that prefeituras joined the PMV). This will include most of the 90 municipalities enrolled in the program by the end of 2011. This will also lead to excluding any municipalities that have joined since 2011 (currently 10); in fact, we expect to drop them from the sample entirely since they have neither been treated for the same length of time nor are they valid controls.

In their agreements with the MPF and/or the PMV, the prefeituras commit to taking several specific steps, but they have made variable progress on implementing these steps. Thus, in addition to evaluating the “PMV on paper”, we propose to evaluate the impact of active participation in the program, as defined by implementation of a core set of steps by the end of 2011 (further below we propose how to define and identify the municipalities treated by “PMV implementation”).

Phase 1 Report to Mercy Corps

Against this backdrop of the PMV, Imazon, with funding from USAID-Skoll, is partnering with 3 of the municipalities where they established technical cooperation agreements in 2011 (Don Eliseu, Tailândia, and Ulianópolis), six new municipalities that signed technical cooperation agreements with Imazon by the end of 2012, and Mojú, which replaced one of the originally selected municipalities that dropped out of the program in 2013 - all of which are participants in the PMV. Imazon’s objective is essentially to replicate the “Paragominas model” with 10 other municipalities in the PMV (later to be scaled up to 40 municipalities), including developing detailed maps and municipal government capacity in order to facilitate the CAR and eventually environmental licensing. For actors in other states considering whether to implement this type of intervention (i.e., the target audience for scaling up), the most relevant question is the impact of implementing the PMV with the level of support provided by Imazon. Thus, we consider the “treatment” of interest to be joining the PMV with support from Imazon. We will consider the period up to 2010 as pre- treatment, and the period starting in 2013 as treatment, with variation in how we treat 2011 and 2012 across the different municipalities in the program, given their staggered entry into the program. We will also consider heterogeneity in the timing and exact bundle of activities implemented in each municipality (e.g., when base maps are finalized and made available), in order to interpret differences in estimated impacts across municipalities.

3.2 Expected impacts (“outcomes”)

While PMV and Imazon agree on the long-term goal of zero net deforestation in Pará (by 2020, according to Imazon’s log frame for the USAID-Skoll project), they are equally clear that their short-term priority is to reduce gross deforestation. Since we are evaluating program impact over less than five years, we will focus on gross deforestation: area of original mature forest in the municipality converted to other land uses. This does not distinguish illegal from legal deforestation (e.g. clearing up to 20% of private properties in municipalities not on the black list is legal), nor consider forest degradation, nor count plantations, restoration, or forest regeneration and potentially new clearing of that regeneration. Again, this is because the immediate goal of the interventions is to bring gross deforestation completely under control, which is considered a pre-requisite for pursuing other objectives, such as developing economic alternatives that do not depend on deforestation and restoring riparian forest zones (as required under the federal Forest Code). One argument is that this ‘closes the frontier’, both increasing the scarcity and hence the value of forest, and reducing the sense of lawlessness and unrestricted exploitation of resources. Another argument is that gross deforestation is one of the specific metrics referenced for getting off the black list, which is another explicit goal of both the PMV and Imazon. In the future, Imazon could use the same evaluation approach to assess impacts on other outcomes, such as net deforestation.

Annual gross deforestation can be represented by the (i) area deforested (in square kilometers), (ii) share of the municipality deforested (square kilometers deforested divided by square kilometers of the municipality), or (iii) percent of the remaining original forest deforested. The third option can be interpreted as the probability that any given area of forest is cleared in a year and is not influenced by municipality size or location vis-à-vis the boundary of the Amazon biome. Thus, from the perspective of modeling the impacts of particular interventions on deforestation agents, percentage deforestation is the preferred outcome measure. However, once on the black list, a municipality can get off only by reducing gross deforestation below 40 square kilometers/year, regardless of the size of the municipality (apparently a deliberate effort to maximize pressure on large municipalities that are major contributors to Brazil’s carbon

Phase 1 Report to Mercy Corps emissions and generally considered “bad actors” in terms of overlooking blatant illegal activity by large holders). This has been incorporated into the PMV as a goal for all participating municipalities. If participating municipalities are really targeting the goal of gross deforestation below 40 square kilometers/year, it suggests that the treatment is heterogeneous, because large municipalities must reduce the pressure to deforest much more than small municipalities. Because this policy goal is defined in terms of the level (rather than the percent) of deforestation, we will also consider that outcome and the possibility of heterogeneous impacts across municipalities of different sizes (by comparing estimates obtained for each municipality using SCM, and possibly by sub-group analysis using covariate matching and DID).

For the USAID-Skoll project, Imazon developed a log-frame that specifies expected outputs and outcomes in different time frames. For example, one output supported by Imazon and required by both the PMV and by the MMA as a requirement of getting off the black list is to register municipal land in the CAR. By registering land, landowners accept responsibility for any deforestation that occurs on that land, making it easier for the government to identify and take enforcement action (e.g. fines, embargos or confiscation of property) against any illegal deforestation (e.g. in excess of 20% of the property or in riparian zones designated for protection (APPs)). Thus, short-term (process-based) program impacts could be evaluated in terms of how much land is registered in the CAR, as a percent of either the eligible land (excluding indigenous territories and protected areas other than APAs) or the land most directly influenced by municipal governments (also excluding federal land reform settlements and undesignated federal lands). However, data on CAR registration is only available for the state of Pará. Therefore, we will consult with Imazon to identify a couple other intermediate outcomes for which data are available across all 550 Brazilian municipalities in the Amazon biome and which are likely to (i) be directly affected by municipal government actions under the PMV and in cooperation with Imazon and (ii) indicate future success in reducing deforestation. Candidate measures including percentage deforestation on the land most directly influenced by municipal governments, expansion of informal roads, and illegal logging (which is correlated with, and often precedes, deforestation). The key is that data on the selected outcomes must be available for both a historical time series and through 2014 for the entire Amazon region and not just the state of Pará, which in turn depends in part on Imazon’s work plan for producing these spatial data layers.

4.0 Preliminary examination of relationship between outcome and interventions

We examined the relationship between treatments and outcome, as defined above, while controlling for time-invariant differences across municipalities and municipality-invariant differences across years, applying a two-way fixed effects model, estimated with panel data on all Brazilian municipalities in the Amazon biome. Because we have only processed deforestation data through 2012, for the Imazon10 we only consider the three municipalities that established partnerships with Imazon in 2011, leading to their later participation in the USAID/Skoll funded project. Participation in the PMV is defined as having signed an agreement with either the PMV or the MPF by the end of a given year. Finally, because both of these programs are closely related to the federal black list, in terms of their motivations and objectives, we also

Phase 1 Report to Mercy Corps include an indicator for a whether a municipality is on the blacklist for at least part of the year (except in the year when a municipality exits the list, which we assume occurs early in the year).

Table 1. Two-way fixed effects model, with balanced panel of 550 municipalities and10 years (2003 – 2012), dependent variable: annual percent deforestation. Program Coefficient Std Error P-value Blacklisting -0.0013 0.0003 0.0000 *** PMV -0.0007 0.0003 0.0101 * Imazon -0.0018 0.0009 0.0483 *

These preliminary examinations suggest that partnering with Imazon effectively reduces deforestation above and beyond the impact of being on the blacklist and in the PMV (because the coefficient on Imazon is statistically significant and negative in this specification that also controls for blacklisting and the PMV). Likewise, the PMV has an effect even after controlling for the blacklist and Imazon initiative. Because this is a fixed effects model, these effects are estimated from variation “within” municipalities, assuming that differences across municipalities are time invariant. Likewise, through fixed effects for years, the model controls for temporal changes in national policies and market conditions that affect all municipalities equally.

In 2010 and 2011, the mean percent deforestation rate across all municipalities in the Amazon Biome was only 0.0002 (0.02%), reflecting a declining trend since 2000. This is partly because there are many municipalities with no or very little deforestation (see figure 2); even the 75th percentile of deforestation was less than 0.0001 (0.01%) in 2010 and 2011. Thus, the fixed effects regression suggests that the programs reduce deforestation by more than the mean deforestation in the sample. This suggests strong selection bias in the choice of municipalities for the blacklist, the PMV, and Imazon. As intended, these are not average municipalities; rather, they have exceptionally high deforestation. This means that to evaluate the impact of these programs, it is critical to narrow down the possible comparison municipalities to a set of controls that are more likely to provide evidence on how much deforestation there would have been in participating municipalities if they had not been selected for these programs. That is, we need to describe the ‘counterfactual’ by identifying municipalities that have as high a deforestation rate as would have been expected in the treated municipalities, absent treatment.

5.0 Challenges in constructing robust comparisons

Spurred by numerous calls for more robust impact evaluation (Kleiman et al, 2000; Pullin and Knight, 2001; Salafsky et al, 2002; Salafsky and Margoluis, 2003; Sutherland et al, 2004; Saterson et al, 2004; Sutherland, 2005; Frondel and Schmidt, 2005; Ferraro and Pattanayak, 2006; Carpenter et al, 2009; Pattanayak et al., 2010), we have finally entered an era in which decision makers are seeking the use of modern causal impacts methods to discern if a particular policy (or driver, treatment, project, program, intervention, treatment)1 is

1 Throughout this section, we use the word policy interchangeably with ‘program’, ‘project’, ‘intervention’, or ‘treatment’, although the term ‘treatment’ is typically used in the program evaluation literature. 12

Phase 1 Report to Mercy Corps

‘causing’ a particular outcome – e.g., reducing deforestation. As discussed in the next section, it imperative to go beyond common empirical designs employed by natural scientists to assess the performance of conservation measures, which rely on comparisons of outcomes (e.g., deforestation) in areas (a) with and without exposure to a conservation policy instrument, or (b) before and after a conservation policy instrument is implemented. “With-without” analyses implicitly assume that (1) the areas with and without the conservation policy are similar in terms of their expected outcomes in the absence of the conservation policy (i.e., similar in characteristics that affect outcomes, such as accessibility, suitability for agriculture and proximity to markets) and (2) there are no spillover effects from the conservation policy to “unexposed” areas. “Before-after” analyses assume that the outcome level (or its trend) before a policy is enacted would have remained constant after the time of treatment (Joppa & Pfaff 2010) and that there is no selection bias in targeting the policy.

When these assumptions fail, estimates of conservation policy effectiveness based on these traditional study designs are biased (Ferraro & Pattanayak 2006; Ferraro, 2009; Joppa & Pfaff, 2010). Consider the case of protected areas in tropical forests. First, deforestation rates may change after the establishment of protected areas for reasons other than protection (e.g., commodity prices), thus invalidating a simple before-after comparison of deforestation (Nagendra, 2008; Joppa & Pfaff, 2010). Second, protected areas, like other conservation interventions, are not established randomly across the landscape. Instead, protected areas tend to be established in poor locations that are far away from cities and unsuitable for agriculture or urbanization (Pfaff et al, 2009; Joppa & Pfaff, 2009, 2010; Andam et al., 2010). The unfavorable location implies that such lands are less profitable and may not experience deforestation even in the absence of protection. In such cases, simple inside-outside (or with-without) analysis will yield upward biased estimates as the deforestation rates in protected areas in the absence of protection are lower than the average deforestation rates of unprotected areas. Finally, the establishment of a protected area may displace the extractive activities to nearby buffer zones (Armsworth et al, 2006; Joppa & Pfaff, 2010). In this case, the estimate of the impact of protection will also be biased upward, but now because deforestation rates in the unprotected areas would have been lower in the absence of the policy. Similar violations of key assumptions arise for other common policies such as payments for ecosystem services, decentralization, and other specific programs of the kind examined under this project (Miteva et al., 2012).

To assess the causal impact of a conservation policy, we must establish what would have happened in areas exposed to a conservation policy if they had not been exposed, i.e. establish the counterfactual outcome, absent the conservation policy. As noted above, estimating this counterfactual outcome is difficult because of the non-random assignment of policies. In econometric terms, non-random assignment induces a correlation between the policy variable (the treatment) and the error term in a regression equation, with the direction of the bias depending on the sign of the correlation between them. Although both experimental and quasi-experimental designs from the program evaluation literature can be used to isolate the causal impacts of a policy, only the latter have been used in the context of deforestation reduction (Ferraro 2009; Joppa & Pfaff, 2010). The latter is relevant in this context because the PMV and Imazon programs were not set up with an experimental design: the selection of municipalities was deliberate rather than random.

Another way to think about the problem is in terms of confounding by some or all the socio-economic, physiographic, and institutional factors listed above that makes it hard to causally relate the policy to deforestation. They are confounders in the sense that they are correlated with the policy and the outcomes, and their omission confounds the relationship between the policy and the outcome. Analysts tend to think

Phase 1 Report to Mercy Corps of these in terms of two critical categories - observed versus unobserved characteristics. Starting with observed characteristics, the first approach is to use multivariate regression to control for the effects of various other factors while trying to estimate the effect of the policy itself. As the workhorse of modern statistical inference, multivariate regression analysis can contribute a great deal when making comparisons across space and or time. That is, if key observed characteristics are included, then the chance of confounding from omission is reduced. For example, including factors in agricultural profit such as distances to markets, slope and soil quality could help a great deal in understanding if payments for ecosystem services or protected areas themselves lower deforestation or just end up on lands that have lower likelihoods of being deforested in the absence of policy because they simply would not yield profits.

However, if there is strong selection in where policies are applied – something which might well be expected in the case of an intervention to reduce deforestation in the Amazon (i.e., highly informed local actors chose where to intervene and that could have been based upon any or a combination of rather different rationales) – then we might suspect that the set of ‘included’ characteristics in policy sites simply do not overlap a great deal with the set of characteristics in comparison (non-policy treatment) sites. If policy and comparison sites are different in terms of many characteristics, then it is more challenging for multivariate regression to independently estimate the policy impact separate from the differences induced by the characteristics.

Selection bias in the Imazon and PMV interventions

It is important to place the PMV and Imazon’s efforts in the context of recent changes at the federal level in Brazil. In 2007, a presidential decree established the legal basis for identifying priority municipalities for preventing, monitoring, and controlling deforestation, including increased enforcement effort by IBAMA, restrictions on commodity sales, and stricter requirements for land registration. In 2008, another presidential decree specified the administrative processes for dealing with environmental crimes, increasing the clarity, speed, and likelihood of punishment through fines, seizure of production inputs, and/or embargoes, including the publication on the internet of a list of embargoed properties. Further, transporting, processing, trading or commercializing products from embargoed areas also become illegal. During 2009, the MPF reached various agreements with industry, producers, and state government in Pará requiring them to stop trade from embargoed areas and implement an environmental licensing system within 12 months. In the same year, Pará created a provisional version of the CAR, which allowed landowners to register before proving definitive title or having a field check of their APPs. However, the municipalities were not able to meet the MPF deadline for an environmental licensing system, and so in 2010 and 2011, the MPF signed another set of agreements with prefeituras in Pará, as well as the state government, Ibama, and producers represented by FAEPA (Federação da Agricultura do Estado do Pará). These agreements postponed the requirement for environmental licensing, conditional on the prefeitura implementing a series of actions to combat deforestation, which in 2011 officially became part of the PMV.

All municipalities in the state of PA are eligible to join the PMV. The administration of the program categorizes municipalities according to whether they are on the federal black list, previously but no longer on the blacklist (with deforestation “monitored and under control”), under strong deforestation pressure, 14

Phase 1 Report to Mercy Corps already largely deforested (“consolidated”), or forested. They prioritize and actively encourage participation by municipalities that are on the federal black list and those under strong deforestation pressure. However, the municipal governments (prefeituras) decide whether to participate. The federal black list is key to participation decisions, because prefeituras hope that the PMV will help them either avoid or get off the black list, thereby ensuring access to credit and markets for their agricultural producers. This motivation is strengthened by the connection between the PMV and the MPF, specifically the federal prosecutor’s office in PA.

The MPF has demonstrated that it can effectively cut off credit and market access for producers in municipalities not complying with agreements to improve municipal environmental governance, in particular access to large beef and soy processors who not only purchase outputs but also finance production. By signing the agreement with the MPF and joining the PMV, prefeituras commit to developing a social pact to reduce deforestation, establishing a working group, monitoring deforestation, registering 80% of the municipality (outside of protected areas) in the CAR, and staying off the black list (by maintaining deforestation below 40 square kilometers per year, regardless of the size of the municipality). Failure to follow-through nullifies the extension of the deadline for environmental licensing. In the agreements with municipalities, the MPF agreed to take all possible measures to guarantee the municipality access to the government agencies necessary to register and license producers; work with INCRA to ensure that producers who comply also are able to register their land in the CCIR (Certificado de Cadastro de Imóvel Rural) system; work with financial institutions to guarantee producers access to financing and to assist the prefeituras with financing infrastructure; and work with the municipalities to help generate benefits in compensation for forest conservation. The federal environmental agency, IBAMA, also agreed not to embargo areas in municipalities that are making progress towards meeting these requirements.

The PMV and Imazon directly help municipalities comply with the requirements of their agreements with the MPF and the requirements of getting off the black list, for example, by facilitating the process of CAR registration, by helping establish GIS capacity in the prefeituras, encouraging NGOs to develop comprehensive base maps, and contracting firms to help smallholders register in the CAR. The PMV accomplishes this largely by coordinating the actions and resource allocations of other state and federal agencies and NGOs, including both Imazon and others such as The Nature Conservancy (TNC). They also develop advertising campaigns to encourage registration in the CAR, and they supported establishment of a new legal mechanism (COTP or Certificado de Ocupação de Terra Pública) to certify that land owners are in the process of obtaining title to state land, based on registration in the CAR. Further, PMV provides a communication channel between prefeituras and both the state police and IBAMA, who can in turn obtain support from the federal police for enforcement actions. Other municipalities in the forested and consolidated zones have a variety of motives for joining the PMV, including offering political support to the governor (if the prefeito and governor are from the same political party), obtaining technical assistance and supplies from the PMV (e.g. “kits” for monitoring deforestation, including motorcycles and financed through a World Bank loan to the state), attaining priority status for state assistance (e.g. from SEPOF), and other financial incentives expected to begin this year (such as allocation of the 25% of the VAT that is returned to municipalities).

Thus, the PMV offers a mix of incentives to participate, some appealing directly to prefeituras (access to the kits from Pará Rural, higher VAT allocation, and support for the governor when they are from the same political party) and some more appealing to agricultural producers (access to markets for beef and soy

Phase 1 Report to Mercy Corps

through CAR registration, liberation of subsidized agricultural credit from the government, provisional land titles). Partnership with Imazon basically provides technical assistance that allows more rapid progress towards the PMV’s goals, including building the local government’s capacity to take responsibility for environmental licensing, which in turn benefits producers who will not need to travel as far or wait as long for licenses. While PMV and Imazon tout the multiple benefits of the program to municipalities, there are clearly also costs, both on the administrative side and in terms of the opportunity cost to producers of not deforesting. In blacklisted municipalities, all deforestation is illegal, and hence these opportunity costs may not be a legitimate part of social cost accounting (and we plan to exclude them from our analysis of cost effectiveness, as discussed below). Nonetheless, the demand for deforested land and the cost of foregoing that deforestation are likely to influence municipal decisions whether to join the PMV and/or partner with Imazon.

All municipalities in the PMV are supported by Imazon through its participation in the overall program, for example, providing monthly deforestation alerts, pursuing clarification on the division of environmental responsibilities between state and municipal governments, and setting standards for municipal environmental agencies. However, Imazon also selected 10 municipalities for more intensive support, seeking to replicate the “Paragominas model.” Specifically, Imazon reports that they first selected municipalities that (1) were on the federal ‘black list’, (2) had a large area of forest remaining, and (3) had demonstrated interest in reducing deforestation (e.g., through signing agreement with the MPF or creating a “social pact” with stakeholders in the municipality to decrease deforestation). They then selected municipalities from this list, along with two other municipalities, in order to obtain a set well distributed across regions of the state facing different deforestation pressures (i.e., speculation vs. colonization vs. expansion of commercial ranching and agriculture). For example, they picked Monte Alegre because it faces the most deforestation pressure in the Calha Norte region, although it does not have a high deforestation rate relative to the rest of the state and was not on the federal black list. Imazon had offered similar support to the municipality of Paragominas starting in 2008, and then expanded to nearby municipalities with financing from the Amazon Fund. Several of these nearby municipalities were also selected for the USAID/Skoll project, although Paragominas is not among the ten. Some of the municipalities that Imazon originally identified as potential participants in the program ultimately decided not to partner with Imazon. This decision of the prefeitura is likely related to both their level of interest in controlling deforestation and their support for the PMV, which is closely identified with both Imazon and the current governor of PA (from the PSDB political party). Thus, both Imazon’s selection rule and the prefeituras’ choices about whether to participate are inherently related to deforestation. This creates a fundamental challenge for identifying the causal impact of these interventions.

6.0 Methods to reduce bias

6.1 Quasi-experimental methods: SCM, matching, and DID

‘Matching’ can be useful in essentially simplifying the task of stripping out the influence of characteristics on the selection process. Matching forces comparisons with non-policy sites which have characteristics

Phase 1 Report to Mercy Corps similar to the policy sites. Naturally, there may not be any sites that can be matched using a set of observables, but even then it is good to have documented that this is the situation in which the analysts find themselves. For example, if we can reliably match some policy sites to similar comparison sites and estimate impacts, then we can consider whether such estimated impacts are reliable estimates for other sites that we cannot match (i.e., off support). Just being transparent about this is useful. When there is a significant group of policy and comparison sites with the same characteristics (with ‘same’ defined in different ways in different matching approaches), clearly it is easier to estimate the deforestation impacts of the policy.

Yet for all the potential value in such transparent comparisons, we must acknowledge that it is by default the case that the data possessed by analysts cannot possibly include all of the key deforestation factors, nor even the smaller set of confounders, or key deforestation factors that also are correlated with location of policy (i.e., assignment). Put another way, it is unquestionable that there are unobserved factors that could sway the estimated impacts. Two ways of dealing with unobserved differences involve the use of time-series data on the outcome, by themselves or in combination with other temporal factors.

Differences-in-Differences (DID): simply having data on the same locations over time provides one useful approach to (not observing but still) removing the influence of unobserved differences across space, i.e., between locations with and without policy. The two-way FE estimates reported in Table 1 are a form of DID model because changes in deforestation in blacklisted, PMV and Imazon municipalities were being compared to controls without these attributes. When no locations had any policies in the past but then one site gets a policy, if we think that trends over time are the same across space then by looking at changes in deforestation for policy versus comparison site, we essentially difference out the effect of fixed (time- invariant) spatial differences and see whether the policy changed the outcome over time. That is, differences between policy and comparison site before policy implementation are differenced from differences between policy and comparison site after policy implementation.

Synthetic Control: We can then do better if we also can observe how outcomes over time progressed in multiple years before any policy was implemented, since that could allow us to pick out locations that, given characteristics, evolve similarly. This is precisely what the synthetic control methodology does, in combination with other characteristics of the policy and control sites. The method is especially relevant when the number of ‘treated’ policy sites is small, ruling out the use of more common matching methods, which choose from relatively large samples of ‘policy’ and potential control sites.

6.2 Alternative quasi-experimental methods

As explained above, matching can eliminate bias from observable factors that influence selection into a program, but it may not be possible to observe all of the factors that influence both deforestation and selection into the PMV or partnership with Imazon. If these unobservable factors are reflected in historical deforestation trends, the resulting selection bias can be greatly reduced by matching on historical deforestation (as is standard in construction of a synthetic control, but can also be included in the set of covariates used for matching). An alternative approach is to identify some observable factor that affects selection into the PMV or partnership with Imazon but not deforestation, called an “instrument”. Using the instrumental variables (IV) method, we could identify the effect of variation in participation driven by that

Phase 1 Report to Mercy Corps

exogenous instrument. IV requires a valid instrument, or factor that (a) is observable, (b) is strongly associated with participation, and (c) does not affect the outcome other than through its influence on participation. If this factor is an administrative rule that PMV or Imazon used to select municipalities, creating a sharp cut-off between municipalities included and excluded based solely on that rule (a quantitative assignment variable or QAV), then the effect of participation on municipalities close to that cut- off can be identified through a regression discontinuity design. For example, municipalities are supposed to be placed on the blacklist based on a clear cut-off rule defined based on historical deforestation. If true, the historical deforestation metric referenced by that rule could be used as a QAV.

These alternative quasi-experimental methods are not viable options for evaluating Imazon’s intervention or the PMV because municipalities were selected based on factors that also drive deforestation. For PMV, we considered a measure of PSDB strength in the municipality (either whether the prefeito is from that party, or the percentage vote for the PSDB in the last election before the PMV was launched), but this would not be a credible instrument because either affiliation with the PSDB or just political alignment between the governor and the prefeito could have other effects on municipal governance that would influence deforestation, outside of the PMV path. For Imazon, we considered using their measures of deforestation risk or risk of entering the black list, but they described selecting municipalities from a diversity of settings, rather than basing the selection on one consistent rule. Further, their program is a partnership with municipalities that requires both Imazon to select and the prefeitura to accept. As with the PMV, the factors entering these decisions are inherently related to deforestation. While personal connections and politics also matter in both of these selection processes, we are not aware of any measure of these available for all municipalities in Pará, let alone the entire Amazon biome. Finally, we also considered whether the selection criteria for the federal black list could be leveraged for the impact evaluation. In our evaluation, we will create measures of historical deforestation corresponding to the published rules for being placed on the black list in a given year, as a potential factor driving selection into the programs. However, this was not the only factor hat determined selection into the PMV or Imazon10, and thus cannot serve as a QAV. Because there is strong temporal and spatial correlation in deforestation in the Amazon, no measure of historical deforestation can be treated as independent of future deforestation, and thus cannot serve as an IV.

7.0 Recommendations

7.1 Proposed research questions

Based on conversations with Imazon and PMV, and our knowledge of the literature and policy debates about tropical deforestation, we suggest that the following three evaluation questions are the most relevant to scaling the intervention supported by USAID and Skoll to other states in the Brazilian Amazon. These questions are oriented towards assessing the effectiveness of different levels of effort implementing the intervention in terms of reducing gross deforestation. Ideally, we would like to know the marginal effect and the marginal cost of each component of the intervention. However, they are bundled together both empirically and in the theory of change underlying the intervention, which suggests that reducing deforestation requires a multifaceted approach.

Phase 1 Report to Mercy Corps

(1) What are the effects of getting local governments to commit to a low-deforestation development path, by signing agreements with state and federal governments? Specifically, what has been the impact on deforestation in a municipality of that municipality’s government (prefeitura) joining the governor’s initiative to support and recognize “green municipalities” (PMV), by signing a commitment with the federal prosecutor (from the Ministério Público Federal or MPF) assigned to Pará? The stated goals of the PMV are to create a sustainable basis for local economies by helping the prefeituras improve municipal governance, clarify legal issues, attract new investments, reduce deforestation and degradation, and promote environmental restoration and conservation of natural resources. However, there is a clear consensus that over the three years since the program was launched in March of 2011, the focus has been on reducing gross deforestation. Hence we ask what impact signing up for the program by July 31 of 2011 (corresponding to the date that INPE uses to define years of deforestation) has had on gross deforestation from August 1 of 2011 to July 31 of 2015, averaging over significant heterogeneity in implementation and enforcement of the program across municipalities (e.g., the tighter restriction on large municipalities imposed by a cap on the level rather than the percentage of deforestation), and comparing to counterfactual outcomes defined by similar Brazilian municipalities in the Amazon biome that have not signed up for the PMV. We will define ‘similar’ based on two different types of matching from all municipalities in the biome where there was at least some minimum level of deforestation in the decade before the PMV program was launched. (2) Both the PMV and the agreement with the MPF clearly state a series of requirements for participating municipalities2, supported by “carrots” from the PMV (e.g., additional revenues from the VAT, due to a change in the formula that rewards compliance with those requirements in the distribution of the 25% that is returned to municipalities) and from the MPF (e.g., promise to facilitate access to credit for producers in municipalities meeting the requirements), as well as “sticks” from the MPF (such as revoking an extension of the deadline to meet environmental licensing requirements in order to maintain market access to sell to agribusinesses). Progress towards meeting these requirements varies across the municipalities that have signed up for the program. Thus, we will also evaluate the impact of a municipality implementing a core set of the required actions before January (or July) 2012 on gross deforestation from August 2011 (or 2012) to July 2015. We will define the “core actions” by (1) eliciting opinions of key informants (at least 2 PMV staff, at least 2 Imazon staff, and at least 2 others familiar with the PMV) using a structured ranking exercise, and (2) exploring patterns in the data (e.g., using cluster analysis) to identify clusters or discontinuities in implementation. For this analysis, we will consider the seven goals of the PMV, possibly disaggregating the seventh goal, since it requires that the municipal government undertake several specific steps that have been identified as important by the PMV, for example establishing a municipal environmental fund and hiring adequate staff including a GIS specialist and lawyer for the municipal environmental secretariat SEMMA. As in #1, we will compare outcomes in municipalities implementing these “core actions” to counterfactual outcomes defined by similar Brazilian municipalities in the Amazon biome that have not signed up for the PMV.

2 META 1: Social pact against deforestation (i.e., stakeholder meeting and agreement), META 2: Municipal government has established a working group to combat deforestation, META 3: 80% of CAR-eligible area registered in the CAR, META 4: Deforestation below 40km², META 5: Working group verifies in the field reports of deforestation received from remote sensing, META 6: Not be on the black list, META 7: Have a system and municipal agency for environmental management (expanding the initial goal of environmental education in municipal schools. 19

Phase 1 Report to Mercy Corps

(3) What has been the impact of a municipal government joining the PMV with technical cooperation and support from Imazon? Here, we will identify the effect of a municipality both signing up for the PMV and being part of the Imazon10, supported by USAID/Skoll starting in 2013 (and by the Amazon Fund starting in 2011 for 3 of the municipalities). We will compare outcomes in these 10 municipalities in 2013 and 2014 to counterfactual outcomes in a weighted combination of similar Brazilian municipalities drawn from a donor pool of all Brazilian municipalities the Amazon biome that have not signed up for the PMV and that have undergone significant deforestation in the 2000s. Conditional on interest from Imazon and Mercy Corps, we will also consider the impact of the Imazon10 specifically on municipalities in Pará and the impact of Imazon’s initiative on municipalities in the PMV. While we believe that the marginal effect of support from Imazon on municipalities already participating in the PMV is less relevant to ‘scalers’ in other states, we will be able to discuss this by using municipalities in the PMV as the donor pool, by comparing estimates obtained in #3 with estimates obtained in #1 using the synthetic control methodology, and by examining the coefficient on Imazon in the fixed effects panel regression approach illustrated in Table 1. (4) For each of the above interventions, we will also evaluate their impact on Brazilian municipalities in the Amazon biome that have been ‘black-listed’ (or placed on MMA’s priority list) at some point. This will allow us to comment on the effects of the interventions on the municipalities that are considered the worst offenders in terms of deforestation and therefore subject to the most enforcement action. This is clearly relevant to local and state governments concerned about the effects of the black list.

Note that the research questions as framed above acknowledge the inter-dependence of the Imazon and PMV interventions, and represent an evaluation of increasing levels of implementation (from 1 to 3) by municipalities, as well as an evaluation of impacts on the sub-set of black-listed municipalities (4).

7.2 Donor pools

By specifying the evaluation questions in the previous section, we have essentially already defined the relevant “donor pools” - the sets of municipalities from which to draw “control” municipalities for comparison with the treated municipalities. All quasi-experimental methods rely on observations of the outcomes in control (or untreated) units to project counterfactual outcomes, e.g., estimates of what deforestation rates would have been in the municipalities in the PMV and partnering with Imazon if they had not selected into those programs. This typically involves two steps: (1) defining and gathering data on a set of relevant control units, and (2) “matching” to select the control units most similar to the treated units and therefore most likely to provide a good estimate of the counterfactual. In the literature on synthetic controls, the first step is said to define the “donor pool,” but we use this term for the first step for any kind of matching (covariate or construction of synthetic controls). Factors to consider in defining the donor pool include the (1) evaluation question, (2) sample size, (3) data availability, (4) avoiding contamination by spill- overs from the treated municipalities, (5) trends in the outcome(s) before treatment, and (6) distribution of covariates that affect participation in the program and the outcomes (i.e., confounders). Considering these in more detail:

Phase 1 Report to Mercy Corps

(1) Quasi-experimental evaluation measures the impact of a particular “treatment,” which involves shifting from one status to a second status. In this sense, the treatment of interest defines both the treated units (units in the second status) and the control units (units in the first status). (2) The literature commonly recommends that the “donor pool” for covariate matching should have 2-3 times more units than the number of treated units, while for SCM, the donor pool should be limited to no more than a couple hundred units (and is often less than 50) because of computational limitations. (3) Data on the same covariates, treatment, and outcomes must be available for both treated and control units. (4) The control units should not be affected by treatment of the treated units, e.g. through municipal governments learning from governments of neighboring municipalities or through displacement of land speculation pressure to neighboring municipalities. In technical terms, this would violate SUTVA, the stable unit treatment value assumption. (5) To employ DID and thereby control for time-invariant unobservables, there must be at least parallel trends in the outcomes before the treatment in the treated and the selected control units. For a synthetic control, the objective is to define a weighted combination of control units with the same outcomes before treatment. To accomplish this, historical outcomes in the treated units must fall within the range of historical outcomes in the donor pool. (6) To match control with treated observations, there should be common support on all covariates of interest, that is, the range of values in the donor pool must be at least as large as the range of values among the treated units on key covariates. In the case of covariate matching, those key covariates are confounders that influence both selection into treatment and outcomes. In the case of synthetic controls, those key covariates are factors other than treatment that are structurally related to deforestation.

Bearing these factors in mind, we evaluated three possible donor pools:

(1) All Brazilian municipalities in the Amazon biome other than those participating in the PMV. Out of 778 municipalities in the Legal Amazon, 550 have at least 50% of their area in the Amazon biome (covered predominantly by dense moist tropical forest); of those 550, another 48 are located only partly in the biome, i.e., with 51-99% of their areas in the biome. We plan to further limit the pool of 550 municipalities to those with significant deforestation in the 2000s (i.e., dropping at least the municipalities with no deforestation in that decade, as shown in Figure 2). a. Evaluation questions: what is the impact of a prefeitura (a) signing up for a state program to reduce deforestation, (b) actively participating in a state program to reduce deforestation, or (c) signing up for a state program to reduce deforestation with intensive support from an NGO (like Imazon), relative to “business as usual” for prefeitura action in the Amazon; a. Advantages: large sample size, allowing multiple draws and sensitivity checks to trimming in different ways (e.g. eliminating municipalities with no deforestation), can exclude municipalities in Pará most susceptible to spill-over effects; b. Disadvantages: some measures of treatment, covariates, and outcomes only available for PA; underlying unobservable structural differences in the drivers of deforestation across different states. 21

Phase 1 Report to Mercy Corps

Figure 2. 107 municipalities with no deforestation for the years 2000-2012 are highlighted in blue.

(2) All municipalities on the Brazilian federal government’s black list (including all municipalities ever on this list, even if they have exited); a. Evaluation questions: among municipalities where agricultural producers have been denied credit and market access as a result of being placed on the federal ‘black list’ for deforestation, what is the impact of a prefeitura (a) signing up for a state program to reduce deforestation, (b) actively participating in a state program to reduce deforestation, or (c) signing up for a state program to reduce deforestation with intensive support from an NGO; b. Advantages: likely to display similar trends in deforestation as the municipalities in the PMV and partnered with Imazon that are also on the blacklist and thus most critical from the perspective of reducing deforestation; geographically dispersed, so unlikely to be affected by spill-overs from treated municipalities; c. Disadvantages: smaller sample size; some measures of treatment, covariates, and outcomes only available for the state of PA; underlying unobservable structural differences in the drivers of deforestation across different states; only allows estimation of program effects on sub-set of treated municipalities that are on the blacklist. (3) All municipalities in the state of Pará not participating in the PMV a. Evaluation questions: what is the impact of a prefeitura in the state of Pará (a) signing up for the PMV, (b) actively participating in the PMV, or (c) receiving Imazon’s support for participation in the PMV; b. Advantages: all measures of treatment, covariates, and outcomes available for the state of PA; municipalities in PA face the same legal and institutional setting, e.g. as reflected in PA’s implementing regulations for the Forest Code; c. Disadvantages: sample size below 50 (too small for evaluating the PMV). Within the pool of municipalities in PA, Imazon has selected a unique set of municipalities for the program, which may not have equivalents in the rest of the state in terms of the combination of high

Phase 1 Report to Mercy Corps

deforestation pressure and high governance capacity; likewise, the PMV has targeted high- priority municipalities in the state of PA, meaning that other municipalities in the state are different by definition. There could also be spill-overs of various kinds, e.g. the PMV could draw resources away from non-participating municipalities, and within the participating municipalities, the PMV could direct resources (e.g. contributions from other NGOs) to municipalities not benefiting from direct collaboration with Imazon. This potential violations of SUTVA could be mitigated by eliminating municipalities contiguous (i.e., sharing a border) with treated municipalities from the donor pool.

Note that the latter treatment would significantly reduce the donor pool from within Para - there are 52 municipalities that share a border with the Imazon10 municipalities (Figure 3), and 189 that share a border with the municipalities that have signed on to the GMP by August 2011 (Figure 4).

Figure 3. Municipalities (n=52) that share a boundary (adjacent) with the Imazon10 municipalities

Phase 1 Report to Mercy Corps

Figure 4. Municipalities (n=189) that share a boundary (adjacent) with signed GMP (PMV 100) municipalities.

Appendix A shows deforestation trends in Paragominas, the Imazon10, and these three potential donor pools, and Appendix B shows histograms comparing covariate distributions in the PMV and the rest of the Amazon biome. These figures do not rule out any possible donor pools for either construction of synthetic controls or covariate matching: the Imazon10 fall within the distribution of deforestation trends observed in each of the donor pools (with the exception of Brasil Novo in 2003), and likewise, the histograms of municipalities in the PMV overlap with histograms of all other municipalities in the biome (although the distributions are different, as expected).

Thus, given that our primary audience is potential “scalers” in other states of the Brazilian Amazon, we propose to focus on the first two donor pools: (1) all municipalities in the Brazilian Amazon biome not in the PMV and with significant deforestation in the 2000s, to evaluate the impact of signing up for the PMV, actively participating in the PMV, or participating in the PMV with support from Imazon, and (2) 24

Phase 1 Report to Mercy Corps

blacklisted municipalities, to evaluate the impact of a blacklisted municipality signing up for the PMV, actively participating in the PMV, or participating in the PMV with support from Imazon (i.e., in this case, we consider only blacklisted municipalities for both treatment and control). As a back-up option, we will consider a donor pool of all municipalities in Pará, for variations on treatment (such as municipal governance institutions), covariates (such as extent of illegal logging in each municipality), and outcomes (such as percent of CAR-eligible areas registered in the CAR) that can only be measured in the state of Pará, due to data availability. As a robustness check, we will exclude municipalities that share borders with treated municipalities from the donor pool (to avoid spillovers) and as discussed below, we will also identify and exclude municipalities that have been the target of similar interventions since 2010.

7.3 Proposed methods

For a robust analysis, we propose to estimate effects using a differences-in-differences framework that sweeps out both observed and unobserved time-invariant differences across municipalities. Within that framework, we will employ three different methods, as appropriate for the evaluation question and the sample size of the intervention. Panel regression implements this with fixed effects; SCM by constructing a synthetic control with the same baseline level of the outcome; and in covariate matching, we will both match on baseline outcomes and examine DID in the matched sample.

To evaluate the PMV (“on paper” and “implementation”) and Imazon10 interventions, we will start by re- estimating the fixed effects model shown in Table 1, adding other covariates that (a) are theoretically related to deforestation, (b) vary over both time and municipalities and (c) can be measured with available data. We will also test robustness of the results to slightly different samples that have been used in the literature on deforestation in the Brazilian Amazon (such as dropping municipalities not located 100% within the Amazon biome). We include this estimation because it a. provides comparable estimates of the marginal effects of the PMV and the Imazon10 and their combined effect; b. can be estimated in the matched sample for “PMV on paper”, dropping treatment municipalities that are far off-support and keeping only the 2 or 3 best matched per PMV municipality, thus providing a “double-robust” estimate; c. is widely understood and therefore easy to explain and discuss.

The fixed effects model effectively soaks up most sources of variation, but does not explicitly account for selection bias. We can see this clearly in the preliminary results in Table 1, which show that the PMV and Imazon’s partnership with 3 of the Imazon10 municipalities starting in 2011 had a large negative effect on deforestation. In fact, the effect is larger than the mean deforestation per municipality in the entire dataset, which includes many municipalities with apparent zero deforestation (Figure 2), demonstrating that the intervention must have been carefully targeted to municipalities with high deforestation.

We will also evaluate the PMV and Imazon10 using matching: (i) synthetic control matching for the Imazon10 and the PMV on paper (which encompasses the PMV implementation), and (ii) covariate matching for the two definitions of PMV treatment (on paper and implementation). We will identify the best set of covariates by examining the root mean squared prediction errors among control municipalities both pre- and post- treatment. Using that covariate set, we will construct synthetic controls that are nearly identical to each of the Imazon10 municipalities in terms of historical deforestation (since 2000), assessed by

Phase 1 Report to Mercy Corps both visual inspection and the root mean squared prediction error, and then calculate the difference in deforestation after treatment.

In order to establish whether the observed impacts on the Imazon10 are significantly different from zero, we will also construct synthetic controls and calculate placebo impacts for all other municipalities in the sample, allowing us to establish where the impacts on Imazon10 fall in that general distribution. (This requires significantcomputational time, but is possible with a dedicated computer.) Finally, we will test robustness of results to different donor pools - either different random draws as illustrated in the appendix or different ways to trim (e.g. dropping municipalities that report zero deforestation in any year) - and to different covariate sets (including using Y alone and X alone).

We also plan to construct synthetic controls for every municipality in the PMV, using a small donor pool, which will (a) allow us to compare the estimated effects of the PMV and the Imazon10 using the same matching methodology, (b) provide an easy way to account for the different dates that municipalities signed agreements with the MPF, (c) allow different covariates for matching municipalities in different deforestation fronts (e.g. where land speculation vs. agricultural colonization vs. commercial ranching and agriculture are key drivers), (d) produce estimates of heterogeneous impacts across municipalities (e.g. allowing us to test hypotheses that effects should vary with size of municipality and according to whether the municipality has just signed onto the PMV or is actively implementing the PMV), and (e) allow us to compare estimated effects of the PMV using two matching methodologies: SCM and covariate matching with DID.

To obtain the matched sample, we will employ covariate matching with a rich set of possible confounders, including historical deforestation, as described in the next section. We will produce matched samples including the single, two, and three “closest” municipalities for each PMV municipality, all with replacement. We will test different methodologies (e.g. genetic matching in R), selecting the method that results in the best balanced sample. We will evaluate balance on the selected covariates as well as testing for any significant differences in historical deforestation trends. After determining the matched sample with the best balance and confirming that historical deforestation trends are the same in the PMV and matched samples, we will test statistical significance of DID in mean deforestation outcomes among PMV and matched control municipalities, with controls for covariates to adjust for any remaining imbalances in the sample.

Phase 1 Report to Mercy Corps

7.4 Summarizing considerations in constructing the analyses

With covariate matching, our objective is to identify municipalities similar to the municipalities in the PMV in all respects except that they did not participate. For this analysis, we will necessarily use a donor pool including municipalities outside PA and in fact, we might restrict the donor pool to municipalities outside of PA in order to avoid any spillover effects. In that case, we can think of identifying municipalities that would be just as likely to join the PMV as the municipalities currently in the program if it were offered in their state. The objective is to create a sample that is balanced on all relevant covariates, except for treatment, similar to what would be obtained by randomly assigning municipalities to participate in the PMV. With random assignment, the participant and non- participant municipalities would have had the same distribution of covariates, that is, they would have been the same in expectation. In order to achieve this through matching, we identify the characteristics that influenced the participation decision in one direction or the other and therefore are not balanced across treatment and control. In particular, we are concerned with variables that influence both the decision to join the program and the deforestation outcome, as those can confound our understanding of the treatment effect unless they are balanced.

Drawing on our understanding of selection into the program, we are therefore seeking measures of the factors that drive deforestation, drive concern about sanctions that can be imposed by the federal government (including the black list, embargos and equipment seizures by IBAMA, and restrictions on market access imposed by the MPF), and drive perceptions that outside assistance is needed to control deforestation. Deforestation is largely a function of agricultural profitability, which in turn is related to both biophysical (soil quality, slope, rainfall) and market access (percent of municipality within 5KM of a road, quarantine for hoof-and-mouth disease) factors. Given the strong spatio-temporal correlation of deforestation in the Brazilian Amazon, historical deforestation trends also drive future deforestation. Concern about federal sanctions depends on whether the municipality supplies large slaughterhouses and soybean processing facilities most compliant with MPF requirements, as well as prior enforcement action (such as number of properties embargoed by IBAMA and whether on the black list). Perceived need for outside assistance depends on both cost drivers (e.g. greatest distance from municipal seat to border of municipality, which captures the difficulty of enforcement and of compliance with the law in irregular shaped municipalities; count of farms, which affects difficulty of achieving 80% CAR coverage) and political alliances (such as whether the prefeito is from the same political party as the governor who created the PMV). Some factors fit into more than one category: for example, the importance of agriculture for the municipality (% PIB from agriculture) is likely to both drive deforestation and interest in securing agricultural credits and markets. Finally, the PMV targets municipalities designated in the “under pressure” zone, and thus 27

Phase 1 Report to Mercy Corps all else equal, those are more likely to be brought into the program as well as being more threatened by deforestation, by definition.

Note that the patterns of selection into PMV implementation is also of interest in and of itself, because Imazon’s log frame includes related measures as intermediate outcomes:

 # of municipalities off the black list  # of municipalities that have signed decentralization agreement with state  # of municipalities that have fully complied with PMV goals.

7.5 Test runs of SCM

To test the construction of synthetic controls, we first identify covariates that are related to deforestation and therefore should be similar in a valid synthetic control. In the test runs of SCM for the Imazon10 presented in the appendix, we included the factors discussed above, and others such as descriptors of land tenure in the municipality, including percent under federal control, in protected areas, indigenous territories, and agricultural settlements; indicators of pressure for land speculation and invasion, including land tenure and also number of conflicts recorded by the Catholic Church; and variables closely associated with deforestation such as population density, cattle density, and mine density.

Appendix C demonstrates that it is possible to construct synthetic controls that are nearly identical to each of the Imazon10 municipalities in terms of historical deforestation (since 2000), even using the most restricted donor pool: municipalities in the state of Pará. Novo Brasil has an unusual spike in deforestation in 2003 that is difficult to replicate in a synthetic control and remains to be investigated. Appendix D illustrates how the synthetic control and the estimated result may change depending on whether we ask about the impact of Imazon’s support on a municipality already enrolled in the PMV, a municipality already blacklisted, or just a similar municipality in the Brazilian Amazon.

We will also give careful consideration to the sensitivity of estimated impacts to the suite of variables used. For example, we further tested the SCM by constructing a synthetic control for the municipality of Paragominas, using covariates and deforestation prior to 2008, and we found that the estimated impact on deforestation was sensitive to the mix of lagged dependent variables (historical deforestation) and independent variables used in the covariate set. Based on the literature and econometric theory, we believe that a combination of lagged deforestation and characteristics of municipalities related to deforestation will result in the best synthetic control (i.e., the synthetic control that gives the most reasonable counterfactual prediction). However, based on our pilot testing in Paragominas, we will also construct synthetic controls using only historical deforestation (the lagged outcome variable) and only covariates (characteristics of municipalities) in order to assess sensitivity of results to this methodological choice.

Phase 1 Report to Mercy Corps 7.6 Cost-effectiveness

Impact estimates should be placed in context of the costs incurred to generate those impacts. In phase 2 of this project, we will develop cost-effectiveness ratios for Imazon’s intervention in the 10 target municipalities, relating avoided deforestation in the municipalities to Imazon’s and PMV’s budget for work in those municipalities. We expect to provide a range of ratios, incorporating different assumptions about (a) the long-term impact of the intervention and the division of costs into start-up and incremental annual costs, and (b) the starting point for the interventions, in particular Imazon’s earlier cooperation with three of the municipalities (Don Eliseu, Tailândia, and Ulianópolis) with support from the Amazon Fund. We will seek to calculate ratios for both the bundled impact of the Imazon and PMV intervention and for the incremental impact of Imazon’s intervention.

To establish these cost-effectiveness ratios, we will acquire information from Imazon’s accounting department on expenditures associated with Imazon’s partnership with the 10 municipalities, under the USAID/Skoll project and potentially under other projects such as the Amazon Fund. We will provide narrative context for these cost effectiveness ratios that explain the importance of Imazon’s existing capacity and activities, which are not counted as part of the costs since they would have occurred even without the intervention, but which are likely to decrease the marginal costs of the intervention. The same applies to the PMV, as a state-wide program that coordinates efforts by multiple government agencies and non- governmental organizations: the incremental cost of that coordination activity in the 10 target municipalities will certainly be lower than the costs of starting up a similar intervention in a different state without a state-wide program and without similar levels of cooperation across federal and state agencies. In order to calculate a cost-effectiveness ratio for the bundled PMV and Imazon10 intervention, we will need budget information from the PMV specific to those 10 municipalities (e.g. the cost of publicity campaigns promoting registration in the CAR).

Note that we are proposing to focus on implementation costs (including transactions and administrative) rather than opportunity costs (foregone benefits from cleared land, net of forest benefits, in areas where deforestation is averted by the intervention). In the context of REDD+, much of the scientific literature and policy discussion about costs focuses exclusively on opportunity costs (omitted implementation and transactions costs). However, opportunity costs of illegal activities are generally not counted as part of the costs of avoided deforestation, because there are potentially moral hazards associated with rewarding those pursuing illegal activities, and because the costs of enforcing the law could be greater or less than the opportunity costs. In eight out of the Imazon10 municipalities, all deforestation is illegal because they are on the federal blacklist (MMA’s priority list). In two of the municipalities (Monte Alegre and Santarém), private landowners are legally allowed to deforest up to 20% of their properties, under federal law, although Imazon, the prefeituras

Phase 1 Report to Mercy Corps and the state government have committed to reducing all deforestation in those municipalities, not distinguishing legal from illegal.

Thus, in phase 2, we will gather information from Imazon on the costs of their collaboration with the 10 partner municipalities. We will focus on “on-budget” costs for the USAID/Skoll project that are tracked through Imazon’s accounting system, working with Imazon to draw a reasonable distinction between costs incurred for support of the 10 municipalities vs. the state-wide PMV. For the three municipalities that signed technical cooperation agreements with Imazon in the first half of 2011 under an initiative funded by the Amazon Fund, we will also request information on Imazon’s expenditures from that project. We will convert all cost information from Imazon and PMV into comparable units, adjusted for inflation, exchange rate, and year of implementation, using a range of discount rates. Further, we will provide Imazon with a plan for developing other cost-effectiveness ratios of their work, including with the numerator measured in tonnes of averted GHG emissions (rather than hectares of averted deforestation) and with the denominator including the costs incurred by the prefeituras of the cooperating municipalities. This plan will also include recommendations for a cost-accounting system that will allow Imazon to report cost- effectiveness ratios for specific activities that they believe will have the most significant long- term impacts (that may include e.g. development of base cartographic and land cover maps, using the 2011 Rapid Eye imagery acquired by Brazil, which is occurring in the 10 target municipalities but also in other municipalities with support from both Imazon and other NGOs; or a relatively new mechanism for imposing embargos on all properties in polygons identified as having a high deforestation rate). Our recommendations will focus on establishing cost-effectiveness ratios that can be compared to other interventions and that can be converted into cost-benefit ratios, using research on the social benefits of averted deforestation. For example, the cost-effectiveness ratios measured in terms of averted GHG emissions can be multiplied by the social cost of carbon in order to obtain the cost-benefit ratio for climate change mitigation, excluding other global to local benefits including reduced risk of biodiversity loss and stabilization of hydrological systems.

Phase 1 Report to Mercy Corps

Appendix A. Deforestation trends in municipalities partnered with IMAZON, compared to potential control pools

Percent of Area Deforested – Paragominas (blue) and Imazon10 (red)

Percent of Area Deforested – Paragominas (blue), Imazon10 (red), and GMP/PMV 100 (gray)

Phase 1 Report to Mercy Corps

Phase 1 Report to Mercy Corps Percent of Area Deforested – Paragominas (blue), Imazon10 (red), Pará state (gray)

Percent of Area Deforested – Paragominas, Imazon10, and Blacklisted Municipalities

Phase 1 Report to Mercy Corps Percent of Area Deforested – Paragominas (blue), Imazon10 (red), and all Brazilian municipalities in the Amazon Forest Biome (gray)3

3 Municipalities are considered in the Amazon Forest Biome if at least 50% of their area is inside the biome. 34

Phase 1 Report to Mercy Corps Area Deforested – Paragominas (blue) and Imazon10 (red)

Area Deforested – Paragominas (blue), Imazon10 (red), and PMV/GMP 100 (gray)

Phase 1 Report to Mercy Corps Area Deforested – Paragominas (blue), Imazon10 (red), and Pará state(gray)

Area Deforested – Paragominas (blue), Imazon10 (red), and Blacklisted Municipalities (gray)

Phase 1 Report to Mercy Corps

Area Deforested – Paragominas (blue), Imazon10 (red), and all Brazilian Municipalities in the Amazon Forest Biome (gray)

Phase 1 Report to Mercy Corps

Appendix B. Comparison of covariate distributions

Comparison of covariate distributions between PMV100 and the rest of the municipalities in the Brazilian Amazon Forest Biome (illustration of how to assess common support across PMV100 and alternative controls pools)

Phase 1 Report to Mercy Corps

Appendix C. Synthetic control trial runs for Imazon10 developed from all other municipalities in Pará

Deforestation trends in IMAZON10 plus Paragominas compared to synthetic controls constructed from remaining municipalities in state of Pará with complete data (N=122); y axis is percent annual gross deforestation. Quality of synthetic control is judged by visual inspection and root mean square prediction error (RMSPE).

Phase 1 Report to Mercy Corps

Appendix D. Synthetic control trial runs for Tailândia (example of Imazon10) developed from alternative donor pools

Synthetic Controls for Tailândia constructed from alternative donor pools; y axis is percent annual gross deforestation.

Donor pool consists of 40 municipalities other than IMAZON10 on the federal blacklist.

Phase 1 Report to Mercy Corps

Donor pool consists of first set of 150 randomly-selected municipalities within the Amazon Biome that have at least one year of non-zero deforestation from 2000 to 2012; possible to use larger donor pool with faster computer, smaller covariate set, etc.

Phase 1 Report to Mercy Corps

Donor pool consists of second set of 150 randomly-selected municipalities within the Amazon Biome that have at least one year of non-zero deforestation from 2000 to 2012; possible to use larger donor pool with faster computer, smaller covariate set, etc.

Phase 1 Report to Mercy Corps

Donor pool consists of 150 randomly-selected municipalities within the Amazon Biome (including those with zero deforestation); possible to use larger donor pool with faster computer, smaller covariate set, etc.

In all of the above, variables such as properties under embargo and registered in the CAR, population density, count of farms, and indigenous territory have substantial influence on construction of synthetic control. Municipalities such as Porto Velho (from the blacklisted pool) and Tome Açu (from the Amazon biome) make up the synthetic control.