GiveDirectly

Goldilocks Toolkit Innovations for Poverty Action poverty-action.org/goldilocks Right-fit monitoring and evaluation (M&E) systems embody the principles of Credible, Actionable, Responsible, and Transportable, or CART. In the Goldilocks case study series, we examine the M&E systems of several innovative organizations and explore how the CART Principles can work in practice.

Acknowledgements: This Goldilocks Toolkit was authored by Mary Kay Gugerty, Dean Karlan, Delia Welsh, Tetyana Zelenska, with editing and design by the IPA Communications Team (David Batcheck, Laura Burke, Jennifer Cowman, Heidi McAnnally-Linz, Megan McGuire). GiveDirectly: Developing a Research Agenda after Impact Evaluation

Unconditional cash transfers are 1) existing cash transfer programs, an impact evaluation. Specifically, increasingly recognized to be a cost- which can be opaque and suffer from GiveDirectly seeks to improve cost- effective strategy for reducing poverty.1 leakage and design challenges, and 2) effectiveness of its operating model, This is in large part due to a small but international philanthropy, for which the measure longer-run impacts, and growing body of research that shows founders of GiveDirectly aim to establish understand its influence on the larger significant positive welfare effects of direct transfers as the benchmark economy (e.g., market prices, new programs that provide cash to the poor against which all other approaches are business development, etc). However, with no strings attached. This subject measured. while building a robust research agenda sparked a debate in the international into the operations can be fruitful, it has development field about an alternative Since its inception, the leadership of its costs. GiveDirectly must make sure approach to improving social welfare, as GiveDirectly has been committed to that its constituents, or others in the compared to traditional forms of aid. rigorously measuring the impact of its development space, will benefit from the cash transfer program using randomized research in order to ensure their strategy GiveDirectly is a New York-based evaluations. Results from a randomized is responsible. nonprofit that provides secure direct cash evaluation implemented in Kenya found transfers to the extreme poor in Kenya the transfers increased investments by and Uganda primarily via mobile phone. 58 percent in assets such as livestock, Since it was founded in 2009, GiveDirectly furniture, and metal roofs, and also has grown rapidly: as of 2013, it had increased spending on goods including transferred $1.5 million to recipients, food, medical and educational expenses, with $3.4 million scheduled for delivery. and social events. There was no increase The organization has enrolled nearly in expenditures on temptation goods, 10,000 recipients to date and planned to such as alcohol and tobacco.2 However, move $20 million during FY2015. Beyond the study also found that transfers the immediate goal of transforming induced jealousy among neighbors.3 lives of the poor by providing direct cash transfers, the founders of GiveDirectly In this case study, we examine intend to influence two markets: GiveDirectly’s learning agenda following What They Do

GiveDirectly provides unconditional cash Typically, recipients receive an SMS alert GiveDirectly shares publically how transfers to the extreme poor in Kenya on their phones to notify them of the much of their budget goes directly and Uganda primarily via mobile phone. transfer and to let them know how they to the poor and how much is spent GiveDirectly’s model contrasts with the can collect the cash from a mobile money delivering the program. Based on our traditional approach to international aid, agent in their village or the nearest town. review of financial statements submitted which delivers specific goods and services GiveDirectly staff check with all recipients to GiveWell, GiveDirectly transfers a to the poor. to make sure they received the transfer high share of funds raised directly to and to assess their experience. After its recipients (close to 90 percent in The program targets families that live in receiving the full transfer amount, the both countries), relative to total costs houses with mud walls and a thatched household is no longer eligible for future (excluding set up costs). roof—these housing materials are transfers. a strong proxy for poverty in these As of early 2015, GiveDirectly has run locations.4 To select who will receive the The cash transfers are equivalent to nine transfer programs in total (seven in transfers, program staff first identify poor approximately one year’s household Kenya and two in Uganda), each with its areas using publically available statistics budget—a large amount compared own recipients, timeline, and objectives. on poverty. They then conduct a village- to a typical government cash transfer Of these programs, 14 campaigns have level census in these areas to collect data program. GiveDirectly’s transfer size was concluded, with all transfers sent and on poverty and enroll recipients. After the designed to be large enough to enable follow-up surveys completed. enrollment process, eligible households the poor to make larger purchases receive a series of cash transfers totaling and investments with the potential to approximately $1,0005 via mobile phones. generate income over time.

4 Theory of Change

GiveDirectly’s program is founded on in which they work. The organization decisions based on their individual needs the idea that large unconditional cash sends money to recipients via cell phones and preferences. The ultimate goal of transfers to the extreme poor should using electronic payment services that the program is to improve individual and improve their welfare. GiveDirectly are secure and transparent. Because household well-being. identifies and enrolls beneficiaries the cash transfers GiveDirectly provides by using objective criteria of housing are unconditional, they are expected material (mud and thatch) that correlate to empower poor households to make highly with poverty levels in the areas their own consumption and investment

Needs Activities Outputs Impact

• Improved individual and • Poor families lack resources to • Identify and enroll poor • Poor households are household well-being: make needed investments in households identified and enrolled education, food, and livelihoods • Increased assets and higher • Cash is transfered income • Traditional development • Send cash transfers • Improved mental health assistance fails to efficiently • Transfer is confirmed improve the welfare of the • Enhanced food security extreme poor • Conduct performance audits • Increased women’s empowerment

FIGURE 1. THEORY OF CHANGE* *Organizations use a variety of methods to present their theories of change. To standardize our discussion of these cases, we present our own simplified version of GiveDirectly’s theory of change here.

5 Activity Monitoring

GiveDirectly has a strong monitoring organizational priorities, and the is made of mud and has a thatched system that relies on mobile data total amount of funds available roof, a credible proxy for extreme collection and multiple quality controls to be allocated. Next, program poverty—to identify households to ensure that eligible households staff visit villages to introduce the eligible for the cash transfer are correctly identified, only eligible program, answer questions, and programs. households are enrolled into the garner support from community program, fraud and theft are minimized, leaders and local officials. 3. Registration. A separate field and eligible households receive the full GiveDirectly believes such visits team visits eligible households to amount of cash transfers. clarify the program’s goals and help enroll them into the program. In mitigate collusion or gaming the Kenya, enrollment involves giving a The program is implemented in several system. household member a SIM card used stages, with data collection and quality for transfers through the M-PESA controls at each step. 2. Village census. Next, field system6, if they do not already have staff conduct door-to-door visits an M-PESA account. The enrollment 1. Village selection. First, to collect basic demographic process is similar in Uganda. GiveDirectly identifies low-income information from everyone in the GiveDirectly staff may also help villages from national census village and take GPS coordinates eligible households to obtain ID data. Villages are selected based of the houses. The team uses cards or sign up for mobile money on a combination of census data, objective criteria–whether a house accounts.

6 4. Back-checks. An independent Uganda: In Kenya, the first transfer accountability and transparency: enumerator visits eligible is approximately $50, and the next the organization began signing households within the next month two transfers are split into two $475 memorandums of understanding to confirm eligibility and to inform payments, disbursed approximately with local government officials, the recipients of GiveDirectly’s two and six months later. In created a formal mechanism security measures to minimize Uganda, transfers are of equal to capture complaints during fraud. The enumerator also collects amounts and are sent monthly. the enrollment stage (such as personal identifying data and house GiveDirectly staff call all recipients households that felt they were GPS coordinates to compare them to confirm that the first transfers unfairly excluded from the with the village census data; any are received. program), and hired a senior field discrepancies are flagged for audit. manager to improve quality control, 6. Follow-up. Between two weeks among other changes. 4. Audits. Audits can be triggered and two months after the initial by mismatched names, GPS transfers, program staff conduct GiveDirectly publishes monthly records, photos, or a change in the short phone-based interviews performance indicators on its website, family structure that happened with the recipients to confirm the including the raw data files. The between the initial registration and transfer was received, inquire organization reports the number of the back-check. An independent about any problems, and record the households a field officer visits each team conducts audits to confirm recipient’s overall experience with day, the percentage of questions about household eligibility. the program. In Kenya, staff visit GiveDirectly that recipients answer vulnerable populations such as the correctly, the average time from census 5. Transfers. Typically, a recipient sick or elderly in person to collect to payment, and the percentage of receives an SMS alert when the these data. In Uganda, all recipients recipients reached by phone for follow- cash transfer is made into his or are visited in person. Collecting up. These numbers are reported against her mobile phone account. Cash feedback from the recipients targets set for each indicator. can be collected from a mobile has been crucial to program money agent in the recipient’s improvement.9 For example, in village or the nearest town.7 The response to a recent campaign in timing and amount of transfers Kenya, GiveDirectly made several varies slightly between Kenya and operational changes to improve

7 Measuring Impact

From the beginning, the founders of treatment conditions, and the other holdings by 58 percent, primarily GiveDirectly planned to test the impact half to control. The additional treatment through investments in livestock of the program using a rigorous impact conditions included whether the husband and the purchase of iron roofs. evaluation. In 2011, they partnered with or wife received the transfer; whether • Income gains from business and Innovations for Poverty Action to conduct the transfer was given in a single lump agriculture translated into a 28 an RCT of their program in Kenya when sum or nine monthly installments; and percent annual rate of return on it was first being rolled-out. To increase whether the transfer was large or small. transfers. transparency and prevent ‘cherry-picking’ In addition to measuring whether the • Significant increases in positive findings, GiveDirectly announced program improved recipient welfare, this expenditures in nearly every the study design and specified which two-level cluster-randomized evaluation category, except for tobacco, outcomes would be examined before allowed researchers to answer a richer alcohol, or gambling. they launched the study. set of questions about how to design the • A 42 percent reduction in the program for maximum impact. number of days children go without The evaluation included two stages: food. first, randomly selecting treatment and The findings from the impact evaluation • Recipients experienced large control villages from a list of 126 villages found positive effects on a range of reductions in stress and depression in a district; and second, randomly outcomes, including: and increases in life satisfaction, assigning half of eligible households measured with validated in treatment villages to one of three • Recipients increased asset psychological scales.

8 Interestingly, the study did not find that After the impact evaluation, GiveDirectly’s participants the flexibility to receive the gender of the recipient had any learning agenda focused on four main either lump-sum or monthly effect on the household response to questions: transfers improve their outcomes? the program. Additionally, there was no evidence of negative impacts on village 1) How can the existing model be Market-Wide and Long-Run crime or conflict, or increased spending even more cost-efficient? on alcohol and tobacco. 2) What are the long-run impacts of Impacts the program? The initial evaluation of the GiveDirectly Learning Agenda After 3) What are the impacts on local program demonstrated that markets? unconditional cash transfers can be very Impact Evaluation 4) How can the program’s effective for the individuals who receive The initial impact assessment validated operational and staff performance them. But questions remain about what the GiveDirectly model, making a be improved? happens to local markets as the program strong case for scaling up the program. scales and if the program’s positive However, a single study, even when Cost Effectiveness impact lasts over time. Answering well designed and credible, may not these questions will make an important answer all the questions an organization While GiveDirectly is considered to be contribution to the growing literature on may have and often raises new ones. highly cost-effective, it may be possible cash transfer and livelihoods programs. To answer these, an organization may to further reduce costs or increase the To address these questions, GiveDirectly deploy a learning agenda, which is a impact of the program. GiveDirectly will conduct two additional studies: structured approach to answering impact plans to explore how tweaking different and operational questions. A successful components of the program model • Impact on local markets. learning agenda generally requires the may improve its cost-effectiveness. The One major concern around same capacity needed to carry out a high- questions it seeks to answer include: GiveDirectly’s model is whether quality impact evaluation: the ability to large cash transfers alter economic identify and prioritize questions, develop • Messaging: How does framing the conditions in the local markets. research methods, identify data sources, messages (e.g., making them more Such effects could be positive, and assess available resources and aspirational) affect certain kinds of including job growth and increased internal capacity. spending? investment, or negative, such as • Timing of transfers: Does giving

9 increasing overall price levels, and efficiency. to deliver clear and consistent jealousy effects. GiveDirectly plans messages to the eligible to evaluate the effect of its program • Other targeting approaches: households during the enrollment on inflation and labor demand in Since using housing material process. the local markets. as a proxy for poverty will • Long-run impacts: Do the not be an appropriate in all • Quality control: Multiple checks positive effects of the transfer settings, GiveDirectly is testing ensure that only the poorest persist over time? To address this new, commonly used targeting households receive transfers, important question, GiveDirectly techniques. These include: 1) but the process is very costly. plans to conduct a separate having staff rate photographs of GiveDirectly would like to simplify study in which a sub-sample recipients and their homes on the quality control process without of participants from its Kenya a poverty scale; 2) using simple compromising program quality or RCT study will be re-surveyed to point-based surveys focused on increasing the risk of fraud. generate data on the longer-term verifiable criteria, such as asking effects of cash transfers. whether someone is a widow or has a toilet; and 3) experimenting Program and Staff with various forms of community- based targeting, such as having Performance communities nominate their own GiveDirectly’s learning agenda extends members in open, GiveDirectly- beyond formal impact evaluations using facilitated meetings. The RCTs. The organization is also evaluating organization will be piloting each different ways to improve program and of these methods and assessing staff performance: their cost, accuracy, and perceived fairness. • Using technology to identify poor villages: GiveDirectly has • Staff motivation and incentives: worked with volunteers to explore Staff may inadvertently influence how remote sensing technology how recipients spend cash. could streamline the process of GiveDirectly is exploring how locating poor villages in Kenya and different incentive structures for Uganda to increase operational field staff may encourage them

10 Goldilocks Recommendations

Overall, GiveDirectly has a well-developed can serve as a model for other programs. RCT was useful for making key decisions monitoring and evaluation system that Credibility requires accurate and reliable about the program, such as determining aligns well with the CART principles. data in addition to credible methods. the optimal size and frequency of cash Below we look at how the CART Based on our review of the organization’s transfers, as well as whether to target principles apply and discuss areas for data collection processes, GiveDirectly women or men. improvement. has instituted multiple quality checks to ensure that it targets the poorest Responsible: Ensure the households. Credible: Collect high benefits of data collection quality data and accurately Actionable: Commit to act outweigh the costs. analyze the data. on the data you collect. Part of GiveDirectly’s learning agenda Since its inception, GiveDirectly has consists of evaluating options to lower been committed to conducting credible GiveDirectly views RCTs as a tool not the costs of data collection without impact evaluations of its program only to rigorously measure impact of sacrificing the quality. Finding this using randomized control trials. The its core program, but also to answer balance is tricky. For example, selecting organization’s ability to work together critical operational questions with the poor families in each village is a key with researchers to measure impact and same degree of rigor. GiveDirectly has component of the program, and a costly answer important operational questions worked closely with an external research one. GiveDirectly is exploring how careful team to ensure that the design of its

11 use of technology could make village administrative costs. By making this selection more efficient. The organization information available on its website, the is also assessing whether the resources organization offers an opportunity for spent on conducting multiple checks on others designing cash transfer programs, eligible households are truly necessary, or looking to support them, to learn from or whether the process can be simplified GiveDirectly’s experience. GiveDirectly without increasing fraud. also provides a literature review on other cash transfer programs on its website. Transportable: Collect To ensure that others can more easily learn from GiveDirectly’s experience, we data that will generate recommend the organization develop knowledge for other a more explicit theory of change that programs. explains step-by-step how the program achieves the desired social impact. It Unconditional cash transfer programs are should include the theory’s underlying a promising new approach to reducing risks (for example, that increased poverty and have generated a great deal tension may occur within targeted of controversy. Rigorous evaluation of communities), and its key assumptions GiveDirectly’s program has helped shape (that poor households will spend money the debate about cash programs and productively, rather then on tobacco and broader policy implications. The data alcohol). have raised new important questions— such as whether positive effects are sustained in the long term, and whether and how scale-up will influence local market conditions—paving the way for future research in this area. GiveDirectly has publically shared the results of its research and is transparent about its operating model and its

12 Lessons for Others

1. Ensure evaluations 2. Invest adequately when 3. Seek help from experts provide actionable data to conducting operational when needed. inform program design. research. Conducting a study to estimate economy- A credible impact evaluation is not Conducting good operational wide impact of a program is significantly an end in and of itself – it often raises research is complicated and requires more difficult than conducting a single new questions and helps inform systems similar to those needed for high- impact evaluation. Few organizations an organization’s learning agenda. quality impact evaluations. Negotiating have the capacity and technical Operational research can help an and supervising external resources sophistication to study economy-wide organization improve its programs and also requires expertise and capacity. responses to their programs – credibly become more cost-efficient, but it must An organization developing its learning conducting such a study will likely require be thoughtfully developed. This includes agenda must consider how to develop external research support from a skilled planning for what will be evaluated and sufficient capacity to invest in high quality economist. which methods will be used, defining internal R&D and, when the time is right, actions to be taken as a result of the to engage external support. findings, and setting timelines and targets for determining success.

13 Endnotes 1. GiveWell. (2012). Cash transfers in the developing world. Available at: http://www.givewell.org/international/ technical/programs/cash-transfers. 2. Haushofer, J., & Shapiro, J. (2013). Household response to income changes: Evidence from an unconditional cash transfer program in Kenya. Available at: https://www.princeton.edu/~joha/publications/Haushofer_ Shapiro_UCT_2013.pdf. 3. Haushofer, J., Reisinger, J., & Shapiro, J. (2015) Your Gain Is My Pain: Negative Psychological Externalities of Cash Transfers. Available at: https://www.princeton.edu/~joha/publications/Haushofer_Reisinger_Shapiro_ Inequality_2015.pdf. 4. As of December 2015, Give Directly has changed its eligibility critieria in order to better capture the poorest households. The new criteria take into account a range of factors including a household’s assets and vulnerability status. The details are confidential to prevent households from gaming the system. 5. The process varies slightly across countries: in Kenya, eligible households receive one small token transfer, followed by two equally sized lump sum payments, whereas in Uganda they receive transfers monthly. In Kenya, recipients receive SMS notifications of the transfer and collect it from a local M-Pesa agent, typically a shopkeeper in the recipient’s village or in the nearest town. In Uganda the process is slightly different because the infrastructure of mobile agents is not as well-developed as in Kenya. 6. If a household does not have a cell phone, they can purchase the device directly from GiveDirectly at the time of enrollment, and the cost of the phone is subtracted from the recipient’s transfer. 7. According to GiveDirectly’s website, on average it takes recipients 32 minutes round-trip to collect their transfers from the closest agent. 8. GiveDirectly maintains a phone “hotline” for recipients to call if they have any questions about the transfers or experience any problems. In the case that hotline calls or follow up surveys reveal an adverse event (e.g., violence, crime, domestic conflict, bribes), staff help resolve issues by giving advice, calling neighbors or family members, bringing in local leaders for counseling, or involving police 9. A longer follow-up survey collecting detailed information on household expenditures is administered to 10% of the recipients over the phone. In Kenya, recipients are called twice (after each lump sum). In Uganda, GiveDirectly plans to conduct these follow-up surveys three times over the ten monthly transfers, and is experimenting with conducting them in-person rather than over the phone, with shorter calls to confirm that the transfer was received and ask if the recipient experienced any problems. Tracking outcomes in this case is useful for learning about the general pattern and composition of household expenditures after the funds are received.

14