Participant-Based Survey Sampling Guide for Feed the Future Annual Monitoring Indicators Diana Maria Stukel, PhD September 2018 This guide is made possible by the generous support of the American people through the support of the Office of Maternal and Child Health and Nutrition, Bureau for Global Health, U.S. Agency for International Development (USAID), USAID Bureau for Food Security and USAID Office of Food for Peace, under terms of Cooperative Agreement No. AID-OAA-A-12-00005, through the Food and Nutrition Technical Assistance III Project (FANTA), managed by FHI 360. The contents are the responsibility of FHI 360 and do not necessarily reflect the views of USAID or the United States Government. September 2018 FANTA III Recommended Citation FOOD AND NUTRITION TECHNICAL A SSISTANCE Stukel, Diana Maria. 2018. Participant-Based Survey Sampling Guide for Feed the Future Annual Monitoring Indicators. Washington, DC: Food and Nutrition Technical Assistance Project, FHI 360. The citation for the original edition of this guide is: Stukel, Diana Maria and Friedman, Gregg. 2016. Sampling Guide for Beneficiary-Based Surveys for Select Feed the Future Agricultural Annual Monitoring Indicators. Washington, DC: Food and NutritionT echnical Assistance Project, FHI 360. Contact Information Food and NutritionT echnical Assistance III Project (FANTA) FHI 360 1825 Connecticut Avenue, NW Washington, DC 20009-5721 www.fantaproject.org Cover photo credit: Nishanth Dangra, courtesy of Photoshare Acknowledgments The author would like to thank Anne Swindale and Arif Rashid for their invaluable comments, suggestions, and insights on an earlier draft of this second edition of the guide. I am also indebted to Jeff Feldmesser for his careful editing of the manuscripts and to the FANTA Communications Team for transforming the guide into a final professional product. Participant-Based Survey Sampling Guide for Feed the Future Annual Monitoring Indicators i Contents Acknowledgments ................................................................................................................................ i Abbreviations and Acronyms ............................................................................................................... v PART 1 1. Purpose and Background ............................................................................................................... 2 1.1 Purpose of the Sampling Guide on Participant-Based Surveys ...................................................... 2 1.2 Background ..................................................................................................................................... 3 2. The Four Selected Feed the Future Agriculture-Related Annual Monitoring Indicators ................... 6 2.1 The “Yield of Agricultural Commodities” Indicator ........................................................................ 8 2.2 The “Number of Hectares under Improved Management Practices” Indicator ............................ 9 2.3 The “Value of Sales” and “Number of Individuals Using Improved Management Practices” Indicators ........................................................................................................................................ 9 PART 2 3. Comparison of Routine Monitoring and Participant-Based Surveys .............................................. 11 3.1 Overview ...................................................................................................................................... 11 3.2 Description and Features of the Approaches ............................................................................... 11 3.2.1 Routine Monitoring........................................................................................................... 11 3.2.2 Participant-Based Surveys ................................................................................................ 12 3.3 Advantages of Each Approach ...................................................................................................... 12 3.3.1 Advantages of Routine Monitoring ................................................................................... 12 3.3.2 Advantages of Participant-Based Surveys ........................................................................ 13 4. When Are Participant-Based Surveys Appropriate? ...................................................................... 14 4.1 Scenario #1: Large Project Size/Inadequate Number of Data Collection Staff ............................ 14 4.2 Scenario #2: Farmer Estimates of Area Considered Unreliable and Direct Measurement of Plots Preferred ......................................................................................................................... 15 4.3 Scenario #3: Lack of Direct Contact between a Project and Its Participants ............................... 15 PART 3 5. Timing and Frequency of Participant-Based Survey Data Collection ............................................. 17 6. Issues to Consider When Outsourcing Work to an External Contractor ......................................... 19 6.1 Time and Effort Required to Procure and Manage an External Contractor ................................. 20 6.2 Importance of a Good Scope of Work to Guide the Process ....................................................... 20 6.3 Judging the Expertise of Potential External Contractors.............................................................. 21 Participant-Based Survey Sampling Guide for Feed the Future Annual Monitoring Indicators ii PART 4 7. Sampling Frame Guidance for Participant-Based Surveys ............................................................. 23 7.1 Information to Include on a Sampling Frame .............................................................................. 23 7.2 Participant Registration Systems as a Source of Establishing Sampling Frames .......................... 25 7.3 Frames for Multiple Participant-Based Surveys Conducted in the Same Year ............................ 26 8. Overview of Various Approaches for Collecting Annual Monitoring Data Using Participant- Based Surveys .............................................................................................................................. 27 8.1 Approach 1: Household Survey Approach ................................................................................... 27 8.2 Approach 2: Producer Groups Approach ..................................................................................... 28 8.3 How to Choose the Right Approach ............................................................................................. 29 8.4 Details on the Two Approaches ................................................................................................... 29 PART 5 9. The Household Survey Approach (Approach 1) ............................................................................ 32 9.1 Choosing a Survey Design Option ................................................................................................ 32 9.1.1 Survey Design Option 1: Two-Stage Cluster Design with Systematic Selection of Participants ....................................................................................................................... 32 9.1.2 Survey Design Option 2: Two-Stage Cluster Design with a Listing Operation and Systematic Selection of Participants ................................................................................. 33 9.1.3 Survey Design Option 3: One-Stage Design with Systematic Selection of Participants ... 34 9.1.4 Summary of the Recommended Survey Design Options under the Household Survey Approach ........................................................................................................................... 35 9.1.5 A Cautionary Note on the Use of Lot Quality Assurance Sampling .................................. 35 9.2 Calculating the Sample Size for All Survey Design Options of the Household Survey Approach ...................................................................................................................................... 36 9.2.1 Types of Surveys and Indicators ....................................................................................... 37 9.2.2 Calculating the Sample Size .............................................................................................. 38 9.2.3 Adjustments to the Sample Size Calculation .................................................................... 44 9.2.4 Final Sample Size ............................................................................................................... 46 9.2.5 Determining the Overall Sample Size for the Survey ........................................................ 49 9.2.6 Updating Elements of the Sample Size Formula in Future Survey Rounds ...................... 53 9.3 Choosing the Number of Clusters to Select for Survey Design Options 1 and 2 of the Household Survey Approach ........................................................................................................ 54 9.4 Selecting a Sample of Clusters for Survey Design Options 1 and 2 for the Household Survey Approach ...................................................................................................................................... 57 9.4.1 Systematic PPS Sampling .................................................................................................. 57 9.4.2 Fractional Interval Systematic Sampling ..........................................................................