Revised Using Statistical May 1992 Sampling GAWPEMD-10.1.6 Preface GAO assists congressional decisionmakers in their deliberative process by furnishing analytical information on issues and opinions under consideration. Many diverse methodologies are needed to develop sound and timely answers to the questions that are posed by the Congress. To provide GAO evaluators with basic information about the commonly used methodologies, GAO’s policy guidance includes documents such as methodology transfer papers and technical guidelines. The purpose of this methodology transfer paper on statistical sampling is to provide its readers with a background on sampling concepts and methods that will enable them to identify jobs that can benefit from statistical sampling, to know when to seek assistance from a statistical sampling specialist, and to work with the specialist to design and execute a sampling plan. This paper describes sample design, selection and estimation procedures, and the concepts of confidence and sampling precision. Two additional topics, treated more briefly, include special applications of sampling to auditing and evaluation and some relationships between sampling and data collection problems. Last, but not least, the strengths and limitations of statistical sampling are summarized. The original paper was authored by Harry Conley and Lou Fink in April 1986. This reissued version supersedes the earlier edition. Using Statistical Sampling is one of a series of papers issued by the Program Evaluation and Methodology Division (PEMD). The purpose of the series is to provide GAO evaluators with guides to various aspects of audit and evaluation methodology, to illustrate applications, and to indicate where more detailed information is available. Page 1 GAO/PEMD-10.1,6 Statistical Sampling Preface We look forward to receiving comments from the readers of this paper. They should be addressed to Eleanor Chelimsky at 202-275-1854. Werner Grosshans I Assistant Comptroller General Office of Policy -w I Eleanor Chelimsky Assistant Comptroller General for Program Evaluation and Methodology Page 2 GAO/PEMD-10.1.6 Statistical SumpIining Page 3 GAO/PEMD-10.1.6 Statistical Sampling Contents Preface Chapter 1 10 Introduction Evaluation Design 11 Sample Design As an Element of an Assignment 13 Design Representativeness: The Goal of Statistid 14 sampling Random Selection 15 Distinguishing Between Samples and Examples and 16 Between Sampling Operations The Organization of This Paper 17 Chapter 2 19 A Review of Basic Statistks: A Shorthand Description of a Population 20 Measurement Levels of Data 20 Concepts Central Tendency 22 Dispersion 26 Some Comments on Looking Summary statistics in 29 the Eye Chapter 3 30 An Overview of Sample Design 30 Selection Procedures 41 Sample Design Estimation Procedures 42 and Selection and Three Statistical Sampling Methods 43 Estimation Procedures Chapter 4 47 Basic Estimation The Concepts of Precision and Confidence 43 Procedures and Variable Sampling 51 Calculating Sample Size 57 F’urther Sample Attribute Sanmlinn 62 Design A Distinction Be&ken Precision and Accuracy 67 Considerations Page4 GAO/PEMD- 10.1.6 Statistical Sampling Chapter 5 69 Advanced Ratio Estimation 69 Regression Estimation 70 Estimation Difference Estimation 71 Procedures Example of Advanced Estimation Procedures 71 Which Method to Use 80 Other Advantages and Disadvantages of Ratio, 80 Regression, and Difference Estimation Chapter 6 82 Sampling in the Discovery Sampling 82 Au& Environment Acceptance saPh3 88 Audit Judgment and Statistical Sampling 92 The Characteristics of a Good Sample 93 Chapter 7 96 Random Selection The Selection of Sampling Units 97 Procedures The Application of Selection Procedures 104 Selection With Probability Proportional to Size 105 A Final Check 108 Chapter 8 109 Data Collection Checking the Quality of Data Collection 110 and Analysis The Problem of Nonresponse 112 Considerations Related to sampling Chapter 9 116 Statistical Sampling As a Friend of Evaluators Page 6 GAO/PEMD-10.1.6 Statistical Sampling Content5 Appendixes Appendix I: Theoretical Background 122 Appendix II: A Comprehensive Description of 144 Sampling Procedures Appendix III: Details on Stratified and Cluster 180 Sampling Appendix IV: Computer Software Packages 214 Bibliography 221 Glossary 225 Papers in This 238 Series Tables Table 1.1: Four Strategies and Their Evaluation Questions Table 2.1: A Frequency Distribution 19 Table 2.2: Days of Downtime for 10 Computers 22 Table 2.3: Data for Measures of Dispersion 28 Table 2.4: Interpreting the Value of the Coefficient 29 of Variation Table 4.1: ExampIe of Sample Values 53 Table 4.2: Data for Figure 4.1 62 Table 5.1: Data for Advanced Estimation 72 Procedures Example Table 6.1: The Probability of Finding at Least One 84 Error in a Population of 500 for Various Numbers of Errors and Sample Sizes Table 6.2: The Probability of Finding at Least One 86 Error in a Population of 600 for Various Numbers of Errors and Sample Sizes Table 6.3: An Acceptance Sampling Table 90 Table 7.1: Selection With Probability ProportionaI 106 to Size Table 8.1: Tabulation of Responsesto Question on 112 Abolition of the Death Penalty Table 8.2: Type of Sentence Respondents Received 113 Page 6 GAOIPEMD-10.1.6 Statistical Sampling Contents Table I. 1: Dice-Throwing Probabilities 123 Table 1.2: Number and Percent of Red Beads 123 Found in 400 Random Samples of 400 From a Population of 20,000 Containing 4,000 Red Beads Table 1.3: The Relative Frequency of Means of 129 Samples of Two From a Population of Digits Between 0 and 9 Table 1.4: The Relative Frequency of Means of 131 Samples of Three From a Population of Digits Between 0 and 9 Table I.5: The Amounts of 100 Small Purchases I37 Table I.6: Table oft Factors 143 Table II. 1: Random Decimal Digits 146 Table 11.2:Selecting Random Numbers and 157 Random Letters in Compound Numbering Systems With the Same Quantity of Items for Each Letter Table 11.3:Matching Random Numbers and 159 Random Letters in Compound Numbering System With the Same Quantity of Items for Each Letter Table 11.4:Sorting Random Number-Letter 160 Combinations in Compound Numbering Systems With the Same Quantity of Items for Each Letter Table 11.5:Selecting Random Numbers and 162 Random Letters in Compound Numbering Systems With a Different Quantity of Items for Each Letter Table 11.6:Matching Random Numbers and 164 Random Letters in Compound Numbering Systems With a Different Quantity of Items for Each Letter Table II.7: Sorting Random Letter-Number 166 Combinations in Compound Numbering Systems With a Different Quantity of Items for Each Letter Table 11.8:A Population of Records in Four 170 Groups Page 7 GAO/PEMD-10.1.6 Statistical Sampling Contents Table 11.9:Selecting a Single Systematic Sample 17 1 From Four Groups of Records Table III. 1: Air Freight Forwarder Shipping Costs 182 Table 111.2:Savings From Using Direct Air 182 Shipment Instead of Air Freight Forwarder Classified by Air Freight Forwarder Shipping Costs of Less Than $100 Table III. 3: Savings From Using Direct Air 188 Shipment Instead of Air Freight Forwarder Classified by Air Freight Forwarder Shipping Costs of More Than $100 Table III.4: Preliminary Sample Selection 192 Table III.5: Calculated Savings 193 Table 111.6:Final Sample Items Under Neyman 196 Allocation Table 111.7:Sample Results Under Judgmental 198 Allocation Table 111.8:Independent Random Samples of 201 Payroll Records Table 111.9:Assumed Air Freight Forwarder Costs 204 Table III. 10: Allocation of a Sample of 50 205 Table III. 11: Frequency Distribution on 207 Transaction Dollar Amounts Table III. 12: Dollar Amount Multiplied by Strata 208 Numbers Table III. 13: Strata Boundaries 208 Table III. 14: Random Number Sample 210 Figures Figure 4.1: Final Sample Size As Population Size 61 Increases Figure 4.2: Sample Data for Supply Depot Example 63 Figure 4.3: Sample Data for Supply Depot Example 66 for a Given Precision Figure 5.1: Computer Output for Advanced 74 Estimation Procedures Example Figure 5.2: Statistics for Advanced Estimation 78 Procedures Example Figure I. 1: Number of Samples Classified by 126 Percentage of Red Beads Page 8 GAO/PEMD-10.1.6 Statistical Sampling Content8 Figure I.2: The Relative Frequency of a Large 128 Population of Single Digits Between 0 and 9 Figure I.3: The Relative Frequency of Means of 130 Samples of Two From a Population of Digits Between 0 and 9 Figure 1.4: The Relative Frequency of Means of 132 Samples of Three From a Population of Digits Between 0 and 9 Figure 1.5: The Smoothed Curve of the Relative 133 Frequency of Means of Samples of Three From a Population of Digits Between 0 and 9 Figure 1.6: Frequency Distributions of the Means 135 of Samples Drawn From 228 U.S. Cities With Populations Greater Than 50,000 in 1950 Figure III. 1: Results From SRO-STATS 195 Calculation Abbreviations FTC Finite population correction GAO U.S. General Accounting Office ICC Interstate Commerce Commission LTPD Lot tolerance percent defective PEMD Program Evaluation and Methodology Division PPS Probability proportional to size Page 9 GAO/PEMD-10.1.6 Statistical Sampling Chapter 1 Introduction Sampling is a very important element in the design and planning of a job here at GAO as well as any evaluation. The purpose of this paper is to help GAO managers and evaluators learn more about statistical (or probability) sampling and the role it plays in the design and execution of a job. We have attempted TV take the mystery out of what is often thought of as an esoteric subject by “walking” the reader through the various sampling procedures. In this document, we have chosen to describe the computations and sample selection procedures as they are normally done by computer in GAO, but formulas necessary to do the necessary calculations by hand can be found in any sampling textbook.