F A I R F A X C O U N T Y D E P A R T M E N T O F M A N A G E M E N T A N D B U D G E T

October 2017

Informational Brochure

Overview of Sampling Procedures

The purpose of this brochure is to provide an overview of the sampling Table of Contents procedures available to a researcher. The differences between the various

Why Sample 1 sampling procedures are discussed and examples are provided to illustrate the use of these procedures. The emphasis of this manual is placed on Sampling 2 underlying ideas and methods rather than detailed mathematical derivations. Methodologies For the reader who is interested in pursuing a more thorough approach to the Nonprobability 3 topics discussed, a list of technical references is provided in the box below.

Probability Samples 4 Summary 6 Why Sample Glossary 7 If a researcher desires to extremely expen- obtain information about a sive, difficult, and Technical References: population through time consuming. A American Statistical Association questioning or testing, properly designed 732 North Washington Street he/she has two basic probability sample, Alexandria, VA 22314-1943 703-684-1221 · 888-231-3473 options: however, provides www.amstat.org a reliable means of 1. Every member of the inferring infor- American Association for Public population can be Opinion Research questioned or tested, mation about a population without examining One Parkview Plaza, Suite 800 a census; or every member or element. Oakbrook Terrace, IL 60015 2. A sample can be 1-847-686-2230 conducted; that is, Often, researchers are working under strict www.aapor.org only selected time constraints which make conducting a cen- members of the sus unwieldy. For instance, national polling Insights Association population are firms frequently must provide information on 170 N. Country Road, Suite 4 questioned or tested. Port Jefferson, NY 11777 the public's perceptions of current events or 202-800-2545 Contacting, questioning, issues. These polling firms tend to limit their www.insightsassociation.org and obtaining information national sample sizes to approximately 1,500 from a large population, respondents. When properly conducted, a Federal Committee on such as all of the Statistical Methodology probability sample of this size provides relia- Statistical Policy Working Papers households residing in ble information with a very small margin of er-

fcsm.sites.usa.gov Fairfax County, is ror for the whole population of the United Page 2 Overview of Sampling Procedures

States, which is more than 300 data processing and analysis operating. A relatively long million persons. errors. In part, these and difficult questionnaire can nonsampling errors are be administered to a sample A probability sample reduced through pretesting more easily than a brief frequently is more accurate which allows careful testing of questionnaire can be than a census of the entire the survey questionnaire and administered to the entire population. The smaller procedures. Pretesting cannot population. However, not all sampling operation lends itself be done when conducting a samples are accurate or the to the application of more census without causing appropriate vehicle for rigorous controls, thus possible contamination of some gathering information or ensuring better accuracy. of the respondents. The detail testing a hypothesis about a These rigorous controls allow of information that can be population. The following the researcher to reduce asked in a sample is greater sections of this brochure will nonsampling errors such as than that in a census due to the briefly discuss the merits and interviewer bias and mistakes, cost and time constraints under disadvantages of various nonresponse problems, which most researchers are sampling procedures. questionnaire design flaws, and

Sampling Methodologies mates ob- Sampling methodologies are classified under tained from two general categories: the sample and to specify the sam- 1. Probability sampling and pling error. 2. Nonprobability sampling. Nonprobability samples, In the former, the researcher knows the exact in contrast, do not allow the study's findings to possibility of selecting each member of the pop- be generalized from the sample to the popula- ulation; in the latter, the chance of being includ- tion. When discussing the results of a nonprob- ed in the sample is not known. A probability ability sample, the researcher must limit his/her sample tends to be more difficult and costly to findings to the persons or elements sampled. conduct. However, probability samples are the This procedure also does not allow the re- only type of samples where the results can be searcher to calculate sampling statistics that generalized from the sample to the population. In provide information about the precision of the addition, probability samples allow the re- results. The advantage of nonprobability sam- searcher to calculate the precision of the esti- pling is the ease in which it can be adminis- Page 3

tered. Nonprobability samples tend to be less complicated and less time consuming than probability samples. If the researcher has no intention of generalizing beyond the sample, one of the nonproba- bility sampling methodologies will provide the desired information.

Nonprobability Samples

The three common types of collecting information can nonprobability samples are be reduced. convenience sampling, quota B. Quota Sampling sampling, and judgmental sampling. Quota sampling is often confused A. Convenience Sampling with stratified As the name implies, and cluster convenience sampling involves sampling—two choosing respondents at the probability sampling convenience of the researcher. methodologies. All of these Examples of convenience methodologies sample a additional respondents that samples include people-in-the- population that has been would have fallen into these street interviews—the subdivided into classes or classes are rejected or sampling of people to which categories. The primary excluded from the results. the researcher has easy access, differences between the An example of a quota sample such as a class of students; and methodologies is that with would be a survey in which the studies that use people who stratified and cluster sampling researcher desires to obtain a have volunteered to be the classes are mutually certain number of respondents questioned as a result of an exclusive and are isolated from various income advertisement or another type prior to sampling. Thus, the categories. Generally, of promotion. A drawback to probability of being selected is researchers do not know the this methodology is the lack of known, and members of the incomes of the persons they sampling accuracy. Because population selected to be are sampling until they ask the probability of inclusion in sampled are not arbitrarily about income. Therefore, the the sample is unknown for each disqualified from being researcher is unable to respondent, none of the included in the results. In subdivide the population from reliability or sampling quota sampling, the classes which the sample is drawn into precision statistics can be cannot be isolated prior to mutually exclusive income calculated. Convenience sampling and respondents are categories prior to drawing the samples, however, are categorized into the classes as sample. Bias can be employed by researchers the survey proceeds. As each introduced into this type of because the time and cost of class fills or reaches its quota, sample when the respondents Page 4 Overview of Sampling Procedures

who are rejected, because the class to which within two miles of the new facility. Expert they belong has reached its quota, differ from judgment, based on past experience, indicates those who are used. that most of the use of this type of facility comes from persons living within two miles. However, C. Judgmental Sampling by limiting the sample to only this group, usage In judgmental or purposive sampling, the projections may not be reliable if the usage researcher employs his or her own "expert” characteristics of the new facility vary from those judgment about who to include in the sample previously experienced. As with all frame. Prior knowledge and research skill are nonprobability sampling methods, the degree used in selecting the respondents or elements to and direction of error introduced by the be sampled. researcher cannot be measured and statistics An example of this type of sample would be a that measure the precision of the estimates study of potential users of a new recreational cannot be calculated. facility that is limited to those persons who live

Probability Samples Four basic types of methodolo- round. Samples may be drawn gies are most commonly used with or without replacement. In for conducting probability sam- practice, however, most simple ples; these are simple random, random sampling for survey re- stratified, cluster, and systemat- search is done without replace- ic sampling. Simple random ment; that is, a person or item sampling provides the base selected for sampling is re- be sampled. from which the other more com- moved from the population for An example of a simple random plex sampling methodologies all subsequent selections. At sample would be a survey of are derived. any draw, the process for a sim- County employees. An exhaus- ple random sample without re- A. Simple Random Sampling tive list of all County employees placement must provide an as of a certain date could be ob- To conduct a simple random equal chance of inclusion to any tained from the Department of sample, the researcher must member of the population not Human Resources. If 100 names first prepare an exhaustive list already drawn. To draw a sim- were selected from this list us- (sampling frame) of all mem- ple random sample without in- ing a random number table or a bers of the population of inter- troducing researcher bias, com- computerized sampling pro- est. From this list, the sample is puterized sampling programs gram, then a simple random drawn so that each person or and random numbers tables are sample would be created. Such item has an equal chance of be- used to impartially select the a random sampling procedure ing drawn during each selection members of the population to Page 5

has the advantage of reducing differences in income among bias and enables the researcher the regions or groupings are to estimate sampling errors and greater than the income differ- the precision of the estimates ences within the regions or derived through statistical cal- groupings, precision of the esti- culations. mates is improved. In addition, if the research organization has B. Stratified Random Sam- branch offices located in these pling regions, the administration of Stratified random sampling in- the survey can be decentralized volves categorizing the mem- and perhaps conducted in a bers of the population into mu- more cost-efficient manner. cedure is defined as two-stage tually exclusive and collectively cluster sampling. C. Cluster Sampling exhaustive groups. An inde- Cluster sampling is frequently pendent simple random sample Cluster sampling is similar to employed when the researcher is then drawn from each group. stratified sampling because the is unable to compile a compre- Stratified sampling techniques population to be sampled is hensive list of all the elements in can provide more precise esti- subdivided into mutually exclu- the population of interest. A mates if the population being sive groups. However, in clus- cluster sample might be used surveyed is more heterogene- ter sampling the groups are de- by a researcher attempting to ous than the categorized fined so as to maintain the het- measure the age distribution of groups, can enable the re- erogeneity of the population. It persons residing in Fairfax searcher to determine desired is the researcher’s goal to es- County. It would be much more levels of sampling precision for tablish clusters that are repre- difficult for the researcher to each group, and can provide sentative of the population as a compile a list of every person administrative efficiency. whole, although in practice this residing in Fairfax County than may be difficult to achieve. Af- An example of a stratified sam- to compile a list of residential ter the clusters are established, ple would be a sample conduct- addresses. In this example, a simple random sample of the ed to determine the average each address would represent a clusters is drawn and the mem- income earned by families in cluster of elements (persons) to bers of the chosen clusters are the United States. To obtain be sampled. If the elements sampled. If all of the elements more precise estimates of in- contained in the clusters are as (members) of the clusters se- heterogeneous as the popula- come, the researcher may want lected are sampled, then the tion, then estimates derived to stratify the sample by geo- sampling procedure is defined from cluster sampling are as graphic region (northeast, mid- as one-stage cluster sampling. precise as those from simple Atlantic, et cetera) and/or strati- If a random sample of the ele- random sampling. However, if fy the sample by urban, subur- ments of each selected cluster is the heterogeneity of the clusters ban, and rural groupings. If the drawn, then the sampling pro- Page 6 Overview of Sampling Procedures

is less than that of the population, the estimates One of the most attractive aspects of systematic will be less precise. sampling is that this method can allow the re- searcher to draw a probability sample without D. Systematic Sampling complete prior knowledge of the sampling frame. Systematic sampling, a form of one-stage cluster For example, a survey of visitors to the County's sampling, is often used in place of simple random publications desk could be conducted by sam- sampling. In systematic sampling, the researcher pling every 10th visitor after randomly selecting selects every nth member after randomly select- the first through 10th visitor as the starting point. ing the first through nth element as the starting By conducting the sample in this manner, it would point. For example, if the not be necessary for the re- researcher decides to sam- searcher to obtain a compre- ple every 20th member of hensive list of visitors prior to the population, a 5 percent drawing the sample. sample, the starting point for As with other types of cluster the sample is randomly se- sampling, systematic sampling lected from the first 20 mem- is as precise as simple random bers. A systematic sample is sampling if the members con- a type of cluster sample be- tained in the clusters are as het- cause each of the first 20 erogeneous as the population. members of the sampling If this assumption is not valid, frame defines a cluster that contains 5 percent of then systematic sampling will be less precise than the population. simple random sampling. In conducting system- A researcher may choose to conduct a systematic atic sampling, it is also essential that the re- sample instead of a simple random sample for sev- searcher does not introduce bias into the sample eral reasons. Systematic samples tend to be easi- by selecting an inappropriate sampling interval. er to draw and execute. The researcher does not For instance, when conducting a sample of finan- have to jump backward and forward through the cial records, or other items that follow a calendar sampling frame to draw the members to be sam- schedule, the researcher would not want to select pled. A systematic sample may spread the mem- “7” as the sampling interval because the sample bers selected for measurement more evenly would then be comprised of observations that across the entire population than simple random were all on the same day of the week. Day-of-the- sampling. Therefore, in some cases, systematic week influences may cause contamination of the sampling may be more representative of the pop- sample, giving the researcher biased results. ulation and more precise. Summary

Sampling can be a powerful tool for accurately measuring opinions and characteristics of a population. However, there is a genuine potential for misuse of this tool by researchers who do not understand the Page 7

limitations of various sampling procedures. The differences between nonprobability and probability sampling procedures are often difficult to discern but are extremely important for determining how the results of the research can be used. Nonprobability sampling techniques can provide valuable infor- mation but the results cannot be generalized to a larger population nor can statistics indicating the relia- bility of the results be calculated. Well conducted probability samples provide the researcher with the ability to gather information from a relatively small number of members of a large population and accu- rately generalize the results to the entire population. In addition, probability samples enable the re- searcher to calculate statistics that indicate the precision of the data.

Glossary

 Bias (error): Distorted or unreliable survey results. All surveys contain some bias. Bias is increased when the respondents (persons answering the survey) are not representative of the population being questioned, when questions are poorly written or misunderstood, and when the researcher uses inappropriate techniques to analyze the data. household, car, dog, et  Instrument: The tool or device cetera). used for survey measurement;  Census: A study using all usually a questionnaire. available elements  Heterogeneous: A population (members) of a population. whose elements have  Nonresponse: Unit dissimilar characteristics. nonresponse refers to the  Data: The collection of Heterogeneity is the state of refusal of persons selected to observations and being dissimilar. be sampled to participate in a information resulting from survey (i.e., person does not the survey process.  Homogeneous: A population return the mail questionnaire). whose elements have similar  Element (member): The Item nonresponse refers to characteristics. Homogeneity basic unit about which selected questions left as the state of being similar. survey information is sought unanswered by the person (i.e., person, business, surveyed. Page 8 Overview of Sampling Procedures

 Population: The universe or collection of all  Sampling Frame: An exhaustive list of all elements (persons, businesses, et cetera) members of the population from which a being described or measured by a sample. sample can be drawn.

 Pretest: An initial evaluation of the survey  Survey: A process of inquiry for the purpose design by using a small, subsample of the of data collection and analysis using intended population for preliminary observation, polls, questionnaires, and/or information. interviews.

 Questionnaire: A measuring device used to query a population/sample in order to obtain information for analysis.

 Respondent: An element or member of the population selected to be sampled.

 Sample: Any portion of the population, less than the total.

Fairfax County Department of Management and Budget

Economic, Demographic and Statistical Research

12000 Government Center Parkway Fairfax, VA 22035 703-324-2391 fairfaxcounty.gov/demographics

Fairfax County is committed to nondiscrimination in all county programs,

services and activities. To request reasonable accommodations or to re- ceive this information in an alternate format, call 703-324-4600, TTY 711.

A Fairfax County, VA, publication