Sampling One/All Persons Per Household: Pros and Cons
Total Page：16
File Type：pdf, Size：1020Kb
Sampling one/all persons per household: pros and cons1
This paper will discuss how many persons should be interviewed per household in the EU Safety Survey (EU-SASU), particularly whether one or all members should be interviewed in every sampled household. As we’ll see, the choice has implications for data quality, mainly sampling variance, non-response rate and measurement errors, and for the overall cost of the survey. Because statistical accuracy and cost efficiency are often conflicting requirements, a tradeoff will have to be made. The paper starts with a review of some technical aspects in relation to selecting and interviewing all the members of a household instead of one. We then establish variance formulas in order to compare the efficiency of the two options. Finally, numerical examples using the data from the US National Crime and Victimization Survey (NCVS) will help to illustrate our findings.
Technical aspects of interviewing one person in a household versus two or more
Most surveys on crime and victimization employ multi-stage designs whereby a sample of households is drawn using any conventional sampling design (simple random sampling without replacement, systematic sampling, stratified sampling, multi-stage sampling and so on) and then individuals are selected for interview from every sampled household. There are two main options at this stage. The first option consists of selecting and interviewing one person per sampled household. This is the approach used by the International Crime Victims Survey (ICVS)2. The respondent is generally selected using the next/last birthday method or the Kish grid method. An alternative option consists of surveying all the household members above a certain age limit, as the National Crime and Victimization Survey does. The NCVS has been conducted since 1973 by the US Bureau of Justice Statistics3 in order to collect data on household and personal victimization in the US. In the NCVS, all household members aged 12 or more are eligible for interview.
The main advantage of interviewing all household members is that survey costs are reduced: in order to achieve a target sample size, this option implies contacting much less households than if we interviewed one person per
1 Mr Guillaume OSIER, Institut National de la Statistique et des Etudes Economiques (STATEC), 13 rue Erasme, L-1468 Luxembourg. Email : [email protected]. Paper prepared for the meeting of the EU Task Force on Victimization (Luxembourg, 27-28 October 2011)
2 For more information, http://rechten.uvt.nl/icvs/ 3 For more information, http://bjs.ojp.usdoj.gov/ 2 household. In case of face-to-face surveys, the number of trips to a segment area can be minimized, which help us to save money by reducing travel costs. Nevertheless, with telephone or web surveys the cost of contacting a household is generally small: basically, it corresponds to the amount of time needed by the interviewer to self-introduce, introduce the survey and get the household to participate. Thus, the gain in survey cost of interviewing all household members instead of one should be limited.
Another advantage of sampling all household members is that household respondents may help interviewers by providing contact information for the other household members as well as times when they are likely to be available. Further, if their experience was positive, household respondents help to locate and motivate other household members to respond, a burden which would otherwise fall on interviewers. Thus, because the field work can be supervised more easily, non-response should be reduced.
Having all the members of a household interviewed may also produce more accurate results, especially to household-level questions. On the other hand, data for multi-respondent households may be subject to certain biases for intra- household crimes such as domestic violence. For example, a violent husband who has been interviewed may tell his wife not to report domestic violence. More generally, for questions asking about personal attitudes, it is best not to have anybody else present, so that the respondent will feel free to give his or her true opinion. This measurement bias could be reduced if only one person per household was interviewed. The US National Crime and Victimization Survey calls for self-responses from each of these household members. This may require interviewers to make callbacks, so that each household member aged over 12 can be directly talked to.
When we sample all household members, proxy interviewing must not be allowed. A way of keeping costs down when all household members are to be interviewed is to interview one household member asking this person to give information about the other members of the household. This technique has evident implications in terms of cost, but may lead to severe measurement errors when we are dealing with victimization issues, since a proxy respondent is more likely not to report a crime incident and less likely to know the full details concerning reported incidents. However, as a last resort and only under specific conditions, the US National Crime and Victimization Survey (NCVS) allow another person to answer questions for a household member. Proxy interviews are currently conducted for: (1) household members aged 12 or 13 if a knowledgeable household member insists they not be interviewed directly by the interviewer, (2) persons incapable of responding due to physical or mental incapacity, (3) persons who are away from the household during the entire interview period. 3
It has been long-established that when we interview all household members the sampling efficiency is generally deteriorated. The reason for this is that the members of a household tend to be homogeneous with respect to many characteristics. Because of this homogeneity, the information collected from within a household tends to be redundant. For instance, it is common for crime to be concentrated in a few, often economically disadvantaged areas. Thus, people living in those areas tend to have had similar victimization experiences. This phenomenon is referred to as “cluster effect”. As a result, the sampling precision, as measured by the sampling variance, is generally higher than that which would have been obtained if a one-stage sample of individuals had been selected.
The SASU indicators
The target EU-SASU indicators are victimization rates, measuring the share of individuals who have been victim of a type of crime during a given period of time, usually set at the last five years or the last twelve months preceding the interview4. The SASU survey covers household-related crimes like vehicle theft5 (car, van, bicycle, motorcycle…) and home burglary, person-level crimes like physical and sexual violence, theft of personal property and robbery, and also “non-conventional” crimes like card/on-line fraud, consumer fraud and bribery. Of course, a synthetic victimization rate can be constructed by aggregating different types of crimes. In addition, the survey collects information on topics like feeling of safety, security precautions and attitudes to law enforcement.
Let U be the target EU-SASU population of individuals aged 16 or more, of size
N , and let Ud (Ud U ) be the sub-population, of size Nd , of individuals aged 16 or more who have been victim of a type of crime during a given period of time. Let us introduce the dummy variable Y :
1 j Ud Yj (1) 0 j Ud
Y N j The victimization rate P is given by: P d jU (2) N N
When observations are only available over a sample s of the population U , the victimization rate P can be estimated by (assuming that the total population size N is known):
4 The US National Crime and Victimization Survey (NCVS) consider crimes which happened during the last six months preceding the interview
5 Whether vehicle theft should be considered a household or a personal crime is not clear-cut. However, it has been decided to consider it a household-related crime in the EU-SASU 4
w Y Nˆ j j Pˆ d js (3) N N
w j is the sample weight of unit j. Sample weights allow for inference from the sample s of individuals aged 16+ to the target population U. They are usually calculated as design weights, by taking the inverse of the selection probabilities of the individuals, and are adjusted for unit non-response. The sum of the sample weights provides an estimator of the total number N of individuals aged 16 +.
If the size N of the population happens to be unknown, the following estimator can be used instead: w Y Nˆ j j Pˆ alt d js ˆ (4) N w j js ˆ (4) is a non-linear estimator, defined as a ratio between the linear estimators Nd and Nˆ . As to (3), this is a linear estimator as it is calculated as the weighted sample mean of the dummy variable Y .
Figure 1: The SASU indicators – Victimization rate for a type of crime
Population aged 16+ who have been victim of a crime during a given period of time (last five years, last EU-SASU sample of twelve months…) individuals aged 16+
Population aged 16+ (SASU target population)
Victimization rates can be calculated for the whole SASU target population of individuals aged 16 or more and for subpopulations like, for instance, the young, the elderly or the foreigners. Subpopulations are treated as domains of study. The above formulas (3) and (4) are still valid for domain estimation, with the sample being limited to those units who fall into the subpopulation: 5
w Y w Y j j j j ˆ alt ,D jsD D jsD (5) P (6) Pˆ w N D j jsD D is the domain of study (the young, the elderly, the foreigners…), of size N D (in number of individuals aged 16 or more) A general variance formula
In order to determine whether it is more efficient to interview one or all household members, we need a variance criterion. Let N be the total number of individuals in the population aged 16 or more. This population is divided into M households, of average size N N M . We want to estimate the total number T of individuals who have had a victimization experience during a given period of time6. The standard estimator of the total T is given by:
ˆ T w j Yj (7) js
Yj 1 if individual j had the experience, 0 otherwise.
w j is the sampling weight of j.
From now on, instead of j, we use a double index notation to identify individuals: ij, where i identify the household and j the individual aged 16 or more within i. We have Yij 1 if the individual j from household i had the victimization experience, 0 otherwise.
Let’s assume the following: (H1) a simple random sample of m households has been selected without replacement from the total population of M households,
(H2) a simple random sample of ni individuals aged 16 or more has been selected without replacement from each household i, of size Ni , (H3) the selection of individuals is invariant and independent from one household to another.
Under these assumptions, the variance of Tˆ is given by:
m S 2 M M n S 2 ˆ 2 T 2 i i V T M 1 Ni 1 (8) M m m i 1 Ni ni
6 In this section, the total number of individuals who have been victim of a crime is considered, instead of the victimization rate. However, assuming that the population size N of individuals aged 16+ is known, the variance of the corresponding victimization rate can be obtained as a simple by-product through dividing the variance of the total by N2 . Estimating the total number of individuals who have been victim of a crime or the victimization rate is therefore equivalent in terms of accuracy 6
M 2 1 2 ST Ti T M 1 i 1
Ni Ti Yij Ni Yi j 1 1 M T Ti M i 1
Ni 2 1 2 Si Yij Yi Ni 1 j 1
Assuming Ni N and ni n for all i (all households have the same number of individuals aged 16 or more, and we select the same number of individuals per household), (8) leads to:
2 ˆ 2 S N 1 V T N 1 ρ n ρ (9) n0 N 1 N
M Ni 2 1 2 S Yij Y P1 P, where P is the victimization rate for N 1 i 1 j 1 the type of crime considered,
n0 m n is the total sample size of individuals. n0 is assumed to be fixed, and refers to the minimum sample size at country level as set out in the draft SASU Regulation7,
M 1 2 Y Y M 1 i S 2 ρ i 1 B M N 1 i 2 S Yij Y N 1 i 1 j 1
The proof is given in annex 1.
Basically ρ measures the within-household variance: the higher ρ , the more homogenous households with regard to variable Y , that is, Yij Yik for all individuals j and k in household i. When ρ =1, either all the members of a
7 Minimum sample sizes at country level (Article 5 of the draft SASU Regulation): 8 000 persons in Member States having a population aged 16 and over higher than 10 million, 7 000 persons in Member States having a population aged 16 and over higher than 5 million and lower than 10 million, 6 000 persons in Member States having a population aged 16 and over higher than 1.5 million and lower than 5 million, 5 000 persons in Member States having a population aged 16 and over higher than 0.5 million and lower than 1.5 million, 3000 persons in Member States having a population aged 16 and over lower than 0.5 million 7 household have been victim of the type of crime considered or all the members have not. ρ measures the so-called “cluster effect” (see first section). ρ is also referred to in statistical literature as the intra-household correlation coefficient (Cochran, 1977) If n 1 , that is, if we select one individual per household, then (9) is similar to:
S2 V Tˆ N 2 (10) n0
This is the variance expression for a simple random sampling without replacement of size n0 .
If n N , that is, if all household members are included in the sample, (9) leads to the following well-known result (Särndal et al, 1992 ; Ardilly and Tillé, 2005 ; Ardilly, 2006):
S2 S2 V Tˆ N 2 Nρ N 2 1 μN 1 (11) n0 n0 where:
M Ni Ni Yik Y Yil Y i 1 k 1 l 1 1 k l Nρ 1 μ M N N 1 i 2 N 1 Yij Y i 1 j 1
The main lesson of (9) is that, under a fixed sample size n0 , the variance of the total Tˆ is a linear function of n : if n increases by 1, there is an additional variance term equal to:
S2 N 1 P1 P N 1 N 2 ρ N 2 ρ (12) n0 N 1 N n0 N 1 N
The sign of this variance term Δ depends on the value of the intra-household correlation coefficient ρ : 1 0 ρ (13) N
As to the intensity of Δ , it also depends on the value of ρ : the closer ρ to 1, the higher the increase of variance. Conversely, the closer ρ to 0, the higher the decrease of variance. The intensity also depends on the victimization rate P. We have P1 P 0 for all P, with a maximum value for P=0.5. Thus, the closer P to 0.5, the higher the increase (if ρ 1 ) or the decrease (if ρ 1 ) of variance. N N 8
If we consider the relative variance, defined by dividing the variance by the square value of the parameter, the additional variance (in % points) is given by: Δ 1 P 1 N 1 Δrel 2 ρ (14) NP P n0 N 1 N
The previous statements are still valid, except that the increase depends on how close P is to 0 (and not to 0.5).
As a conclusion, when ρ 1 intra-household correlation makes sampling two N or more individuals per household less accurate (in terms of sampling variance) than if we sampled only one person per household. The loss of accuracy is even stronger as ρ gets closer to 1. On the other side, if ρ is close to 0, sampling more than one household member would lead to more accurate results, even more so as ρ is close to 0. Let’s illustrate this point with an example. In the extreme case of ρ =0, all the households have the same composition with regard to victimization. For instance, let’s assume that N =4 and that for all households in the population one member out of 4 have had the victimization experience. By sampling all household members, the victimization rate will be estimated by 1/4 from all possible samples. As a result, the variance will be zero. On the other hand, if we select for interview one member per household there is a chance, although very small, that only members who have had the experience (estimated victimization rate = 100%) or members who have not had it (estimated victimization rate = 0%) be selected. Therefore, there will be some variability from one sample to another. Thus, when the cluster effect is small, the option of sampling all household members turns out to be more efficient than that consisting of sampling one member per household.
However, in most SASU cases, the intra-household correlation coefficient ρ should be high. For household-level crimes like vehicle theft or home burglary, all the members of a household are expected to provide the same answer 8 (unless there are measurement errors), thus making ρ equal to 1. As to person-level crimes like robbery or violence, the degree of homogeneity within household should be lower than 1. For instance, if a member of a household has been robbed, this does not mean that the other members have been robbed too. However, ρ should remain high not only because, as pointed out in the first section, crime generally tend to be concentrated in a few areas, but also because household sizes are generally small: as a matter of fact, clusters of 3 or 4 individuals tend to be more homogeneous than clusters of 300 or 400.
Thus, the condition (13) is likely to be satisfied in most cases in the SASU survey, that is, interviewing two or more individuals per household instead of one will make statistical accuracy, as measured by the sampling variance, worse. The
8 Because of this, questions that relate to the household are generally asked only once 9 loss of accuracy might be important: if ρ =1, that is, either all the members of a household have been victim of the crime under study or not, (9) is similar to: S2 S2 S2 V Tˆ N 2 n N 2 N 2 (15) n0 n0 n m
The variance in (15) is of order 1/m, where m is the sample size of households. This result is intuitive: if all household members are similar with regard to the variable of interest, then there is no gain in accuracy by interviewing more than one member per household. The level we can achieve by surveying n individuals per household is in fact equal to that we would obtain if we only surveyed one member per household, with the household sample size m fixed. This represents a major decrease in accuracy. More generally, when n N , (11) m shows that the variance is of order 1 . If ρ =1, we obtain (15). Otherwise, we ρ see that the loss of accuracy may still remain important if ρ is close to 1 (like, as we suspect, in most SASU cases). We’ll get back to this later on in the document when numerical examples will be shown.
For domain estimation, (9) is still valid, with every term in the variance formula being replaced by its corresponding value over the subpopulation D:
D 2 D ˆ D D 2 S N D D D 1 V T N D D 1 ρ n ρ D (16) n0 N 1 N
Because n D n 9, the impact on sampling variance of interviewing all household members instead of one tends to decrease. This is relevant for certain types of crime like vehicle theft, as they are only applicable to the individuals who own a vehicle.
Tradeoff between cost and accuracy
According to the previous statements, sampling one person per household is beneficial to sampling variance, although the solution is not optimal in terms of cost. In order to make proper decisions, a tradeoff needs to be made between survey cost and accuracy. We may seek the sample size m of households and that n of individuals within households so to minimize the variance
2 2 S 1 ρ N 1 ρ n ρ (17) mn N
under the cost constraint c1m c2 mn C0 .
9 For instance, when D is the male or the female population , we have on average n D n 2 10
The first component of cost c1m is proportional to the number of households ( m ) in the sample; while the second component, c2 mn , is proportional to the total number of individuals ( mn ) in the sample. c1 refer to the cost of contacting a household. Basically, c1 includes travel cost – location visit and on the day –, the amount of time needed by the interviewer to present the survey and put the household at ease. As to c2 , this is the cost of an interview, excluding the cost of contacting the households: c2 includes the amount of time needed to contact the person (given the household has been contacted), to collect and process the information. The total budget C0 is enough to achieve at national level the minimum sample size n0 which is required by the EU-SASU Regulation.
If ρ 1 , the optimal sample sizes are given by (see annex 2): N
C m 0 c 1 ρ 1 ρ n 1 c c c (18) c 1 (19) 1 1 2 1 2 ρ ρ N N
As ρ gets closer to 1 (for instance, with household-related crimes), then the sample size of individuals n must be lower and, conversely, the sample size of households m higher. This makes sense: when households are homogeneous with regard to the variable of interest, it is better from the statistical accuracy point of view to select as many households as possible instead of collecting redundant information on every member of a household. On the other hand, when households are “representative” of the whole population (that is, ρ small) then it is better to select fewer households and more individuals per household: we lose in terms of accuracy but we save a lot of money.
Besides, (19) shows that n does not depend on the total budget C0 but on the ratio c1 c2 . If the cost c1 of contacting a household is small compared to the cost c2 of an interview, as with telephone or web surveys, the sample size of households must be chosen higher and the sample size of individuals lower. This makes sense too: when the cost of contacting a household is small then more households must be selected (this way, we can always make accuracy better, especially when ρ 1). On the other hand, with face-to-face surveys the cost c1 of contacting a household is higher. If the cost burden c1 c2 is strong then it might be better for cost efficiency to select fewer households and to interview more than one member per household.
Having said this, it is difficult to develop any further, because the solution depends on a set of parameters ( c1 ,c2 ,C0 ,ρ ) which are unknown. In particular, 11 since ρ is variable-specific, the optimal sample sizes are variable-specific too: what is optimal for one variable may not be for another.
To illustrate this point, let’s plot the number n of individuals to be interviewed per household against the intra-household correlation coefficient ρ . We assume here that N 4 . Two different cases are considered: We assumed that the cost (in number of days) of contacting a household
was 4 times higher than the cost of an interview, that is c1 c2 4 . This is the order of value we might have with face-to-face surveys.
On the other hand, with telephone or web surveys, the ratio c1 c2 is likely
to be smaller, for instance, c1 c2 0.5 .
Of course, these values are here for illustrative purposes and may be different in practical situations.
Figure 2: the number n of individuals to be interviewed per household against the intra-household correlation coefficient ρ
19 18 17 16 15
d 14 e
w 13 e i ) v 12 r r a e
t 11 b n i n ( 10
e d b l 9
o o t h 8
e s l s 7 a u u o 6 d h i
r v 5 i e d p 4 n i
f 3 o
r 2 e
b 1
m 0 u n 0.26 0.36 0.46 0.56 0.66 0.76 0.86 0.96 rho
c1/c2 = 4 (face-to-face surveys) c1/c2 = .5 (telephone or web surveys)
We can see that in both cases n decreases as ρ gets closer to 1. Looking at the previous figure, we see that the number n of individuals to be interviewed per household depends on ρ : c1/c2 = 0.5 (“telephone” or “web” situation)
ρ 0.27 n 4
0.28 ρ 0.30 n 3 12
0.31 ρ 0.38 n 2
0.39 ρ n 1
c1/c2 = 4 (“face-to-face” situation)
ρ 0.43 n 4
0.44 ρ 0.54 n 3
0.55 ρ 0.73 n 2
0.74 ρ n 1
With telephone or web surveys, we see that it is often advantageous to interview only one person per household. This strategy is beneficial for statistical accuracy, while the additional cost is limited. With face-to-face surveys, the results are more variable-dependant.
Numerical examples
In order to illustrate our findings, let’s use the data from the US National Crime and Victimization Survey (NCVS). The National Crime Victimization Survey (NCVS) has been collecting data on personal and household victimization since 1973. The survey focuses on gathering information on the following crimes: assault, burglary, larceny, motor vehicle theft, rape, and robbery. Twice each year, data are obtained from a nationally representative sample of roughly 49,000 households comprising about 100,000 persons on the frequency, characteristics, and consequences of criminal victimization in the United States. The survey is administered by the U.S. Census Bureau (under the U.S. Department of Commerce) on behalf of the Bureau of Justice Statistics (under the U.S. Department of Justice).
NCVS surveys households randomly selected from a stratified multistage cluster sample, with the interviews administered by the United States Census Bureau. The entire selected household (above age 12) is interviewed instead of just one member selected. The selected household remains in the survey sample for three years, with interviews conducted every six months. Proxy interviews are allowaed for currently conducted for: (i) household members aged 12 or 13 if a knowledgeable household member insists they not be interviewed directly by the interviewer, (ii) persons incapable of responding due to physical or mental incapacity, (iii) persons who are away from the household during the entire interview period. 13
NCVS micro-data files are access-free from the Inter-University Consortium for Political and Social Research (ICPSR) at the University of Michigan (http://www.icpsr.umich.edu/icpsrweb/ICPSR), on condition that they are used for research purposes only.
In the following, we use the NCVS data for 2009 in order to estimate the intra- household correlation coefficient for different types of crimes. These numerical results will serve to illustrate the previous theoretical findings.
We consider the following types of crimes (see Annex 3 for the exact wording of the questions) - something stolen or attempt - broken in or attempted - motor vehicle theft - attack, threat, theft (location cues) - attack, threat (weapon, attack cues) - stolen, attack, threats: offender known - forced or coerced unwanted sex
These variables are individual’s screen questions. For all of them, the “cluster effect” as measured by the intra-household correlation coefficient is higher than 0.5. For the items “broken in or attempted” and “motor vehicle theft”, it is even equal to 1. These values are rather high and, according to what we have said, are likely to make sampling more than one person per household a less accurate strategy than sampling one person only (see Table 1).
In order to assess the impact on accuracy of sampling more than one person per household, we use two common measures of accuracy: the absolute margin of error and the relative margin of error.
The absolute margin of error is the half-length of a confidence interval. For example, if the estimated victimization rate is 10 percentage points, and if the half-length of a confidence interval is 5 percentage points, then the absolute margin of error is 5 percentage points.
If we consider a victimization rate P , using a normal approximation a confidence interval with a confidence level α 0 α 1 is given by:
ˆ CI P p z1α 2 σˆ , p z1α 2 σˆ (20)
where p refers to the estimated proportion (from the sample) and z1α 2 is the quantile value at 1 α 2 for the normal distribution. If α 0.95 , we have z1α 2 1.96 . σˆ is the estimated standard error for the estimator of P , that is, the square root of the variance. Using (9), we have: 14
S2 N 1 p1 p N 1 σˆ 1 ρ n ρ 1 ρ n ρ (21) n0 N 1 N n0 N 1 N
Thus, an estimator for the absolute margin of error (AME) at 95% confidence level is given by (in percentage points):
ˆ p1 p N 1 AME 196 σˆ 196 1 ρ n ρ (22) n0 N 1 N
If we now use the relative margin of error as a measure of accuracy, then we express the absolute margin of error as a percentage of the parameter value. Coming back to our first example, if the estimated value of the victimization rate is 10 percentage points, and if the half-length of a confidence interval is 5 percentage points, then the absolute margin of error is 5 percentage points and the relative margin of error is equal to 50% (because 5 percentage points are 50 percent of 10 percentage points).
More generally, with the previous notations, the relative margin of error (in %) is estimated by:
ˆ ˆ AME 1 p N 1 RME 100 196 1 ρ n ρ (23) p n0 p N 1 N
Table 1: intra-household correlation coefficient (“cluster effect”) ρ , prevalence rate (%), relative margin of error (%) and absolute margin of error (% points) for different types of crimes, 2009 stolen, attack, attack, somethin broken in attack, forced or motor threat, threat: g stolen or threats: coerced vehicl theft: weapon or attempte offende unwante e theft locatio , attack attempt d r d sex n cues cues known Intra- household correlation 0.64 1.00 1.00 0.61 0.58 0.58 0.64 coefficient (rho) Prevalenc 2.9 0.8 1.5 0.9 0.3 0.2 0.0 e rate (%) Relative margin of error (%) Number of 1 12.7 24.0 17.9 23.3 38.0 48.8 117.1 individuals 2 15.8 33.9 25.3 28.6 46.2 59.1 146.0 to be interviewe 3 18.3 41.5 30.9 33.1 53.2 67.9 170.0 d per 4 20.6 47.9 35.7 37.0 59.4 75.7 191.0 household 5 22.6 53.6 39.9 40.6 64.9 82.7 210.0 (nbar) 15
Absolute margin of error (% points) Number of 1 0.37 0.20 0.26 0.20 0.13 0.10 0.04 individuals 2 0.46 0.28 0.37 0.25 0.15 0.12 0.05 to be interviewe 3 0.53 0.34 0.46 0.29 0.18 0.14 0.06 d per 4 0.60 0.40 0.53 0.32 0.20 0.15 0.07 household 5 (nbar) 0.66 0.44 0.59 0.36 0.22 0.17 0.07 Source: National Crime and Victimization Survey
The next two figures show that the accuracy as measured by the absolute or the relative margin of error may significantly increase if we select more than one person per household. For instance, in case of crimes having a low prevalence rate such as forced or coerced unwanted sex, the relative margin of error is greater than 200% when 5 individuals are interviewed per household, that is, when all the members of a household are eligible for interview. On the other hand, when only one member per household is interviewed, the value is 120%.
Figure 3: Absolute margin of error (% points) against the number of individuals interviewed per household, by type of crime, 2009
0.7
0.6
0.5 ) s t n i o p
% (
r o
r 0.4 r e
f o
n i g r a m
e
t 0.3 u l o s b A
0.2
0.1
0 0 1 2 3 4 5 6
Number of individuals to be interviewed per household (nbar) something stolen or attempt broken in or attempted motor vehicle theft attack, threat, theft: location cues attack, threat: weapon, attack cues stolen, attack, threats: offender known forced or coerced unwanted sex Source: National Crime and Victimization Survey Note: we set n0 (total sample size at individual level) to 8000 and N (household size) to 5 16
Figure 4: Relative margin of error (%) against the number of individuals interviewed per household, by type of crime, 2009
250
200 ) % (
r o r r e
f
o 150
n i g r a m
e v i t a l e R 100
50
0 0 1 2 3 4 5 6
Number of individuals interviewed per household (nbar)
something stolen or attempt broken in or attempted motor vehicle theft attack, threat, theft: location cues attack, threat: weapon, attack cues stolen, attack, threats: offender known forced or coerced unwanted sex Source: National Crime and Victimization Survey Note: we set n0 (total sample size at individual level) to 8000 and N (household size) to 5
Comparison in a Simple random sampling design
A practical comparison with specific designs can be presented based on previous analysis and results. The comparison will be based on specific choices for elements of the design (sampling unit, stratification, clustering etc. )
In a simple case we can compare two simple random samples of households, in the first, only one person is selected (at random) while in the second all members of the household that are over 16 are sampled.
In order to facilitate the comparison we use the concept of the design effect (Deff), introduced by Kish (1965) as the ratio of the variance of a more complex design (CD) to the variance of a simple random sample (SRS) of the same size.
VarCD Deff (22) Varsrs The simple random sample with which to compare is a sample of persons from a population register. A related term is the effective sample size n eff which is the sample size of a SRS that would yield the same sampling variance as achieved by the actual design. 17
n0 neff (23) Deff
Where no is the nominal sample size.
The Design effect of selecting one person (the first design) from each household is due to weighting, which is needed to account for unequal selection probabilities for people living in households of unequal size.
Indeed, if all households have the same probability of inclusion p, then people living in single person households have p probability of inclusion, people living in a two-person household have p/2 probability, people living in a three-person household have p/3 probability etc. The simplified estimate for the design effect is (Gabler et al., 1999)
I 2 n ni wi i1 Deff 2 (24) I ni wi i1
Where ni and wi denote the number of interviews and the corresponding design weight assigned to household size i.
The design effect in this case can be estimated using the structure of households in each country as household sizes vary between Member states. It is therefore recommended that a different computation is made for each country. However, for the purpose of this report we will use data from a household grid of a household survey (European Working Conditions Survey - 2010). The dataset includes persons over 15 which is slightly different from the definition of the target population in the SASU. Based on the results from about 43 thousand households and 109 thousand individuals we expect the proportion (distribution) of each household size to be as follows:
Table 2 Household size characteristics from EWCS 2010 1 2 3 4 5 6 7 8 9 10 Total Number of 6768 20018 8696 5618 1770 547 151 42 23 13 43816 households Percent 15.45 45.69 19.85 12.82 4.04 1.25 0.34 0.10 0.05 0.03 100.00 distribution (%) Note: Average household size is about 2.5 which is a bit lower than the European average household size including all persons which is about 2.8 18
Based on these data the design effect of weighting for unequal probabilities of selection (the first design) is 1.24 and the corresponding sample size for a nominal no=8000 is 6437.
The design effect of selecting all members of the household is due to the correlation of variables between members of the same household. Based on the standard formula for the design effect (Kish, 1965)
eff D0 1 (N 1) (25)
Where N is the size of the household. Because this is variable then the formula is summed up among different sizes (i) as follows: eff D0 1 (N i 1) (26) i
Based on the distribution of household size as displayed in Table 2 and the intrac-household correlation coefficients computed earlier for some variables from the NCVS survey (see Table 1), we display in the following table the design effects and corresponding effective sample sizes for the different variables variable and for nominal sample size no=8000.
Table 3. Intra-household correlation coefficient effective sample size and design effect for different types of crimes, 2009 Intra- household Effective Design Type of crime correlation sample size effect coefficient (ICC) something stolen or attempt 0.64 3949 2.03 broken in or attempted 1 3197 2.50 motor vehicle theft 1 3197 2.50 attack, threat, theft: location cues 0.61 4033 1.98 attack, threat: weapon, attack cues 0.58 4120 1.94 stolen, attack, threats: offender known 0.58 4120 1.94 forced or coerced unwanted sex 0.64 3949 2.03 Source: National Crime and Victimization Survey
From the table above, we can observe that the household related questions that have a intra-household correlation coefficient (ICC) of 1 have an effective sample size of about 3200 (which is actually the number of households). The other 19 variables that have ICC close to 0.6 (from 0.58 to 0.64) yield an effective sample size of about 4000. Since including the first member of the household yields 3200 effective sample size, it should be noted that the rest of the interviews (8000- 3200=4800 interviews) contribute towards an increase of just 800 in the effective sample size.
When considering the results of this analysis, please note the exceptionally high ICCs in this statistical domain (victimization related surveys). In other statistical areas coefficients of less than 0.2 are generally expected.
In terms of cost, the results above can be used in conjunction with the cost model described earlier
c1m c2 mn C0 where the first component of the cost is proportional to the number m of households in the sample, and the second component is proportional to the total number of individuals m*Nbar in the sample.
A comparison of costs between the two designs for the same effective sample size is shown in the following figure for various values of the ratio between C1/C2. The cost of the corresponding SRS (a sample with the effective sample size drawn at random from a population register) is also shown. All household members 1 person per household SRS
600,000 €
500,000 €
400,000 €
300,000 €
200,000 €
100,000 €
- € 0.7 1.0 1.3 1.7 2.0 2.3 2.7 3.0 3.3 3.7 4.0 4.3 4.7 5.0 5.3 5.7 6.0 6.3 6.7 C1/C2
With some simple computations the all household design is more cost efficient than the single person design if C1>1.7 C2. That is the cost of reaching a household is more than 1.7 times the cost of interviewing. more cost efficient than the SRS only if C1>5C2. 20
The case of geographical clustering
The comparison would have been different if multistage clustered designs, which are fairly common because of cost considerations, were compared. In this case the population is divided into groups or clusters based on a some geographical area (a census tract, neighborhoods, settlements etc.). Clusters serve as primary sampling units (PSUs) and are selected at random. Then households are selected as Secondary Sampling Units (SSUs) and finally one, or all members of selected households are interviewed.
Clustered designs are particularly successful when the within cluster variability is large compared with between cluster variability, i.e. when the intraclass correlation coefficient is small.
In order to make a comparison one will need to have the definition of the cluster scheme and estimates of the intraclass correlation coefficient. Although this is not available we can make some observations
The intraclass correlation coefficient will depend on the extent of the cluster. Crime frequencies are highly depended on geography and, in fact, statistics of crime incidents are one of the first applications of spatial econometrics. More extended clusters will have larger within cluster variability (smaller intraclass correlation coefficient). The effect of the cluster size (i.e. the number of SSUs within each cluster) is also important. Small numbers reduce the influence of intraclass correlation on the variance of estimates. We have already observed that the intrahousehold correlation coefficient is particularly large in the dataset we used. We can assume that this is explained to some degree, by the fact that household members live in the same area. It is also related to other characteristics that household members have in common and therefore presents an upper limit to the infraclass correlation coefficient of even the smallest cluster. It is reasonable to conjecture that the interclass correlation coefficient between subjects in the same geographical cluster will include (account for) a substantial amount of the correlation that is now possible to observe among members of the same household. Therefore the design effect of including all members of the household is expected to be smaller in multistage sampling than the values computed under a SRS in this report.
Conclusion
1. The option of interviewing all household members instead of one often leads to less accurate results in terms of sampling variance, mainly because the members of a household tend to be homogeneous with regard to victimization experiences. The loss of accuracy should be 21
particularly important with household-level crimes like vehicle theft or home burglary. It should be important with person-level crimes (robbery, violence…) as well since the within-household correlation should remain high for these types of crimes.
2. From the cost perspective, interviewing all household members helps us to save money, particularly when the survey is carried out face-to-face. There are other advantages with regard to non-response rate and measurement errors; provided that one’s response is not influenced by the other members of the household (for instance, intra-household violence). The loss of sampling efficiency caused by interviewing more than one person per household should be balanced with all these aspects.
3. In order to reconcile statistical accuracy with survey cost, a tradeoff has to be made. However, the compromise solution in terms of both statistical accuracy and cost efficiency is variable-specific, especially with face-to- face surveys.
4. In the specific case of simple random sample, the comparison yields that the nominal sample should almost be doubled in order to achieve the same level of accuracy. This is cost efficient only if the cost of contacting a household is more than 5 times the cost of implementing an interview.
5. Given the impact on accuracy of interviewing more than one household member, the article of the SASU Regulation dealing with minimum sample sizes should be clarified: The minimum sample sizes must be inflated in order to take unit non-response and design effects (stratification, clustering, unequal weighting) into account In particular, the impact on accuracy of interviewing more than one household member must be taken into account
Questions to the TF:
What about national experiences (national surveys on victimization, pilot SASU surveys…)? Have you interviewed all household members or only one? What were the pros and cons?
What kind of survey designs are generally used (stratified, clustered, etc.)?
What about intra-household correlation (“cluster effect”)? Have you observed the same results (high values for the intra-household correlation coefficient ρ ) with your data? 22
What about cost? Given the available budget, can you afford interviewing one member per household? If not, do you think that the loss of accuracy caused by interviewing more than one member per household is “acceptable” given the objectives of the survey?
References
- Ardilly, P. (2006). Les techniques de sondage. Technip. - Ardilly, P. and Tillé, Y. (2006). Sampling methods: exercises and solutions. Springer. - Cochran, W. G. (1977). Sampling techniques. Wiley. - Clark, R., Steel, D., The Effect of using Household as a Sampling Unit, International Statistical Review, Volume 70, Issue 2, pages 289–314, August 2002
- Gabler, S., Häder, S., Lahiri, P. (1999). A model based justification of Kish's formula for design effects for weighting and clustering. Survey Methodology 25:1, 105-106.
- Krenzke, T., Li, L. Rust, K., Evaluating within household selection rules under a multi-stage design, SURVEY METHODOLOGY Volume: 36 Issue: 1 Pages: 111-119, JUN 2010
- The National Crime Victimization Survey Resource Guide http://www.icpsr.umich.edu/icpsrweb/NACJD/NCVS/ - Kish, L. (1965). Survey Sampling. New York: John Wiley & Sons, Inc
- National Crime and Victimization Survey. CAPI Interviewing Manual for Field Representatives. http://bjs.ojp.usdoj.gov/content/pub/pdf/manual08.pdf - Särndal, C.E., Swensson, B. and Wretman, J.H. (1992). Model Assisted Survey Sampling. New York: Springer. 23
Annex 1
The proof relies on the variance decomposition:
M Ni 2 1 2 S Yij Y N 1 i 1 j 1 M M Ni 1 2 Ni 2 2 2 Si Yi Y SW SB i 1 N 1 i 1 N 1
M M Ni 2 Ni 1 2 Ni 1 1 2 (i) SW Si Yij Yi is the within-household i 1 N 1 i 1 N 1 Ni 1 j 1 variance
M 2 Ni 2 (ii) SB Yi Y is the between-household variance i 1 N 1 ********************************************
1. Assuming Ni N and ni n for all i, the variance can be written:
V Tˆ m S 2 M M n S 2 2 T 2 i i M 1 Ni 1 M m m i 1 Ni ni
N 1 2 SB 2 M 2 m 2 N N M N n 2 M 1 N 1 Si M m m n N i 1
m 2. Assuming 1 1 and N N 1 N N , we obtain: M V Tˆ 2 2 SB 2 1 n N 2 N N 1 SW m n0 N N 1
2 1 2 n N 2 N nSB 1 SW n0 N N 1 2 2 S n N N nρ 1 1 ρ n0 N N 1 2 2 2 S N 1 ρ 2 S N 1 N 1 ρ n ρ N 1 ρ n ρ n0 N 1 N 1 n0 N 1 N 24
Annex 2
We seek m and n which minimize the variance
2 2 S N 1 N 1 ρ n ρ mn N 1 N
under the cost constraint c1m c2 mn C0 .
*************************************************
1. The Lagrangian function L is given by:
1 1 Lm,n,λ 1 ρ n ρ λc1m c2mn C0 mn N
2. The partial derivatives of L are given by:
L 1 1 m,n,λ 2 1 ρ n ρ λc1 c2n m m n N L 1 m,n,λ 1 ρ λc m n mn 2 2
3. The derivatives are equal to 0 if:
1 1 2 1 ρ n ρ λc1 c2n m n N 1 1 1 ρ n ρ λc1m λc2mn E1 mn N
1 1 ρ λc m mn 2 2 1 1 ρ λc mn E2 mn 2 25
4. By subtracting (E1) and (E2), we obtain:
1 1 n ρ λc1m mn N 1 ρ N λc m m 1 1 ρ m N λc1 and,
1 1 ρ λc mn mn 2 1 ρ 1 ρ c 1 ρ n 2 1 λc m2 1 c 1 2 ρ 2 ρ N N λc2 λc1 c 1 ρ n 1 c 1 2 ρ N
5. By using the cost constraint c1m c2 mn C0 , we obtain:
1 1 ρ ρ c 1 ρ c N c N 1 C 1 λc 2 λc c 1 0 1 1 2 ρ N 1 1 1 c ρ c 1 ρ C λ 1 N λ 2 0 1 C 0 λc 1 1 c ρ c c 1 ρ 1 N 1 2 1 C ρ 0 C m N 0 1 1 ρ c ρ c c 1 ρ c c c 1 1 2 1 1 2 1 N ρ N 26
Annex 3: the NCVS questions
V3034: something stolen or attempt
V3036: broken in or attempted 27
V3038: motor vehicle theft
V3040: attack, threat, theft: location cues 28
V3042: attack, threat: weapon, attack cues
V3044: stolen, attack, threats: offender known
V3046: forced or coerced unwanted sex