Convergence of the Biostatistical and Survey Worlds Thomas A. Louis

Convergence of the Biostatistical and Survey Worlds

Thomas A. Louis, PhD

Department of Biostatistics Johns Hopkins Bloomberg SPH [email protected]

Research & Methodology U. S. Census Bureau

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 1 Outline

The Census Bureau A sampling of research at Census Adaptive design Disclosure avoidance A few other topics Design-based/Model-based Convergence of the Biostatistical and survey cultures

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 2 HAPPY 50th!

Preamble

Historically, survey, biostatistical and epidemiological methods and cultures were quite distinct, or at least appeared to be so However, service as Associate Director for Research & Methodology and Chief Scientist at the U. S. Census Bureau has heightened my awareness of the similarities of goals and methods, and of the many potentials Convergence steadily increases to the beneﬁt of all I highlight some examples, but ﬁrst

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 3 Preamble

HAPPY 50th!

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 4 Selected surveys (of ≈ 130/yr) The American Community Survey (continuous) The Current Population Survey (CPS) Includes Health Insurance Qs The Survey of Income and Program Participation (SIPP) Ditto The National Survey of College Graduates The National Crime Victimization Survey (NCVS) The National Survey on Family Growth (NSFG) The Health Interview Survey International surveys and censuses

The U. S. Census Bureau Employees ≈ 15,000 employees, of these, ≈ 5,000 are on permanent appointments The remainder are primarily part-time interviewers and other field staff Central office in Suitland MD, and 6 Regional offices Censuses The decennial census (the only activity embedded in the U. S. Constitution) The Population & Housing Census - every 10 years The Economic Census - every 5 years The Census of Governments - every 5 years Monthly Import/Export compilations

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 5 The U. S. Census Bureau Employees ≈ 15,000 employees, of these, ≈ 5,000 are on permanent appointments The remainder are primarily part-time interviewers and other field staff Central office in Suitland MD, and 6 Regional offices Censuses The decennial census (the only activity embedded in the U. S. Constitution) The Population & Housing Census - every 10 years The Economic Census - every 5 years The Census of Governments - every 5 years Monthly Import/Export compilations Selected surveys (of ≈ 130/yr) The American Community Survey (continuous) The Current Population Survey (CPS) Includes Health Insurance Qs The Survey of Income and Program Participation (SIPP) Ditto The National Survey of College Graduates The National Crime Victimization Survey (NCVS) The National Survey on Family Growth (NSFG) The Health Interview Survey International surveys and censuses

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 6 Necessary Inputs Sampling frame (under-utilized in clinical and ﬁeld trials) Paradata= ⇒ propensity models Cost & Quality metrics Measures of statistical information Timely and accurate data

Adaptive Design

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 7 Adaptive Design

Goals & Methods Reduce the time/expense from the start of data collection to completion Efficiently allocate data collection resources Use dynamic mode-switching to increase efficiency and enhance quality (dynamic treatment regimens) Employ stopping rules (possibly stratum-specific) Necessary Inputs Sampling frame (under-utilized in clinical and field trials) Paradata= ⇒ propensity models Cost & Quality metrics Measures of statistical information Timely and accurate data

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 8 R-indicators: Overview

Based on the sampling frame and attributes, R-Indicators quantify representativeness of survey coverage They identify the attributes that drive variation in response propensities and support adaptation by evaluating which subgroups are over/under represented The sample R-indicator

ρi is the estimated (possibly adjusted) response propensity for group i v u N u 1 X R(ρ) = 1 − 2t (ρ − ρ¯)2 N − 1 i 1 R(ρ) = 1 indicates that the sample is fully representative

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 9 The National Survey of College Graduates1

Data are collected by a variety of modes: web, telephone, . . . The 2013 NSCG uses monitoring to identify target cases for mode-switching with the goal of moving a case to the mode with the highest response propensity or to control costs by not moving Hold a case in web if it is “low impact” Switch to CATI (Computer assisted telephone interview) if it has not responded via web and is “high impact” Put a CATI case on hold (no contacts) if the “R-indicator” shows that the group is over-represented Strike an eﬀective cost/quality tradeoﬀ

1Thanks to Ben Reist T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 10 Comparison of incentive approaches in the NSCG 4 separate surveys each using a diﬀerent set of incentives, but with the same attributes used in the propensity model

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 11 Partial, unconditional, R-indicators

Identify subgroups that are over/under represented and use the information to encourage or “not encourage” speciﬁc cases or groups Adapt by switching modes, incentives, etc.

With ρk the estimated (possibly adjusted) response propensity for group X = k, ρ the composed vector, andρ ¯ the (weighted) mean, the (partial) unconditional R-indicator is „ « 1 Nk 2 Ru(X = k, ρ) = (ρk − ρ¯) N+

It’s a residual and Ru = 0 ⇒ balance

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 12 NSCG Data Monitoring Example

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 13 How long to wait before sending hard copy? Event-time analysis

In the American Community Survey (ACS), need to determine how long to wait for an internet response before sending hard-copy Demographic group-speciﬁc, event-time distributions were estimated with the event being “answered via the internet” The event-time is administratively censored via sending hard-copy, contacting by phone, etc. With T the internet return time, compute,

P(s, d) = pr(T ≤ s + d | T > s)

Switch to hard copy if P(s, d) < γ for a speciﬁed delay d. Optimize wrt (d, γ) to reduce delay and control costs

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 14 Internet response time distributions2

2 From ACS Memorandum #ACS13–RER–18 T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 15 Stopping rules

When is there suﬃcient information to stop conducting interviews? The “stop and impute rule”

θˆnow : Use currently collected data, augmented by imputation of missing values The “project rule”

θˆfuture : Collect a speciﬁed number of additional interviews, and then augment by imputation of missing values If a prediction model indicates that “ ” pr | θˆnow − θˆfuture |> < γ

then stop and use θˆnow

Similar to futility assessment in a clinical trial

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 16 Learning from data generated by an adaptive design is complex Adaptation may induce confounding that needs to be removed The good news is that the propensities are available The database may be less useful for learning than one produced non-adaptively There is a trade-oﬀ between generating a learning database and optimizing survey performance A single mode becomes a “vector mode”

Very similar to issues in adaptive clinical trials

Issues with adaptive designs

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 17 Very similar to issues in adaptive clinical trials

Issues with adaptive designs

Need robust approaches to avoid degrading quality due to inappropriate adaptation wrt identiﬁed subgroups of interest To avoid degrading coverage for other subgroups You are creating the database; don’t mess it up! Learning from data generated by an adaptive design is complex Adaptation may induce confounding that needs to be removed The good news is that the propensities are available The database may be less useful for learning than one produced non-adaptively There is a trade-oﬀ between generating a learning database and optimizing survey performance A single mode becomes a “vector mode”

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 18 Issues with adaptive designs

Very similar to issues in adaptive clinical trials

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 19 Disclosure Avoidance and Data Dissemination Setting the scene Make data available while protecting confidentiality Average income in small areas, industrial output in small areas Local Employment and Housing Dynamics (LEHD) American Fact Finder or “On the Map” Micro-data Analysis System (MAS) Releasing full “micro-data” provides complete information, but with 100% disclosure risk Releasing no data provides no information, with 0% risk The trade-off should be societally determined, but formalism is needed to guide the choice Achieving an acceptable trade-off is becoming more difficult in the context of “big data” and active intruder threats

Record linkage is closely related (for both good and ill)

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 20 Trade-offs, as in diagnostic testing Disclosure risk and the value of the data are positively related The trade-off is very similar to that for an ROC The X-axis is disclosure risk, rather than (1 - specificity) The Y-axis is available information, rather than sensitivity

Available Info

Disclosure Risk

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 21 Methods to reduce disclosure risk

Bureaucratic/legal: Titles 13 & 26, RDCs, . . . In the big data era, these may become the mainstays Cell suppression Aggregation Random swapping Add noise, “noisy fusion” Add random N(0, σ2) noise, split noise, . . . The variance controls disclosure risk and available information Added noise inflates variance, but aggregation or modeling still supports informative inferences Synthetic data Develop a (Bayesian) model for the full micro-data that preserves important relations Generate one or more datasets based on the model-based, posterior predictive distribution Can provide an effective information/protection trade-off

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 22 Partially Synthetic data allow users to select custom geographies in “OnTheMap”

Commuting Patterns, Portland OR Hurricane Sandy

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 23 Measuring disclosure risk

There is always a disclosure risk when data are made available, and it is best measured by the probability of disclosure For example, among n identiﬁed people, one with “income > $100,000” and with no other information available, the disclosure risk is 1/n More sophisticated measures are available In the era of big data there is other information available from record matching and melding, increasing the risk beyond (sometimes far beyond) what a “local” assessment computes

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 24 Example: Reporting mean salary in successive years with one hire for the second year confers almost no protection The trimmed mean or an M-estimate confer protection So do other robust statistics, noisy fusion, synthetic data To compute you need to know D and in this Big Data era, it is likely bigger than you assume

Probabilistic Differential Privacy3 For ≥ 0, a randomized function K gives -differential privacy, if for all data sets D and D0 differing in at most one element (e.g., row of data), and all S ⊆ range(K), ˛ » –˛ ˛ pr{K(D) ∈ S} ˛ ˛log ˛ ≤ ˛ pr{K(D0) ∈ S} ˛ A global guarantee: The protection is for all possible deletions for datasets that you have identified Plausible deniability: A reported value has a “similar” probability irrespective of whether your data are or are not included in the data set

3Dwork & Smith, J. Privacy and Confidentiality, 2009 T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 25 Probabilistic Differential Privacy3 For ≥ 0, a randomized function K gives -differential privacy, if for all data sets D and D0 differing in at most one element (e.g., row of data), and all S ⊆ range(K), ˛ » –˛ ˛ pr{K(D) ∈ S} ˛ ˛log ˛ ≤ ˛ pr{K(D0) ∈ S} ˛ A global guarantee: The protection is for all possible deletions for datasets that you have identified Plausible deniability: A reported value has a “similar” probability irrespective of whether your data are or are not included in the data set Example: Reporting mean salary in successive years with one hire for the second year confers almost no protection The trimmed mean or an M-estimate confer protection So do other robust statistics, noisy fusion, synthetic data To compute you need to know D and in this Big Data era, it is likely bigger than you assume

3Dwork & Smith, J. Privacy and Conﬁdentiality, 2009 T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 26 Record Matching

Latanya Sweeney A large percentage of the U.S. population has a high probability of being identiﬁed based only on place, gender, date of birth This probability soars when other information is matched. Applications e-Health: Combining Medical records and iPhone info, and . . . Precision Medicine: Gene signatures and medical records Who was where doing what: credit card charges melding with Facebook posts Social networks: Cell phone meta-data Real-time population estimates: Cell phone meta-data Facial recognition Mortality in a war zone Scans of the universe: Are this galaxy in Sloan Sky the same as the one in another database?

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 27 More on record matching

In conducting a survey Is the person at address A the same as the person in the IRS data? If so, can use the IRS information to augment other information De-duplication: root out double counting Imputation: Help build the imputation model

Approaches and challenges Frequentist: “Big Match” Bayesian structuring: pr(match | data) =⇒ fractional matches In general, a challenging computing problem, especially for matching using ≥ 3 sources Computer science to the rescue?

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 28 Micro-simulation/agent-based model for the 2020 Census

Households are “agents” Inputs: Mode-specific and mode-sequence specific response probability, quality and NRFU (non-response follow-up) values Outputs: National and domain-specific (cost, quality) probability distributions that help to identify high-leverage research Leading to a design with a high probability of success (cost, quality) in an acceptable region

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 29 Logic Diagram for Administrative Records

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 30 Statistical inference is all about missing data

Inﬁnite reference population

Infer T (Yn+1,..., Y∞), conditional on (Y1,..., Yn) and on the sampling plan Finite population

Infer T (Yn+1,..., YN ), conditional on (Y1,..., Yn) and on the sampling plan Always need to account for uncertainties

Uncertainty due to (Y1,..., Yn) providing only ﬁnite information on the predictive distribution Uncertainty in predicting the unobserved Y s using a known predictive distribution

Bayesian formulations are the way to go!

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 31 Design vs. model-based inference

Design-based (randomization) inference: The Y s are ﬁxed and inference is based on the distribution of sample inclusion indicators Model-based inference: The Y s are also random variables from a probability distribution Superpopulation: Frequentist inference based on repeated samples the super-population and from the resulting sample Bayes: add a prior for parameters; inference based on posterior distribution of ﬁnite population quantities The fundamental distinction is use of a randomization distribution versus a stochastic model for the Ys, however “Weighters” shouldn’t ignore models Modelers can’t ignore (design) weights Bayesian models that incorporate design features can yield inferences with very good design-based properties

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 32 The basic setup

Finite population: U = {1, 2,..., N}

Values of interest: Yk , k ∈ U

The Yk are a set of ﬁxed, but unknown numbers, not necessarily from a probability distribution

Goal: Estimate a function of the Yk , any function, but we’ll focus on the population total or mean

N X T (Y) total: T (Y) = Y mean: k N k=1

Draw a sample S ∈ U with,

pr(unit k ∈ S) = πk > 0 (can depend on covariates) pr(k and ` ∈ S) = πk`

pr(k1,..., kn ∈ S) = πk1k2...kn

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 33 The weighting game

Sample membership indicators:

 1, k ∈ S Z = k 0, k ∈/ S

E(Zk ) = πk

E(Zk Z`) = πk`

The Zk are random variables; the Yk are constants The Horvitz-Thompson, unbiased estimate of T :

X Yk X Zk Yk Tˆ = HT [Y ] = = k π π k∈S k k∈U k

E(Z )Y π Y ˆ X k k X k k X E(T ) = = = Yk = T (Y) k∈U πk k∈U πk k∈U

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 34 Using auxiliary information

Assume we have information Xk , k ∈ U Examples include location, information from administrative records (e.g., tax data), etc.

And have a model m(Xk ) to predict Yk Then, „ « X X Yk − m(Xk ) Tˆ = m(X ) + Z · GReG k k π k∈U k∈U k X = m(Xk ) + HT [Yk − m(Xk )] k∈U

TˆGReG is the Generalized Regression Estimator (GReG) It is the original doubly robust estimator!

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 35 “Pure” design-based is not so pure

Beneﬁt: If the πk are correct, the estimator is unbiased

However, in complicated surveys producing the πk is a complicated 2 business and computing the πk` is (complicated) For example, the American Community Survey (ACS) uses a very

complicated, cluster design ⇒ complicated πk

And adjustments of the πk are needed to reﬂect non-response and imputation Models are used for the adjustments and imputations Variance computations can be complicated Successive diﬀerence replication Bootstrap, but beware ... Generally, inference for non-linear functions of the Y s requires a model As does small domain estimation

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 36 Analogously, in clinical trials many (most) interesting questions are not protected by randomization, are not Intent to Treat (ITT), but progress can be made, with care! Collecting information to support “causal analysis” is key

Challenges of collecting probability samples

Most state that nonprobability or volunteer samples, can’t be used for population estimates But, “Would you rather have 60% response rate from a well-designed and conducted Gallup survey or a 95% rate from a self-selected group? Advantage Gallup: The 60% is also self-selected, but information on the relation of respondents to non-respondents is available from the sampling frame and generalizing from the sample is possible However: For the self-selected survey, there may be other data that can be used to develop reasonable weights for some reference population

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 37 Challenges of collecting probability samples

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 38 Informative sample size Mean menstrual cycle length (MCL) in a prospective pregnancy study

Enroll couples who are trying to have a child Follow until pregnancy or end of study Average the MCLs to get a “population” estimate µˆ Informative sample size: the relatively less fecund couples provide relatively more cycles and so the average is over-weighted towards the MCLs for less fecund couples If MCL and fecundity are related, µˆ will be biased relative to the population value There are ﬁxes (e.g., equal weighting), but using them depends on recognizing the issue

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 39 Transportability as a unifying theme, see Pearl J, Bareinboim E (2014). External Validity: From do-calculus to Transportability across Populations. Statistical Science, 29: 579–595 Big Data (all data!) potentials To support adaptation To make sense of collected data To transport to a reference population

Internal and External Worlds Surveys focus on external validity, representation of a well-speciﬁed reference population Clinical and epidemiological studies traditionally focus on internal validity with relatively little direct attention to representation Without question, The biostat/epi communities should pay more attention to survey goals and methods The survey communities should pay more attention to biostat/epi goals and methods

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 40 Internal and External Worlds Surveys focus on external validity, representation of a well-speciﬁed reference population Clinical and epidemiological studies traditionally focus on internal validity with relatively little direct attention to representation Without question, The biostat/epi communities should pay more attention to survey goals and methods The survey communities should pay more attention to biostat/epi goals and methods Transportability as a unifying theme, see Pearl J, Bareinboim E (2014). External Validity: From do-calculus to Transportability across Populations. Statistical Science, 29: 579–595 Big Data (all data!) potentials To support adaptation To make sense of collected data To transport to a reference population

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 41 Miettinen’s view, at least in 1985 Miettinen, O. S. (1985). Theoretical Epidemiology. Wiley, New York

“In science the generalization from the actual study experience is not made to a population of which the study experience is a sample in a technical sense of probability sampling. In science the generalization is from the actual study experience to the abstract, with no referent in place or time.”

Olie’s view is far too optimistic; far too trusting in immutable truths

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 42 A pleasing trend

Convergence There is convergence, with some clinical/epi studies identifying in a reasonably well-defined reference population Sometimes using all data (big, small, in between) to help identify the population, to compute weights and transport to it A stumbling block Different interpretations of “representative” In Epi/Bio it is commonly reserved for a “self-weighting” sample In the Survey world, if the sampling weights are known, the sample is representative The broader (and correct!) definition opens up opportunities for beneficial convergence

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 43 Enjoy the journey to your 75th and 100th

Coda

The goals and methods of the epi/biostat and survey communities will never completely converge, however there are considerable similarities in goals and methods with more “sims” available These, anchored by overarching principles will empower convergence that will beneﬁt each ﬁeld, science and society Consider the opportunities as you,

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 44 Coda

Enjoy the journey to your 75th and 100th

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 45 THANK YOU

T. A. Louis: Johns Hopkins Biostatistics & Census Bureau McGill, Epidemiology/Biostatistics 50th, 2015 46