arXiv:0804.3244v1 [stat.ME] 21 Apr 2008 odnB ilsi ontv scooit Applied Psychologist, Cognitive is Willis B. e-mail: Gordon Park, USA Triangle 27709-2194, Research Carolina 12194, North Box PO Rd. 3040 Cornwallis International, RTI and Statistician, Sciences, Research Health Senior in Methods Statistical Fellow, ar .Gabr,Pu .Lv n odnB Willis B. Sirken Gordon and Levy Monroe S. Paul Graubard, I. with Barry Conversation A Domain Public the In 637–650 4, DOI: No. 22, Vol. 2007, Science Statistical [email protected] ehsa ayad282 S e-mail: USA 20892, Blvd, Maryland Executive Bethesda, 6120 Genetics, Institute, and Cancer Epidemiology National Cancer of Division Branch, ar .Gabr sSno netgtr Biostatistics Investigator, Senior is Graubard I. Barry 10.1214/07-STS245 tBree n h ffieo h sitn ietrfrResear for Director Assistant Maryland. the Suitland, of in Office Census S the the W. and Laborato of Statistics Berkeley Z. Fellow the at Professor at Post-Doctoral 1950–1951 a where spent As Monroe 1950 Council, mi advisor. a in thesis degr with Washington sociology and M.A. in of mentor 1950 and University in Ph.D. B.A. the a and at earned 1947, and He 1946 in California. UCLA Pasadena, of suburb Abstract. 96we ebcm eirrsac cets,tettehe title the scientist, research an senior Research a of became Office A he Director, the when the was 1996 and he admi and Methodology and (NCHS) and research Statistics search of Nati Health variety the for a Center held mathemat to National actuarial has moved an He gover was and statistician. he his ical statistician, where began 1953 mathematical in He (NOVS) a remain. Statistics as to 1951 decided and in methods, survey and eea ttsia gnishr n abroad. and th a here to first agencies led laboratories, statistical methods federal research units. survey design observation of questionnaire same aspects manent cognitive the the to on ra imp units research which of selection sampling design network multiple the of ing development in the interest r to His led statistical surveys statistics. and federal survey of fostering needs and conducting been have h S S oe ero wr o noaini Government in Innovation Inter for Award the S Service Herriot of Health Roger Public member WSS the ASA elected of the recipient an the is is Amer He and the Institute. and Science, Sci Association of National Statistical Advancement the by American by funded the is of administered Fellow that and th research agencies, subcommittee survey Com statistical and research Federal statistical the its in of chairs member program and charter a (FCSM) is Methodology He a (WSS). (ASA) Society Association tical Statistical American the of committees orevstdteCnu ueua ieo ra hnei t in change great of time a at Bureau Census the visited Monroe sd rmamnsrtv epniiiis oresmaj Monroe’s responsibilities, administrative from Aside orehsbe ciei evn h ttsia community statistical the serving in active been has Monroe alS eyi T Senior RTI is Levy S. Paul . onJnay1,12 nNwYr iy oreSre rwu i up grew Sirken Monroe City, York New in 1921 11, January Born [email protected] . 1 xctv ld ehsa ayad282 USA 20892, e-mail: Maryland Bethesda, 6130 Blvd, Institute, Executive Cancer National and Sciences, Control Population Cancer of Division Program, Research ern iesfo h rgnli aiainand pagination in detail. original typographic the from differs reprint ttsia Science article Statistical original the the of by reprint published electronic an is This [email protected] eadsniiepopulation sensitive and re CSadltra other at later and NCHS t pro evc wr,and Award, Service uperior y nvriyo California of University ry, dteWsigo Statis- Washington the nd neFudto.H sa is He Foundation. ence itaiepstosa the at positions nistrative sac epniet the to responsive esearch nttt fMteaia Statistics Mathematical of Institute rpoesoa interests professional or , i neeti fostering in interest His urnl holds. currently ehssre nmany on served has He . cnAscainfrthe for Association ican osrimo federal of consortium a salsmn fper- of establishment e 07 o.2,N.4 637–650 4, No. 22, Vol. 2007, oe rcso ylink- by precision roves h ..Bra fthe of Bureau U.S. ch, ca n mathemat- a and ician ca cec Research Science ocial scaeDrco,Re- Director, ssociate ehdlg until Methodology d toese grants a oversees at iteo Statistical on mittee nlOc fVital of Office onal mn aerthere career nment ainlStatistical national e nscooyat sociology in ees o nmathematics in nor eueo sampling of use he inamwshis was Birnbaum Statistics. . a n This . in 2 B. I. GRAUBARD, P. S. LEVY AND G. R. WILLIS

This conversation ranges over Monroe’s education and research agendas during a 55-year career as a government statistician. The conversation took place at the National Center for Health Statistics, Hy- attsville, Maryland, in three sessions during the spring of 2006.

EARLY LIFE Graubard: Could we begin by talking about your early life? Sirken: My mother was born in Scranton, Penn- sylvania and married my father in 1919. My father was born in Poland and his family migrated to the Fig. 1. Monroe’s UCLA Student Membership Card, 1938. United States about 1910 when he was about 12 years old. He was a disabled World War I veteran changed my major to sociology thinking that I might and died when I was about 14 years old. I was born become a social worker. However, some faculty in in New York City, moved to upstate New York when the sociology department encouraged me to think I was about two, and ten years later moved to Cali- about becoming a sociologist and advised me to take fornia. My mother, sister and I lived in Sierra Madre, as much mathematics as possible. So I did, without a suburb of Pasadena, where I attended school be- really appreciating how math would help me as a fore moving to Los Angeles and graduating from sociologist or anything else. About 1945, I took my Fairfax High School in June 1938. first course in mathematical statistics from Dr. Paul Hoel. UNIVERSITY OF CALIFORNIA AT LOS Graubard: Why didn’t you stay at UCLA for your ANGELES Ph.D.? Graubard: Then you went to the University of Sirken: Well, for one thing, students were not en- California, Los Angeles. Why that school? couraged to get the Ph.D. at the same universities Sirken: Well, I never considered going elsewhere. at which they were undergraduates. However, the UCLA was virtually free for California high school more important reason is that I had become quite graduates with good grades. As I recall, UCLA tu- interested in quantitative sociology, and UCLA did ition my first semester in September 1938 was $29 not offer that kind of graduate program at the time. plus $4 for a student membership card that enti- I was awarded a fellowship in the sociology depart- tled me to admission for all UCLA sports events. ment at the University of Washington, which my Another financially related reason is, like most stu- UCLA advisors said was strongly quantitatively ori- dents attending UCLA at that time, I couldn’t af- ented. ford to live on campus, and UCLA was close enough to where we lived that I could commute. Foremost, UNIVERSITY OF WASHINGTON I thought UCLA was a great university. Graubard: I believe that your B.A. and M.A. were Graubard: So what department did you actually in the social sciences, and I wonder how you became end up in at the University of Washington? interested in statistics and mathematics? Sirken: I started out as a teaching assistant in Sirken: I got a B.A. in sociology in 1946 and the the sociology department. But one of the greatest following year an M.A. in anthropology and sociol- strokes of good fortune in my professional life oc- ogy. How I became interested in statistics is a longer curred soon after I arrived in Seattle though I didn’t story. I began UCLA as a pre-med major with in- realize it at the time. And that good fortune was tentions of going to medical school but during my ultimately responsible for me ending up straddling sophomore year, I contracted tuberculosis. After re- two departments. covering my health, about three years later, I re- In 1947, the math department at the University turned to UCLA in 1943. In my financial situation, of Washington began to offer a two-year sequence it was unrealistic to think of medical school. So I of graduate courses in statistics and the person in A CONVERSATION WITH MONROE SIRKEN 3 charge of the program was Professor Z. W. Birn- minimized the joint effects of sampling and nonre- baum. I was unsure that I qualified so I went over to sponse bias in survey estimates of categorical vari- Dr. (Bill) Birnbaum and described my background ables (Birnbaum and Sirken, 1950). in math and statistics and my interests in quan- I worked in Bill’s laboratory for two years, and titative sociology. To make a long story short he wrote my thesis under his direction. Bill taught me accepted me, and I took the courses he offered in how to think in terms of statistical models. That 1947–1948 and did very well. He was an excellent was an invaluable gift. instructor, took an interest in his students and they Graubard: What did you do after completing your very much appreciated him. I recall that on the fi- Ph.D.? nal day of the last class in the sequence, the students Sirken: Well, I got the Ph.D. in June, 1950 and presented Bill with a goldplated multicolor pen with stayed in Bill’s laboratory that summer working on the inscription “nature is not vicious” because that the problem of optimizing the scheduling of call- was the phrase he often used when the math got backs of survey interviewers. I got this idea while really complicated. When Bill died a few years ago, writing the Ph.D. thesis on bias due to nonresponse his daughter told me that she found the inscribed in sample surveys. I learned much later that it was pen on top of his desk. a linear programming problem. In 1948, Dr. Birnbaum offered me a job as a re- search assistant in his newly established statistics POST-DOCTORAL FELLOWSHIP AND FIRST laboratory, which I instantly accepted. However, I JOB had already accepted a position as a research assis- Graubard: What was your first position after get- tant in the newly formed Washington Public Opin- ting the Ph.D.? ion Poll (WPOL), which was housed in the sociology Sirken: In the fall of 1950, I headed to the Statis- department, and where I was becoming acquainted tics Laboratory at the University of California, Berke- with sampling and sample survey methods. So dur- ley. I had a Social Science Research Council Post- ing 1948, I had distinct appointments in the soci- Doctoral Fellowship and was planning to spend most ology and mathematics departments which I believe of the next 12 months at Berkeley. I took courses was quite unusual at the time and may have been that fall and the following spring from Jerzy Ney- illegal. man and Erich Lehman and consulted while there Graubard: I suppose that kind of interdisciplinary with Ed Barankin. Ed thought he might have the so- type work would be encouraged now; back then it lution to my linear programming problem, which I must have been pretty unusual. had originally solved for relatively simple situations, Sirken: Yes, I believe it was quite unusual in those but I was unsuccessful in applying his theory. I left days. I was just plain lucky to be in the right place at Berkeley in June, 1951. the right time. It happened because George Lund- Graubard: So that was a case when theory didn’t berg, Chair of Sociology Department, encouraged solve a practical problem for you. After working at interdisciplinary research at the intersection of the Berkeley for eight or nine months, what did you do? social and mathematical sciences, and Z. W. Birn- Sirken: I was pretty sure that I wanted to learn baum was willing to take a chance on a social scien- more about sampling and survey research and what tist. better place to get that kind of experience than at Graubard: Professor Birnbaum is well known for the U.S. Census Bureau. So I continued my fellow- his theoretical contributions in mathematics and ship there. statistics. It’s unusual, I would think, for someone Graubard: Whom did you work with at the Cen- with your background to work at his level. sus Bureau? Sirken: I could not and did not work with Bill Sirken: Well, I was located in Morris Hansen’s on those kinds of problems. He helped me solve my office. The staff included really outstanding peo- applied statistical problems, and my problem then ple including Bill Hurwitz, Joe Daly, Max Bershad was measuring “bias due to nonresponse in sample and Margaret Gurney. These people were part of surveys.” It so happened that my survey research the central staff and there were others also tech- problem was nested in Bill’s broader interest in trun- nically responsible to Morris who worked through- cated population distributions. We successfully de- out the Bureau including Joe Steinberg, Joe Waks- rived the expression the mean squared error that berg and Harold Nisselson, to name a few. However, 4 B. I. GRAUBARD, P. S. LEVY AND G. R. WILLIS

I was mostly involved with a small group of sur- my predecessor at NOVS, was an exceptionally good vey methodologists including Eli Marks and Leon communicator and his publications were very help- Pritzker who were working on the Post-Enumeration ful. Also, Mortimer Spiegelman, a well-known de- Survey (PES). The PES evaluated the quality of the mographer at Metropolitan Life Insurance, was my 1950 Census of Population by renumerating a sam- collaborator in constructing the 1950 U.S. and state ple of the census population. By the time I arrived, life tables. the PES fieldwork had been completed and the staff Graubard: What did you do after you finished the was programming the UNIVAC to run the PES tab- life table assignment, and who did you work with? ulations. I was very impressed with the research at Sirken: Unlike the U.S. Census Bureau where most the Census Bureau so after the post-doctoral ex- national population data are collected periodically pired I took a job with Morris Hansen’s group. I got in censuses, at NOVS, national vital statistics (births, involved in several projects. I particularly remember deaths, marriages and divorces) are compiled as by- working with Hal Nisselson and Ted Woolsey on the products of information reported on vital records. pretest of the National Health Survey in San Jose, Because vital records serve primarily as legal docu- California. Little did I realize then how relevant that ments, the information reported on vital records is experience would become in my future work. necessarily limited and virtually changeless. I had an Graubard: So was your academic career over? understanding with Dr. Dunn that after completing Sirken: I didn’t think so at the time. When I ar- the life tables I would work on sample survey meth- rived at the Census Bureau in 1951, my original plan ods to improve vital statistics. In 1955, just after the was to stay three months and then take an academic life table project was completed, I had a chance to appointment. In fact, I have been in government for over 55 years with brief visiting appointments in do just that when Bill Haenszel, a well-known epi- biostatistics departments at University of Califor- demiologist at the National Cancer Institute (NCI), nia at Berkeley and University of North Carolina at proposed a collaborative research project in which Chapel Hill. NOVS would design and test sample survey method- ologies to collect retrospective residence and smok- NATIONAL OFFICE OF VITAL STATISTICS ing histories for samples of deceased persons from their surviving relatives. With funding support from Graubard: What made you leave the Census Bu- NCI, a small statistical unit, including Mort Brown reau and work at the National Office of Vital Statis- and Jim Pifer and myself and a clerical staff, was tics? established in NOVS to conduct Haenszel’s pilot Sirken: It wasn’t my decision. There was a so- study. Soon after the successful completion of that called Reduction in Force (RIF) throughout govern- pilot, NOVS established a long-range research sam- ment soon after Dwight Eisenhower moved into the ple survey program to expand the scope and improve White House in 1952. I didn’t have tenure at the the quality of vital statistics by conducting retro- Census Bureau, and was RIF’ed in midyear 1953. spective sample surveys linked to birth and death A couple of months later, Morris Hansen got me a job at the National Office of Vital Statistics (NOVS) records (Sirken, 1963). which was quite a feat in view of the government- Graubard: So you were using the idea that people wide employment freeze on outside hires. Well, Mor- like Morris Hanson had promoted at the Census Bu- ris Hansen and Dr. Halbert Dunn, Director of NOVS, reau that by sampling you could expand the scope were old friends and Morris apparently convinced and improve the quality of vital statistics. Dr. Dunn that I was a very competent actuarial Sirken: The NOVS survey program was sustained mathematician. Later, when Dr. Dunn interviewed by conducting work for other government agencies. me, I noted my very limited knowledge (virtually For example, Haenszel expanded the lung cancer pi- none) in actuarial science, and luckily he thought lot study into a national mortality survey, and he I was being modest. arranged with the Census Bureau to collect informa- Graubard: So you launched into an area that you tion on smoking and residence histories for the na- had no real experience. tional population in the Current Population Survey Sirken: That’s right. However, I had a couple of (CPS). Thus, we were able to estimate national lung fine mentors. T. N. E. Greville, an outstanding actu- cancer death rates by smoking habits and residence arial mathematician and statistician who had been histories (Haenszel, Loveland and Sirken, 1962). As A CONVERSATION WITH MONROE SIRKEN 5

I recall, these findings were cited in the first report errors. This way of thinking helped me to recognize of the Surgeon General on Smoking and Health. and exploit research opportunities that I might oth- Graubard: And actually that’s one of the first erwise have overlooked. examples of a population-based case-control study, Graubard: That way of thinking explains how you where the mortality follow-back survey provided ex- got involved in research on network sampling and posure and other covariate information for the lung the cognitive aspects of survey methods. Both areas cancer cases and the CPS provided these variables seek to improve the survey quality by using informa- for the control sample of the population at risk. tion about the population being surveyed. It seems Sirken: Exactly. I have always felt that the linked to me that you took almost a personal responsibility mortality/population sample survey methodology de- for the total quality of your surveys—sampling and serves much more attention than it has received nonsampling errors. from epidemiologists. Sirken: I guess so. Generally, NOVS was not re- Graubard: Weren’t you also involved in designing sponsible for survey data analyses. That was the re- other surveys for federal health agencies? sponsibility of the contracting agency. But NOVS Sirken: Yes. NOVS developed the methodology was responsible for describing the data limitations, of the follow-back surveys linked to birth records, both sampling and nonsampling errors, and we took and with funding from the Division of Radiologi- those responsibilities very seriously. cal Health, U.S. Public Health Service, conducted the first national natality survey on the exposure of EARLY DAYS AT THE NATIONAL CENTER pregnant women to medical radiation. With funding FOR HEALTH STATISTICS from the U.S. Public Health Service, NOVS con- tracted with the Census Bureau for a CPS supple- Graubard: When and why did you leave NOVS? ment on the population’s utilization of the Salk vac- Sirken: In 1960, NOVS was merged with the Na- cine. Using data from the CPS polio supplement, tional Health Survey (NHS) to form the National NOVS produced the first national statistics on the Center for Health Statistics (NCHS). Without any utilization and effectiveness of the Salk vaccine action on my part, I became a charter member of (Sirken, 1962). Thereafter, the Public Health Ser- the NCHS, though for a couple of years afterward I vice often used the CPS supplements to monitor the remained in NOVS, which was renamed the Division immunization status of the national population. On of Vital Statistics (DVS). The NHS was the other another occasion, the U.S. Children’s Bureau asked NCHS division. The NHS, located in the Office of NOVS to conduct a survey on the prevalence of cys- the Surgeon General, was chartered in 1956 and was tic fibrosis, a debilitating and often lethal pediatric responsible for the National Health Interview Sur- disease, and to do so within something like a hun- vey (NHIS)—a large national face-to-face household dred days in order to comply with a Congressional survey that was and is still fielded by the Census Bu- request. An unexpected estimation problem in that reau. You may recall that at the Bureau in 1952, I medical provider survey ultimately led to the devel- had worked on the NHIS San Jose pretest, and next opment of a new kind of sampling called network year NHIS will be celebrating its 50th anniversary. sampling. These were very exciting days when the The NCHS is the official federal agency responsi- findings of the NOVS sample surveys were used in ble for producing national health statistics, and it is real time to address important public health prob- noteworthy that the Congressional Act that created lems. the NCHS empowered NCHS to develop the sur- Graubard: Much of your own early research and vey methodology most suitable for collecting health research collaborations involved measuring the qual- statistics. A lot of credit for establishing the NCHS ity of survey data. How did this come about? goes to Ted Woolsey whom I believe helped draft Sirken: Well, it started with my thesis and con- the enabling legislation, and he also directed the de- tinued with my work at the Census Bureau. Also, velopment of the NHIS. Forrest Linder was the first my training in demography was a factor. I tended NCHS Director. He was a demographer who had to think about the demographic aspects of survey formerly worked in the Vital Statistics Division at measurement errors, and in the back of my mind I the Census Bureau and had recently resigned from wondered how information about the population be- the United Nations to become the NHS Director. ing surveyed could be used in the survey measure- I knew Forrest Linder and the other NHS staff in- ment process to reduce sampling and measurement cluding Ted Woolsey, Ozzie Sagen, Phil Lawrence, 6 B. I. GRAUBARD, P. S. LEVY AND G. R. WILLIS

Walt Simmons and Earl Bryant because NHS and vital record linked surveys. I became the first DHRS NOVS occupied quarters in the same building and Director. the staffs often met for lunch. Graubard: What were the surveys that eventually Graubard: You’ve given us an overview of the for- came out of DHRS? mation of the National Center for Health Statistics Sirken: During my four years at DHRS, we estab- and noted some of the staff. What kind of place was lished the Master Facility Inventory of Health Care NCHS in those early days? Providers, the National Hospital Discharge Survey, Sirken: A very exciting place. Forrest Linder was a the National Nursing Home Survey and the Linked good administrator and had the vision of developing Birth Record Sample Survey. a family of national data systems that intersected Just before Dr. Linder retired in 1967, he appointed all important health-related activities of the popu- me the director of the newly created Office of Sta- lation that would be capable of meeting the increas- tistical Methods with responsibility for directing the ing needs for national health and vital statistics. I NCHS survey and statistical methodology programs. was particularly excited because fulfilling Forrest’s Earl Bryant was the deputy director. It was a won- vision implied the need for a strong survey methods derful job and offered many opportunities. Our work research program. was divided between serving as statistical consul- Even before the NCHS was established, the NHS tants and advisors to the NCHS programs and con- had begun to conduct pilot studies of the National ducting research relevant to the mission of NCHS Health Examination Survey, now the National Health that emerged from the consulting activities. We hired and Nutrition Examination Survey (NHANES)—a very talented people like Bob Casady, Paul Levy, Iris medical examination survey that uses mobile medi- Shimizu and you, and over the years we welcomed cal trailers to physically examine random samples of input from many outstanding statisticians includ- the national population. When the NCHS was estab- ing Jerzy Neyman, Z. W. Birnbaum, Gad Nathan, lished, the vital record follow-back surveys became Richard Royal, T. N. E. Greville, Phil McCarthy, an integral part of NCHS data systems. Soon af- Chin Long Chiang and Tom Jabine. Their reports ter NCHS was established, we began to think about were published in the NCHS Series 2 reports on Data developing a family of national health care provider Evaluation and Methods Research. surveys of hospitals, physicians, clinics, home health Graubard: I’m curious about how you were able services, etc. to establish these research collaborations and what As a result of an NCHS reorganization in 1963, kind of support did you get from NCHS. there were now four divisions including the Division Sirken: Of course, Center support was essential. of Health Records Statistics (DHRS). The DHRS Relationships with scientists in academia were ac- was responsible for health care provider surveys and tively encouraged and supported by NCHS. How-

Fig. 2. NCHS Seminar 1984. Front row l. to r.: Gad Nathan, Judy Lessler, Keith Eberhardt. Second row l. to r.: Monroe Sirken, Morris Hansen, Ben Tepping, Bob Fay. A CONVERSATION WITH MONROE SIRKEN 7 ever, I think the key to attracting outstanding peo- led to the development of network sampling. How ple is to have interesting problems for them to work did that happen? on. Sirken: In the New England Survey, the estimate Levy: During your many years at NCHS you’ve of cystic fibrosis prevalence was based on a sam- led a large number of projects. What have been your ple survey in which medical providers reported their major research interests and which projects do you cystic fibrosis patients. We did not appreciate until regard as your personal favorites? after the survey was conducted that most cystic fi- Sirken: Methodological research conducted by fed- brosis patients were treated by and eligible to be re- eral statistical agencies is severely constrained by ported by multiple medical sources. Hence, the con- short-term programmatic needs. Resources are hard ventional sampling estimate of cystic fibrosis preva- to come by to pursue many interesting problems lence which implicitly assumes each patient is linked that emerge as by-products of short-term-oriented to one and only one medical source would have been research. Nevertheless, by tapping a variety of sources biased. After I got Bill Birnbaum interested in this I was able to obtain funds to research some inter- estimation problem, we developed three unbiased es- esting problems myself and with collaborators, and timators of disease prevalence in stratified sample to foster methodological research relevant to long- surveys of medical providers with overlapping pa- term needs of federal statistical agencies. My fa- tient case loads (Birnbaum and Sirken, 1965). The vorite research projects are network sampling and estimators utilize information about the multiplicity integrated sample survey design. My main efforts to of linkages between patients reported in the sample foster research involved the CASM movement, which survey and their medical providers. The simplest of fostered research on the cognitive aspects of survey the estimators, the so-called multiplicity estimator, methodology, and a grants program that supports weights each patient report by the inverse of the pa- investigator-initiated research projects that are ori- tient’s multiplicity, namely, the inverse of the num- ented to the future needs of federal statistical agen- ber of different providers that treated the patient. cies. Levy: I remember in your original paper with Dr. Birnbaum, you referred to the unbiased estimation NETWORK SAMPLING technique as multiplicity sampling or multiplicity es- timation. When did you actually begin using the Levy: Let’s begin with network sampling. What term network sampling? is it and how does it differ from conventional sam- Sirken: Initially, our estimation technique was ap- pling? plied when surveys inadvertently used multiplicity Sirken: The essential difference between network rules and then it was called multiplicity estimation and conventional sampling is the kind of counting or multiplicity sampling. After demonstrating that rule used to link the elements of the target popula- multiplicity sampling is potentially more efficient tion to the selection units at which they are eligible than conventional sampling, I realized that our es- to be counted (Sirken, 1997). Conventional sampling timation technique also applied when multiplicity depends on unitary counting rules that uniquely link rules were deliberately used to improve sample sur- each population element to one and only one selec- vey efficiency (Sirken, 1970) especially in surveys of tion unit. Network sampling depends on multiplicity rare and elusive populations (Sudman, Sirken and counting rules that do not limit the number of se- Cowan, 1988). I coined the expression network sam- lection units that are linked to the same population pling because the precision gains of using multiplic- elements. For example, conventional sampling ap- ity instead of unitary rules depend on the statistical plies in household sample surveys that use the de properties of the networks formed by the linkages jure residence rule that uniquely links each individ- between population elements and selection units. ual to his/her usual place of residence. On the other For example, assume the rare disease prevalence sur- hand, multiplicity sampling would apply in house- vey is based on a simple random sample of persons. hold sample surveys that use the counting rule that In conventional sampling, each person responds for links individuals to their de jure and de facto resi- himself or herself. Suppose in network sampling, in dences. addition to each person responding for himself, the Levy: The solution of the estimation problem in person’s siblings respond for him. Assuming all pa- the New England Cystic Fibrosis Survey ultimately tients have the same number of siblings, k, and at 8 B. I. GRAUBARD, P. S. LEVY AND G. R. WILLIS most one sibling has the disease, the ratio of the diabetic. As I recall, diabetes prevalence estimates sampling errors of network and conventional sam- based on kinship rules were substantially larger and pling is approximately the inverse of k. considerably closer to independent sources of dia- Network sampling also has potential to improve betes prevalence estimates, and the network sam- the accuracy of survey estimates of household sur- pling errors were about half as large as the con- veys of sensitive populations where the reports of ventional sampling errors. Barry Graubard and I close friends and relatives may be more accurate worked on this project. than those of the persons themselves, and in house- Levy: As I recall, you and your collaborators were hold surveys of elusive populations that are not linked also busy expanding network sampling theory. to single places of residence. However, care must be Sirken: Because network sampling does not spec- taken in selecting the “right” counting rule because ify rules for selecting samples, technically speaking the rule that increases precision often decreases ac- it is not a sampling technique—it is an estimation curacy. technique. In our original paper, Bill Birnbaum and Levy: Is that what you meant by the “counting I developed three unbiased estimators for stratified rule strategy”? network sample surveys. Much of our work in ex- Sirken: Exactly. The counting rule strategy in- panding network sampling theory involved applying volves selecting the counting rule (unitary or multi- network sampling in more complex and novel sample plicity), which together with the other survey design survey designs, and deriving the unbiased estima- features minimizes the mean square error of the sur- tors and variances. For example, you and I collab- vey estimate for fixed costs. Selecting the optimum orated in deriving the unbiased estimator and vari- counting rules requires a good demographic and so- ance of ratios of random variables in stratified net- ciologic knowledge of the target populations. work sample surveys (Sirken and Levy, 1974), and Levy: What kinds of survey experiments did you Bob Casady and I collaborated with Gad Nathan of do to test the counting rule strategy, and who were Hebrew University on the network sampling designs your collaborators? of dual system surveys (Casady, Nathan and Sirken, Sirken: The experiments compared the sampling 1985). Also, I did some work on the components of and nonsampling errors of conventional and network variance of multiplicity estimators (Sirken, 1972). sampling in household sample surveys of rare and sensitive populations—the kinds of surveys that of- INTEGRATED SURVEY DESIGN ten challenge conventional sampling. The rare popu- lation experiments were cancer and diabetes preva- Levy: Network sampling research at NCHS was lence, and birth, marriage and death incidence. The interrupted in the early 1980s. What happened? sensitive issue experiments were alcohol and illicit Sirken: NCHS was quite a different place in the drug use. In these experiments, the de jure residence late 1970s and early 1980s than when established rule was compared with multiplicity counting rules in 1960, and new kinds of research problems were that incorporate the de jure rule. In the alcohol and emerging. I was Associate Director of Research and illicit drug use surveys, for example, the multiplic- Methodology and the Director of the Office of Re- ity rules were based on self and friendship linkages, search and Methodology and Jim Massey, Andy and in the rare population surveys, multiplicity rules White and Bob Casady succeeded by Randy Curtin were based on self and kinship linkages. For exam- were section chiefs. ple, Trish Royston and I worked on the design ef- By the early 1980s, the three major NCHS health fects counting rules in mortality surveys (Royston surveys, the NHIS, the health care provider surveys, and Sirken, 1978). and the National Health and Nutrition Examina- Levy: Didn’t one experiment involve an NCHS tion Survey (NHANES) were well established. Ad- data system? ditionally, two new population sample surveys had Sirken: Yes, an experiment was embedded in an been successfully fielded—the National Medical Ex- NHIS supplement on diabetes which used multiplic- penditure Survey (NMES) and the National Survey ity rules based on self and kinship relationships with of Family Growth Survey (NSFG). Some surveys, parents, children and siblings. NHIS respondents re- such as the NHIS, were continuous and others, like ported whether they or close relatives had diabetes, the NMES and the NSFG, were conducted periodi- and then reported the multiplicities of each reported cally. Each survey had been independently designed A CONVERSATION WITH MONROE SIRKEN 9 and was being independently implemented. Dorothy Levy: How did you intend to integrate NCHS’s Rice, then NCHS Director, established a Periodic- data systems and what made you think it would ity Committee of the Center’s senior staff to rec- work? ommend ways to reschedule the periodicity of con- Sirken: We proposed to use the NHIS listings of ducting the Center’s data systems in order to be households and persons as the master sampling frame able to maintain existing programs and undertake for NCHS’s other population-based sample surveys new programs. The Periodicity report compared the including the NHANES, NMES and NSFG (Sirken consequences of several short-term periodicity plans, and Greenberg, 1983). Subsequently, the proposal and mentioned a long-term research plan which en- was expanded, I believe at Iris Shimizu’s sugges- visioned the possibility of integrating the sample de- tion, to nest the health care provider surveys in the NHIS primary sampling units (psu’s). There were signs of NCHS’s independently designed population several reasons why it made sense to use the NHIS sample surveys. as the hub of the integrated survey design system. With almost 50,000 households interviewed annu- ally, NHIS was by far the largest of the NCHS pop- ulation sample surveys and provided opportunities to oversample relatively small population domains, like race and ethnicity. Consistently high response rates, greater than 90%, assured that NHIS sam- pling frames would be relatively complete. Also, the NHIS collects a vast amount of health and related data about individuals and households that are rele- vant to the analytic objectives of NCHS’s other pop- ulation surveys. Levy: So what happened next? Sirken: There were confidentiality and funding problems. The NHIS is conducted for NCHS by the U.S. Bureau of the Census. Following each Decen- nial Population Census, the Census Bureau selects the NHIS household samples for the next decade. The penultimate sampling stage of that sampling process involves using the listings of households enu- Fig. 3. Monroe Sirken as NCHS’s Associate Director of Re- search and Methodology, 1980. merated in the Decennial Population Census and that posed a serious administrative problem for the proposed integrated survey design because Census household listings are considered confidential Cen- sus data. Hence, any surveys that might in the fu- ture be linked to the NHIS would have to be con- ducted by the Census Bureau because the addresses of the NHIS households could not be disclosed to other contractors. I thought this restriction was in- compatible with the development of a robust inte- grated survey design program, so prior to redesign- ing the NHIS based on the 1980 Decennial Popu- lation Census, I proposed using an area sampling method that did not depend on Census listings. The Bureau was perfectly willing to use area sampling instead of Census listings but said that it would be substantially more expensive—something like an ad- Fig. 4. Dr. Albert Einstein and admirer, 1983. ditional million dollars. 10 B. I. GRAUBARD, P. S. LEVY AND G. R. WILLIS

I proposed the integrated sample design program In response to the Panel’s proposal, NCHS initi- and the NHIS sample design change to Dorothy Rice ated a research project that investigated the feasibil- and she approved it. The next problem was how ity and efficiency of the Population Based Establish- to get the additional million dollars needed for the ment Survey (PBES)—an establishment survey that NHIS redesign. The money was not included in the uses a population survey-generated sampling frame. existing or future NCHS fiscal year budgets, and as One of our first tasks was to derive expressions of a I recall, the budget requests had already been trans- two-stage PBES unbiased estimator and variance of mitted to the Assistant Secretary for Health. With the volume of transactions between households and administrative help from Marjorie Greenberg, then establishments. The problem eluded solution when working in the Office of the Assistant Secretary for the PBES was modeled as an establishment sample Health, the justification for the amended budget re- survey, but was easily solved subsequently when the quest went forward and Congress appropriated the PBES was modeled as a network sample household additional funds. survey using a multiplicity rule that links an estab- Levy: How did you implement the integrated sur- lishment’s transactions with all households to every vey design? household with which the establishment has trans- Sirken: Beginning in 1985, the first year that the actions (Sirken, Shimizu and Judkins, 1995). The area-based NHIS redesign was implemented, large PBES estimator and variance may be viewed as the pilot surveys were conducted using NHIS household two-stage cluster sampling adaptation of the mul- and person listings as sampling frames for the NMES tiplicity estimator and variance originally proposed and NSFG. In these surveys, minority households by Dr. Birnbaum and myself. were oversampled. The pilot studies demonstrated that integrated survey design was feasible and more COGNITIVE ASPECTS OF SURVEY efficient than the independently designed NMES and METHODOLOGY NSFG, and subsequently both surveys were linked Willis: What is CASM? to the NHIS. Currently, the Medical Expenditure Sirken: CASM is the acronym of the Cognitive Panel Survey, the NMES successor, conducted by Aspects of Survey Methodology. CASM research is Agency for Health Care Research and Quality, con- interdisciplinary and applies the theories and meth- tinues to use NHIS person and household listings as ods of the cognitive sciences in survey research. the sampling frames. Willis: Why would you, a mathematical statisti- Levy: Integrated survey design research ultimately cian in a federal statistical agency, become inter- led you back to network sampling. How did that ested in CASM? come about? Sirken: There was always a concern about the Sirken: In the early 1990s, a Panel of the Com- response error effects of NCHS’s survey question- mittee on National Statistics (CNSTAT) reviewed naires, and considerable effort went into designing NCHS’s plans for redesigning the health care provider questionnaires to make them as accurate and as surveys with samples of medical providers embedded respondent-friendly as possible. My Office of Re- in a subset of the NHIS psu’s as proposed by the search and Methodology (ORM) reviewed the ques- integrated survey design. Though the Panel liked tionnaires designed by NCHS staffs, and it was a the idea of linking sample designs of the health care difficult task because questionnaire design is a craft provider surveys to the NHIS, it favored linkage at rather than a science. I was ready and eager to em- the health care provider level rather than at the psu brace any scientific discipline that could provide a level. In essence, the Panel proposed substituting the more scientific basis for survey questionnaire design. listings of health care providers reported by NHIS Willis: Do you recall the time and circumstances households for the complete sampling frames that when you first became interested in cognition and listed all providers then in use by the health care survey research? provider surveys. With the nation’s health care de- Sirken: About 1980, I attended a two-day work- livery system undergoing rapid change, the Panel shop convened by the Bureau of Justice Statistics felt that the NHIS-generated sampling frames would that discussed the potential uses of cognitive meth- be easier to construct and maintain than the com- ods to improve reporting of victimization by house- plete sampling frames. hold respondents in the National Crime Survey. The A CONVERSATION WITH MONROE SIRKEN 11 workshop introduced me to a recent paradigm change proposal over lunch with Judy Lessler at the Joint in psychology with the two-stage stimulus/response Statistical Meetings in 1983 or ’84, I believe, and model being replaced by a three-stage she was very much interested. So after I got assur- stimulus/cognition/response model with the focus ances that, if awarded, the NSF grant could cover on cognition and how the mind works during the her expenses and salary, and Judy got permission cognitive stage. I got involved in the CASM move- for a leave of absence from Research Triangle In- ment in the early 1980s when I participated in the stitute, NCHS submitted the CASM proposal and Advanced Research Seminar on the Cognitive As- was awarded the NSF grant. The following sum- pects of Survey Methodology, later known as the mer Judy moved to Washington, DC, and during CASM I Seminar. the next year or so she, in collaboration with Roger Willis: What was the CASM I Seminar and what Torangeau and Bill Salter, directed the CASM ex- was your role in it? periment. Findings of that study were published as Sirken: The CASM I Seminar was sponsored by the first report in the newly established series of CNSTAT and supported by a National Science Foun- NCHS reports on Cognition and Survey Measure- dation (NSF) grant. It was organized by Judy Tanur ment. based on a proposal developed by Stephen Fienberg Findings of the CASM experiment were unexpected. and Miron Straf. Tom Jabine was a consultant to They indicated that conventional and cognitive meth- the CNSTAT project. About 25 invited cognitive ods of testing the NHIS questionnaires were comple- scientists and statisticians attended a six-day meet- mentary rather than competitive as Bob Fuchsberg ing in the summer of 1983 on the eastern shore of and I had originally anticipated. The testing meth- Maryland and a two-day follow-up session the fol- ods exposed different rather than the same kinds lowing January in Baltimore. Questionnaires used of response problems. I recall meeting with Trish in several national surveys were discussed at the Royston and Debbie Trunzo, the ORM staff that Seminar including those of NCHS surveys, partic- reviewed NCHS questionnaires, and discussing im- ularly NHIS. The Seminar was essentially the start plications of the experiment’s findings for their work of the CASM movement to foster interdisciplinary in reviewing NCHS questionnaires. We decided that research that benefited survey research and cogni- the experimental findings implied that survey ques- tive psychology. An important factor in the Semi- tionnaires should be laboratory tested before they nar’s success was the availability of NSF funding to are field tested. I concluded that a permanent Ques- support meritorious CASM research projects that tionnaire Design Laboratory should be established were developed at the Seminar. At the close of the to develop NCHS questionnaires. That’s how the June meeting, participants were encouraged to de- idea for a questionnaire design research laboratory velop CASM research proposals based on the ideas was born. they had presented at the Seminar. ORM’s proposal to establish a permanent Ques- When the CASM I Seminar reconvened, Robert tionnaire Design Research Laboratory (QDRL) was Fuchsberg, NHIS Director, and I proposed a research approved by the NCHS Director, Dr. Manning Fein- project that compared two methods of designing lieb, and ORM was given additional space for the the questionnaires for the forthcoming NHIS sup- Laboratory. The QDRL was established in 1985 with plement on dental health. Using the traditional test- Trish Royston and Debbie Trunzo as co-directors. ing method, the dental health questionnaire would They deserve a lot of credit for the success of the be tested under normal interviewing conditions in QDRL. Testing and developing the questionnaires which Census enumerators interviewed NHIS respon- of the NHIS supplements, which changed annually, dents at their households. Using the proposed cogni- became the centerpiece of the Laboratory’s work tive method, the dental health questionnaire would schedule. Laboratory methods were very successful be tested in a laboratory setting in which profession- in detecting and eliminating the kinds of glitches in ally trained staffs conducted cognitive interviews with survey questionnaires that are often missed in field recruited respondents. testing. Soon requests to test the questionnaires of Willis: Was your research proposal funded by NSF? other federal agencies far exceeded the Laboratory’s Sirken: Initially, I was reluctant to submit our pro- capacity and larger quarters and staff were allot- posal to NSF because ORM staff and I were already ted. News of the success of NCHS’s QDRL spread overcommitted. However, I happened to discuss the quickly to other federal statistical agencies, and soon 12 B. I. GRAUBARD, P. S. LEVY AND G. R. WILLIS the Bureau of the Census and the Bureau of Labor Sirken: Yes and no. The influence and success of Statistics established their own laboratories. Cur- the QDRL in applying cognitive techniques to de- rently, cognitive testing of survey questionnaires is sign survey questionnaires far exceeded my expecta- the common practice and several federal statistical tions. On the other hand, I regret the demise of the agencies, here and abroad, have in-house question- Collaborative Research Program in NCHS and wish naire testing facilities. it could be revived. It was great while it lasted. Later, the NCHS, in collaboration with NSF, es- Willis: There was also the CASM II Seminar in tablished the National Laboratory for Collabora- 1996. How did that develop and what were its ob- tive Research in Cognition and Survey Measurement jectives? (Sirken, 1991). It had two components. The QDRL Sirken: About 1993, the tenth anniversary of the was one component. It tested and designed the data CASM I Seminar, I proposed convening a CASM collection instruments of federal surveys and was II Seminar to expand the scope of CASM research funded by NCHS and by the reimbursable work the beyond questionnaire design to other survey design Laboratory did for other agencies. Linda Pickle and features, and to expand the scope of interdisciplinary Doug Hermann did pioneering work in the QDRL on relationships beyond cognitive psychology to other disciplines. The CASM II Seminar was jointly funded the cognitive aspects of designing statistical maps. by NSF and NCHS. It was organized by a plan- The second component, the Collaborative Research ning committee of interdisciplinary-minded survey Program, was funded by the NSF, and fostered ba- researchers, and was managed and supported by ORM sic research on cognitive issues germane to improv- staff, especially Susan Schechter and Karen Whitaker ing data collection instruments of federal surveys. It and Tom Jabine, an ORM consultant. The six-day included a Contract Research Program headed by Seminar met in Charlottesville, Virginia during June, Jared Jobe and a Visiting Scientist Program. About 1996 and featured 15 commissioned papers by sur- ten CASM research projects were completed before vey methodologists and cognitive scientists, which the Collaborative Research Program expired in the along with remarks of discussants appear in the book early 1990s. Findings of these research projects ap- published by Wiley (Sirken et al., 1999). pear in NCHS Series 6 reports on “Cognition and Like the CASM I Seminar, the CASM II Seminar Survey Measurement.” sought to develop interdisciplinary research project Willis: I recall working with you and Gad Nathan proposals. Unfortunately, funding was unavailable on a project funded by the Visiting Scientist Pro- to support CASM II research proposals. Even before gram. the CASM II Seminar convened, I was beating the Sirken: Yes, Gad was a Visiting Scientist from the bushes looking for funds. Hebrew University and the project was “Cognitive Willis: And that’s how you got involved in the Aspects of Designing Sensitive Survey Questions.” Funding Opportunity in Survey Research? Our experiments compared cognitive and behavioral theories of the likelihood of truthful response (LTS) THE FUNDING OPPORTUNITY IN SURVEY in surveys on illicit drug use. We did not actually RESEARCH recruit drug users in the Laboratory, but asked lab- Sirken: A potential source of funding for CASM oratory subjects to respond to survey questionnaires II research proposals surfaced toward the end of as if they were the drug users depicted in vignettes. the CASM Seminar II when Cheryl Eavey, Head In brief, the cognitive theory posited that the LTS of NSF’s Methodology, Measurement and Statistics depends on the drug users’ perceptions of the disclo- program, offered to administer and fund a grants sure risks of truthful response and the consequences, program in basic survey research for a three-year and the behavioral theory posited that the LTS de- period if a consortium of federal statistical agencies pends on two survey features, the type of illicit drug provided matching funds. After some false starts, (marijuana or cocaine) and the extent of privacy I took NSF’s proposal to the Federal Committee protection (confidentiality or anonymity) the sur- on Statistical Methodology (FCSM), an interagency vey provided. The comparisons slightly favored the committee of federal statisticians to which I belong, cognitive theory. and requested its help in recruiting a consortium of Willis: So overall did the Laboratory work out as federal statistical agencies to put up the matching you had hoped? funds. A CONVERSATION WITH MONROE SIRKEN 13

Fig. 5. CASM II Seminar, 1996. From l. to r.: Judy Tanner, Norman Bradburn, Barbara Wilson, Susan Schechter, Monroe Sirken.

Fig. 6. NCHS Questionnaire Design Research Laboratory, 2006. From l. to r.: Monroe Sirken, Mary Moien, Barry Graubard.

Willis: How did FCSM react to the NSF proposal? Willis: Is the Funding Opportunity working out Sirken: FCSM appointed a research subcommit- as you hoped? tee to draft a proposal to the Interagency Commit- Sirken: During its initial seven years, 1999–2006, tee on Statistical Policy (ICSP), a committee of the the Funding Opportunity made grants totaling more directors of about 15 of the largest federal statisti- than four million dollars and supported about 25 cal agencies. Thirteen ICSP agencies agreed to in- investigator-initiated research projects. It is too early dividually support the Funding Opportunity during to assess the long-term contributions of these projects, a three-year period and in January, 1999, the NSF but I’m optimistic. Arguably the most outstanding issued the first announcement of the Funding Op- early achievement of the Funding Opportunity has portunity in Survey Research, a grants program in been the successful development of the infrastruc- interdisciplinary survey and statistical research that ture of a complex grants program that is oriented to is oriented to the future methodological needs of long-term needs of the federal statistical system, and federal statistical agencies. In 2001, the ICSP agen- is funded jointly by a consortium of federal statisti- cies and the MMS extended their agreement beyond cal agencies and the NSF and administered jointly 2002. by the FCSM research committee and the NSF. The 14 B. I. GRAUBARD, P. S. LEVY AND G. R. WILLIS

Funding Opportunity provides a vehicle for statis- Birnbaum, Z. W. and Sirken, M. G. (1965). Design of tical agencies in our decentralized federal statistical sample surveys to estimate the prevalence of rare dis- system to collaborate in funding methodological re- eases: Three unbiased estimates. National Center of Health search which none of them could individually afford. Statistics Series 2 No. 11. MR0523738 Casady, R. J., Nathan, G. and Sirken, M. G. (1985). Al- It provides the FCSM with a vehicle to carry out ternative dual system network estimators. Internat. Statist. its basic mission of improving the quality of federal Rev. 53 183–197. MR0959037 statistics. Yes, I would say, the Funding Opportunity Haenszel, W., Loveland, D. B. and Sirken, M. G. (1962). has worked better than I ever imagined in fostering Lung cancer mortality as related to residence and smoking basic survey research that is oriented to the needs histories. I. White males. J. Nat. Cancer Inst. 28 947–1001. of federal statistical agencies. Royston, P. N., Sirken, M. G. and Bergsten, J. (1978). Bias and sampling errors of mortality counts based on RECENT YEARS: 1996– network surveys. In Proceedings of the American Statis- tical Association, Social Statistics Section 471–475. Amer. Willis: I think we have pretty well covered your Statist. Assoc., Alexandria, VA. major activities until the late 1990s. What have you Sirken, M. G. (1962). National participation trends, 1955– been doing lately? 61 in the poliomyelitis vaccination program. Public Health Sirken: I have been a Senior Research Scientist at Reports 77 661–670. NCHS during the past ten years. I spend full time Sirken, M. G. (1963). Research uses of vital records in vital conducting and fostering survey and statistical re- statistics surveys. Milbank Mem. Fund Quar. 41 309–316. search as I no longer have administrative or policy Sirken, M. G. (1970). Household surveys with multiplicity. J. Am. Statist. Assoc. 65 257–266. responsibilities. I divide my time between conduct- Sirken, M. G. (1972). Variance components of multiplicity ing statistical research mostly on network sampling, estimators. Biometrics 28 869–873. often in collaboration with Iris Shimizu (Sirken and Sirken, M. G. (1991). The role of a cognitive laboratory in Shimizu, 2007), and fostering interdisciplinary sur- a statistical agency. In Statistical Policy 20 268–277. Office vey research, mostly as chair of the FCSM Research of Management and Budget. Committee that steers, oversees and evaluates ac- Sirken, M. G. (1997). Network sampling. In Encyc. of Bio- tivities of the Funding Opportunity in Survey and stat. 2977–2986. Wiley, New York. Statistical Research (Sirken, 2004). I find the work Sirken, M. G. (2004). Charting the interdisciplinary his- of conducting and fostering scientific research very tory of the funding opportunity in survey and statistical 36 satisfying and highly recommend that combination research. Statistical Policy Working Paper . Available at http://www.fcsm.gov/reports. of activities to senior scientists. Sirken, M. G. and Greenberg, M. (1983). Redesign and integration of the population based health survey program. ACKNOWLEDGMENTS In Proceedings of the 44th Session of the Internat. Statist. Many thanks to Mary Moien for organizing and Inst. Madrid. coordinating the conversation, to Karen Whitaker Sirken, M. G. and Levy, P. S. (1974). Multiplicity estima- J. for making arrangements to conduct the conversa- tion of proportions based on ratios of random variables. Amer. Statist. Assoc. 69 68–73. tion in the NCHS Questionnaire Design Research Sirken, M. G., Shimizu, I. M. and Judkins, D. (1995). Pop- Laboratory, and to Debbie Trunzo, Ed Korn and ulation based establishment surveys. In Proceedings of the Tom Fears for editorial comments. Opinions expressed American Statistical Association, Section on Survey Re- in this conversation are Monroe’s and do not neces- search Methods 470–473. sarily represent the views of the National Center for Sirken, M. G., Hermann, D. J., Schechter, S., Health Statistics. Schwartz, N., Tanur, J. and Torangeau, R., eds. (1999). Cognition and Survey Research. Wiley, New York. Sirken, M. G. and Shimizu, I. M. (2007). Linked surveys of REFERENCES health services utilization. Statistics in Medicine 26 1788– Birnbaum, Z. W. and Sirken, M. G. (1950). Bias due to 1801. non-availability in sample surveys. J. Amer. Statist. Assoc. Sudman, S., Sirken, M. G. and Cowan, C. D. (1988). Sam- 45 98–111. pling rare and elusive populations. Science 240 991–996.