Page 1 of 226 Version 01 Codebook ------------------- CODEBOOK APPENDIX FILE 1997 PILOT STUDY (1997.PN) >>1997 NES Pilot Technical Note - Randomization Problem April 24, 1998 The Surveycraft CATI system's 'Random Number Generation' features and their Effects on Analysis of the 1997 NES Pilot "Group threat" Experiment. Steve Heeringa, Division of Survey Technologies, Survey Research Center Executive Summary: A problem has been identified in the random assignment of treatments in an experimental question module of the 1997 NES Pilot survey instrument. The randomization problem has been linked to unexpected correlation in sequences of random number calls made within the Surveycraft computer-assisted interviewing system. The problem does produce an unbalanced distribution of sample cases to the cells of the factorial experimental design but does not lead to a bias in the interpretation of the experimental results. Details are provided below. A report that analyzes these items is the 1997 pilot study report by J. Bowers. A portion of the 1997 NES Pilot questionnaire (section 'J') includes a "group threat" factorial experimental design to study question order and 'threat level' treatment effects in a series of items that explore respondent views and prejudices toward African-Americans and Christian Fundamentalists. The full design involves 2 question sequence orderings - African-Americans first or Christian Fundamentalists first; 2 levels of intended "threat" - high and low; and 3 'threat domains': political, social and economic. The Survey Craft computer assisted interview (CAI) application used an internal random number generator to determine each subject's assignment to target group order and threat level for the questions about each target group. A different Surveycraft function was used to randomize the order of the three threat domains, once the group and threat level were determined. The intent of the CAI programming was to randomly assign the group order, threat level by group and threat domain for each respondent. Complete randomization of choice for each of these three experimental components is expected to yield equal numbers of cases at each combination of treatment for the 2 x 2 x 3 factorial design. In practice, due to sampling variability inherent in the randomization process, the actual counts in each experimental cell will be distributed about the expected sample size for each experimental cell. Within the Surveycraft CAI questionnaire for the 1997 NES Pilot, the random assignment of group order and threat level was determined by a call to an internal system random number generator. Examination of the final sample size distribution across the cells of this experimental question module suggests significant departures from the equal sample size per cell assumption. Specifically, there appears to be a problem in the randomization assignment for group order and threat level. Table 1 compares the expected and actual distributions of 1997 NES Pilot sample to experimental cells: Table 1 1997 NES Pilot Section J Question Experiment. Expected and Actual Distribution of Respondents to Treatment Categories. http://www.electionstudies.org/studypages/1997pilot/APPPIL97.txt 10/20/2009 Page 2 of 226 Target Group Order Threat Expected Actual Level Respondents Respondents First Series African Americans High 138 181 Low 138 116 Christian Fundamentalists High 138 53 Low 138 202 Second series African Americans High 138 100 Low 138 197 Christian Fundamentalists High 138 114 Low 138 141 Through analysis of actual random numbers generated in the course of the 1997 NES Pilot computer-assisted interviews and communication with the authors of Surveycraft, the randomization problem has been traced to Surveycraft's handling of random number seeds in sequential calls of the random number function. Our review finds that the initial random number draws to determine the target group for the first question sequence were performed correctly. Observed variation in numbers of cases assigned at random to the African-American (n=297) and Christian Fundamentalists (n=255) target group question order are due to sampling error in the random draws of binomial (0,1) indicator variables. Since the random draws to determine threat level in the first and second question sequences are correlated with this initial random draw they also are pure random numbers (albeit not independent of the initial draw). The randomization of the experiment is therefore not affected by the problem-the joint probability that a respondent receives a particular configuration of experimental treatments is independent of respondent characteristics or the sample design. Unfortunately, the correlated sequence of random numbers does affect the balance of the distribution of subjects to the experimental design cells. This will have an unspecified, but negative effect on the power to detect effects of target group ordering and threat level that are the object of the factorial experimental design. The third factor in the experimental design, random ordering of each question representing a threat domain, was performed by a separate Surveycraft internal function. To the best of our ability to test the mechanism, this dimension of the experiment appears free of the randomization problem identified for the group order and threat level experimental conditions. ISR/SRC has corrected the problem which created this situation, working with Surveycraft authors to identify programming changes and conventions that now permit independent random number sequence generation directly within the system. Random numbers to determine assignments to experimental treatment in question sequences were drawn in advance, tested for independence and preloaded for use by the interviewing application. These simulations demonstrated that sequences of independent random assignments to treatments are now functioning within the SRC Surveycraft CATI system. >> 1997 CONTACT ISSUE MASTER CODE Codes below were used for "what was the issue involved" follow-ups to contact http://www.electionstudies.org/studypages/1997pilot/APPPIL97.txt 10/20/2009 Page 3 of 226 with six categories of public officials and two questions on mobilization via direct mail and mass media advertising. Codes were applied for up to 3 mentions each: Health and Social Welfare 01. Social Security, saving Social Security. Help to get benefits. (All mentions of aid to the elderly except Medicaid.); Aged/Elderly mentions; retirement. 02. Medicaid/Medicare issues; saving the Medicare system. 03. Veterans Administration; benefits/issues. Getting help from the VA. 04. Welfare; getting public assistance; food stamps. 05. Availability of Medical Care; need for national health insurance; health issues. 06. Drugs; efforts against the spread of drugs; war on drugs. 07. Smoking/tobacco issues. Regulation of the industry; taxes on cigarettes. Support for the tobacco industry. 08. Pro-life issues; anti-abortion; anti partial-birth abortions. Right to life.; abortion, partial-birth abortions. 09. Abortion rights; pro partial-birth abortions; from Planned Parenthood. 10. Womens rights; ERA; equal pay in the workplace. 11. Gay/Lesbian rights. 12. Religious issues; religious rights or freedom. 13. Minority issues; affirmative action; minority rights. 14. Childrens issues; childrens welfare; child abuse; programs for at risk youth. 15. Rights of the disabled/handicapped. Economic/Employment Issues 20. Budget deficit; government spending; the Balanced Budget Amendment. 21. Taxes; taxes are too high; Tax credit for children/family. Tuition tax credits. 22. Deregulation of business/industry; airline, banking, telephone. Keeping prices low through competition. Anti-trust mentions; taxes on a specific industry (non-tobacco). 23. Unions; power and stature of unions (all mentions); labor issues; strikes. 24. Employment; how hard it is to find a job. http://www.electionstudies.org/studypages/1997pilot/APPPIL97.txt 10/20/2009 Page 4 of 226 25. Work related; having to do with ones own job/business/industry. 26. Housing affordability; ability to get a mortgage. Other Specific Issues 40. Environmental; ecology; pollution. 41. Rapid transit. 42. Immigration; against foreigners taking jobs or being on welfare, etc. 43. English as national/state language. 44. Gun control. Brady Bill mentions; from the NRA. 45. Burning Flags; desecrating the flag. 46. Farm issues (all mentions). 47. Natural disaster relief. Help/aid from floods, hurricanes, etc. 48. Television content; what's on TV; harmful to young minds; children having access to the Internet. 49. Foreign Policy (all mentions). Foreign aid; international. All nations or hotspots; NAFTA; foreign trade; being in the UN. 50. Military/defense matters. Weapons; military budget (too much or too little). 51. Term limits. 52. Fine art funding. 53. Prayer in school. 54. Animal rights. 55. Police protection/Public Safety. 56. Fishing rights (between US and Canada). Boundary problem. 57. Casinos/gambling. The Indian casino. 58. Funding of money to city/state from federal government (NA what for). 59. Mandates to cities/states without funding (NA what for). Primarily Local Issues 70. Budgets; spending concerns (local only). 71. Education; need for better schools. (All mentions except sex education). 72. Sex education. http://www.electionstudies.org/studypages/1997pilot/APPPIL97.txt 10/20/2009 Page 5 of 226 73. Local ordinances; signs, zoning, drainage, land use; growth management; Beautification laws; residency laws for employment. 74. Taxes (local