Appendix A: Survey and Reporting Methodology Citywide Customer Survey Results Contents Overview........................................................................................................................................1 Developing the Questionnaire..................................................................................................2 Developing Survey Items ...................................................................................................................2 Pilot Testing..........................................................................................................................................2 Selecting Survey Recipients ......................................................................................................4 Selecting Households..........................................................................................................................4 Sample Size by Community Board ...................................................................................................4 Selecting the Individual......................................................................................................................5 Supplementing the Sample ................................................................................................................5 Survey Administration and Response.....................................................................................6 Margin of Error.............................................................................................................................7 Data Entry......................................................................................................................................8 Survey Analysis............................................................................................................................9 Weighting the Data.................................................................................................................................9 Analyzing the Data...............................................................................................................................12 Survey Results....................................................................................................................................12 Indices .................................................................................................................................................12 Page i Tables Table 1: Number of Households Sampled by Community Board .................................................5 Table 2: Margins of Error for Respondent Subgroups ..................................................................7 Table 3: Hypothetical Weighting of Survey Results by Gender ....................................................9 Table 4: New York City Citizen Survey Weighting Table ............................................................10 Table 5: Statistical Properties of the Indices...............................................................................13 Page ii Citywide Customer Survey Results Overview The NYC Feedback Citywide Survey was conducted from June 13, 2008 to August 15, 2008. The survey contained a total of 34 questions (some with multiple parts) and took approximately 15 minutes to complete. Surveys were distributed across all 59 community boards (CBs) so results would represent neighborhoods from all over the City. A total of 24,339 surveys were completed by mail or online. An average of 4,439 residents from each borough completed the survey with highest percentage coming from Brooklyn (7,214 surveys) and the lowest percentage coming from Staten Island (1,138 surveys). Responses from CBs ranged from as many as 645 (for Queens Community Board #1) to as few as 253 (for Queens Community Board #12), and the average number of responses from each community boards was 376. The NYC Feedback Citywide Survey and report of results was conducted by the Mayor’s Office of Operations, the Office of the Public Advocate, The Fund for the City of New York and the team from National Research Center (NRC). NRC is a well-known and highly regarded citizen survey research firm that operates world wide, and has conducted hundreds of resident surveys in more than 40 states and abroad. The NRC team included staff from NRC, as well as NRC senior fellow, Gregg VanRyzin of Rutgers University, Harry Hatry of the Urban Institute and Chris O’Brien and Neil Harrison of Diamond Consulting. Appendix A: Survey and Reporting Methodology Page 1 Developing the Questionnaire The NYC Feedback Citywide Customer Survey was created as part of Mayor Bloomberg’s performance management system. It was created to permit regular monitoring of residents’ perspectives about the quality of life in New York City (NYC) and the quality of services provided. Stakeholders from the Mayor’s Office of Operations, The Fund for The City of New York and the Public Advocate’s office worked in partnership with survey consultants over a several week period to assure that critical services and characteristics of community quality were included in the survey. Emphasis was placed on including questions for a broad range of services so that many City agencies would gain reliable resident perspectives to help determine if the high quality of service delivery sought was being achieved. Developing Survey Items To determine which services should be covered in the survey, the survey team examined existing performance measures in the Mayor’s Management Report and the Citywide Performance Reporting tool. In addition, the heads of City agencies and Ester Fuchs of Columbia University were consulted. The 2000 and 2001 New York City surveys of residents were also reviewed. Due to the large number of City agencies, not all could be included on the survey; therefore, certain services were excluded because they were either not direct services to New York City residents or would be unfamiliar to most residents. The survey included three types of survey items, in addition to demographic questions: 1. Overall “citywide” perception of quality of life and City services 2. Satisfaction rating of particular City services and the living conditions affected by these services 3. Frequency of using a particular service, or if the respondent used the service at all Response scales for evaluative items were created to have four points, avoiding a neutral medium. Nearly all evaluative items were on a scale of “excellent,” “good,” “fair” and “poor;” with a few items on a scale of “very safe,” “somewhat safe,” “somewhat unsafe” and “very unsafe.” The survey contained a total of 121 items across 34 questions. The survey took approximately 15 minutes to complete Pilot Testing Special care was taken to ensure the layout of the survey was easy to read and that the order of the questions flowed in a way that would make sense to recipients. Once a final survey draft was developed it was translated into Spanish, Russian and Chinese. Pilot testing in the form of one-on-one interviews was done with 24 residents across all five boroughs: 10 English speaking residents, four Spanish speaking residents, four Russian speaking residents, three Mandarin speaking residents and three Cantonese speaking residents. Half of the interviews were conducted by allowing the participant to complete the survey on their own. The interviewer then asked the participant a series of questions about specific items on the survey. The other half of interviews were conducted through a “think aloud” process where participants provided feedback to the interviewer and answered interviewer questions as Page 2 Appendix A: Survey and Reporting Methodology Citywide Customer Survey Results they took the survey. All participants were asked if they felt anything was missing from the survey, if anything was asked that should not have been asked and what they thought of the overall look and flow of the survey. Spanish, Russian and Chinese pilot tests were completed May 20 and 21, and English interviews were conducted on May 16, 2008. Feedback from the survey indicated a few minor changes to the survey including specifying time of day riding the subway, removing public health-related acronyms and clarifying that all five boroughs were included when asking about “New York City.” The Chinese interviews also uncovered the need to provide the survey in both traditional and simplified Chinese. Appendix A: Survey and Reporting Methodology Page 3 Selecting Survey Recipients “Sampling” refers to the method by which survey recipients are chosen. A “sample” is a subgroup of all potential respondents in the population, selected randomly so that their average results are characteristic of the whole. In this way, a random sample helps ensure that the results can be generalized to all adult citizens in the population. There are three important criteria for doing this: 1. That each adult in the population has a chance of being included in the sample. 2. That the number of selected respondents is large enough, as shown by a statistical analysis of the reliability of results. 3. That the respondents be statistically representative of the whole City, in terms of such characteristics as their demographics and their locations, distributed throughout each community board (CB). A key objective of this survey was to state results reliably within every CB. Therefore a goal of 18,000 respondents citywide was set to help achieve a minimum of 250 responses in each CB. Selecting Households All