A Report by a Panel of the
NATIONAL ACADEMY OF PUBLIC ADMINISTRATION
For the Department of Homeland Security
April 2007
ASSISTANCE TO FIREFIGHTERS GRANT PROGRAM:
Assessing Performance
Panel James Kunde, ∗ Panel Chair Paul D. Brooks Glenn Corbett Harry Hatry* Bruce D. McDowell* Darrel W. Stephens*
* Academy Fellow Officers of the Academy
Valerie A. Lemmie, Chair of the Board Jonathan D. Breul, Vice Chair Jennifer L. Dorn, President and Chief Executive Officer J. Christopher Mihm, Secretary Franklin S. Reeder, Treasurer
Project Staff
J. William Gadsby, ∗ Vice President for Academy Studies Terry F. Buss, PhD, Program Area Director F. Steven Redburn,* Senior Project Advisor James Frech, Senior Project Advisor Rob Cimperman, Senior Project Advisor (Pivotal-Insight, Inc) Ednilson Quintanilla, Research Associate Martha S. Ditmeyer, Senior Administrative Specialist
The views expressed in this report are those of the Panel. They do not necessarily reflect the views of the Academy as an institution.
National Academy of Public Administration 1100 New York Avenue, N.W. Suite 1090 East Washington, D.C. 20005 www.napawash.org
First published April 2007
Printed in the United States of America ISBN 1-57744-149-4
Academy Project Number: 2099
* Academy Fellow
ii FOREWORD
Now in its fourth year of operation, the Assistance to Firefighters Grant program (AFG) seeks to strengthen fire and emergency medical service response in communities across the nation. It awards grants directly to local fire departments to enhance their ability to protect the health and safety of the public and firefighting personnel who face fire-related hazards.
The U.S. Department of Homeland Security, which administers AFG, asked the National Academy of Public Administration to identify potential new strategic directions for the program and recommend ways to most effectively plan, manage and measure accomplishments. An Academy Panel found that AFG could pursue several new directions to improve its impact on public health and safety. These include national preparedness for catastrophic events; prevention in addition to response; social equity to address needs of underserved communities; expanded regional cooperation; and a reevaluation of emergency medical services.
Many thanks go to the members of the expert Academy Panel who oversaw this project. Their insights and guidance were substantial, as were the dedicated efforts of the project team. My sincere appreciation also goes to the AFG leadership, whose openness and accessibility were critical to the success of this study.
We hope that the findings and recommendations contained in this report help AFG to enhance its already well-managed program.
Jennifer L. Dorn
President and Chief Executive Officer
iii iv TABLE OF CONTENTS
FOREWORD...... iii
ACRONYMS...... xiii
EXECUTIVE SUMMARY ...... xv
CHAPTER 1: INTRODUCTION...... 1
The Academy Study...... 1
Overview of the Fire Grants Program...... 1
Study Methodology...... 2
Organization of This Report ...... 3
CHAPTER 2: THE AFG PROGRAM AND ITS USES...... 7
Overview of the AFG Program...... 7
History ...... 8 Eligible Uses of Funds...... 10 Program Priorities and Award Criteria Development ...... 12 Grant Award Process...... 14
AFG Funding of and Awards to Fire and EMS ...... 15
Analysis of AFG AWARDS for FY 2005...... 17
CHAPTER 3: THE FIRE, EMS AND NATIONAL PREPAREDNESS PROBLEMS ...... 23
The Fire Problem ...... 24
Trends in Death and Injury of Civilians...... 25 Career and Volunteer Firefighters...... 29 Trends in Firefighter Deaths...... 30 Trends in Fire-Related Property Loss...... 33 Prevention...... 33 Fire Expenditures in Proportion to the Economy...... 40
v The Emergency Medical Services Problem...... 45
Rural EMS...... 47 Aging and EMS ...... 48 On-the-Job Injuries and Deaths of EMS Personnel...... 48
The National Preparedness Problem...... 49
Fire Service Preparedness...... 50 EMS Preparedness...... 50 Preparedness for Coordinated Response ...... 51
Summary...... 52
CHAPTER 4: THE AFG PROGRAM IN CONTEXT ...... 55
Historical Developments...... 55
Federal, State and Local Government Roles...... 57
The Current Strategy Focused on Building Response Capabilities...... 62
Cost-effectiveness of the AFG Program...... 67
Leveraging Cooperative Relationships...... 72
Aligning to the All-Hazards Policy of DHS...... 73 Other Areas of Federal Interest: Commercial Markets, EMS, Social Equity ...... 75
Balancing Federal with Community Interests...... 76
CHAPTER 5: ALTERNATIVE VIEWS OF THE PROGRAM...... 79
Strategic Questions ...... 81
Views of Stakeholders...... 82 Disconnects and Strategic Choices...... 91 Views of a Strategic Review Panel ...... 92
Framing the Strategic Options ...... 93
vi
CHAPTER 6: STRATEGIC DIRECTIONS...... 95
Helping Fire Departments and EMS Organizations Adapt...... 96
AFG and the National Preparedness Goal ...... 96
Targeting Community and National Risk...... 97
DHS National Preparedness Plan and the Universal Task List...... 98 Community Risk Assessments ...... 102 Fitting the AFG into the Future...... 103
CHAPTER 7: CONTINUOUS STRATEGIC PLANNING...... 107
OMB PART ...... 107
Continuous Strategic Planning...... 110
The Need to Coordinate Federal Fire Safety and Other Emergency Preparedness and Response Planning ...... 112 Setting Long-Term Performance Goals for the AFG Program ...... 113
Findings and Recommendations...... 114
CHAPTER 8: Performance Measurement...... 115
The Basics of Performance Measurement ...... 115
Available Data Sources for Performance Measurement...... 119
Incident-Based Reporting...... 119 Survey Data Sets...... 122 Community Ratings Data ...... 125
Selecting AFG Performance Measures...... 128
Performance Logic Model...... 128 Potential AFG Performance Measures ...... 132 Relating AFG Outcome Measures to National, End-Outcome Measures...... 135
Findings and Recommendations...... 137
vii CHAPTER 9: PERFORMANCE-DRIVEN MANAGEMENT...... 139
Overview: A Different Management Approach ...... 139
Prevention vs. Response ...... 141
National Preparedness...... 143
Social Equity and Community Risk...... 146
Regional Cooperation...... 150 Emergency Medical Response ...... 152 Program Strategic Management ...... 153 Leveraging Best Practices ...... 155 Managing the Grant Process...... 156
Conclusion ...... 158
Findings and Recommendations...... 160
CHAPTER 10: IMPACT EVALUATION ...... 163
OMB PART’s Approach to Performance Evaluation...... 165
Developing the Evaluation Framework ...... 165
Evaluating Preparedness...... 166 Does the AFG Program Contribute Significantly to National Outcomes?...... 166 Who Should Determine which Uses of AFG Funds Have the Largest Impact? ...... 168 Do AFG Grants Leverage or Supplant Other Public Funds? ...... 169 Do AFG Grants Improve Community Capabilities?...... 170
Designing and Implementing the Evaluation Framework ...... 172
A Phased Approach ...... 175
Findings and Recommendations...... 177
APPENDICES
APPENDIX A. Uses of Grant Funds in 2005...... A-1
APPENDIX B. Data Collected by the National Fire Incident Reporting System (NFIRS) ...... B-1
viii APPENDIX C. CDC Inventory of National Injury Data Systems...... C-1
APPENDIX D. Data Sources for Performance Measurement...... D-1
APPENDIX E. Insurance Service Office (ISO) Programs and Data...... E-1
APPENDIX F. Individuals Interviewed or Consulted...... F-1
APPENDIX G. Panel and Staff ...... G-1
ix BOXES, TABLES AND FIGURES
Box 3-1. Effectiveness of Smoke Alarm Give-A-Ways in Reducing Fire Injuries ...... 37 Box 3-2. Community Sprinkler Efforts Prevented By Mini-Max Building Codes ...... 39 Box 3-3. Available Fire Education Programs ...... 40 Box 6-1. New Zealand: Prevention Emphasized; Death Rate Declines ...... 102 Box 8-1. Benchmarks and Risk Assessments—NFPA Standards ...... 117 Box 9-1. NIST/CPSE “VISION” Study and Software Project ...... 159
Figure 2-1. Population Covered by AFG Grants per Year of Full Operations by Community Type ...... 16 Figure 2-2. Relative FY ’05 Award Dollars, by Recipient Type ...... 19 Figure 2-3. Distribution by Award Purpose by Department Type ...... 20 Figure 2-4. Total AFG FY05 Awards to All Fire Departments by Use ...... 21 Figure 3-1. Estimate of Fires by Type ...... 25 Figure 3-2. U.S. Residential Fire/Flame Deaths and Rates per 100,000 ...... 26 Figure 3-3. Number of Civilian Fire Deaths in the Home (1977-2005) ...... 26 Figure 3-4. Population Comparisons for Fire Deaths in Select Countries ...... 27 Figure 3-5. Overall U.S. Fire/Burn Nonfatal Injuries and Rates per 100,000 (2001-2005) .... 28 Figure 3-6. Career/Volunteer Firefighters ...... 30 Figure 3-7. Firefighter Deaths per Year—Career/Volunteer 1977-2005 ...... 31 Figure 3-8. Rates of Firefighter Non-Heart Attack Deaths, by Common Cause ...... 32 Figure 3-9. US Average Structure Loss per Structure Fire (1977-2005) ...... 33 Figure 3-10. Skills Level of EMS Personnel, by Workforce Category ...... 46 Figure 3-11. Percentage of Fire Departments Reporting EMS to be “In Scope” ...... 47 Figure 4-1. State and Local Government Expenditures on Fire Protection and AFG Contribution, Compared to U.S. Population Growth...... 60 Figure 4-2. Basic Fire Related Logic Model—Firefighting and Response ...... 63 Figure 4-3. The AFG is one of the programs, or puzzle pieces, the Department must fit into its most appropriate place ...... 75 Figure 6-1. Intersection of the DHS National Preparedness Plan and Fire-Related Aspects of All-Hazards Management ...... 97 Figure 6-2. Capability-Based Planning ...... 99 Figure 7-1. Diagram Showing Annual Strategic Planning Cycle and its Integration with Budgeting...... 111 Figure 8-1. Relationship between Performance Measures, Baselines and Goals ...... 118 Figure 8-2. Fire and Emergency Response/Preparedness Logic Model ...... 130 Figure 9-1. Sample Best Practice Nomination Form ...... 156 Figure 10-1. A Multi-Faceted Approach to Impact Evaluation ...... 164 Figure 10-2. Linking a Design to the Evaluation Question ...... 174
Table 2-1. Program Legislative History ...... 9 Table 2-2. Program Milestones ...... 10 Table 2-3. Program Areas Eligible for Funding (2006) ...... 11 Table 2-4. Annual Milestones in the AFG Program Award Selection Process ...... 15
x Table 2-5. Population Covered by AFG Grants by Year ...... 16 by Community Type ...... 16 Table 2-6. Total AFG FY05 Project Elements Funded by Grants by Department Type (excluding Prevention) ...... 17 Table 2-7. Total AFG FY05 Awards by Department Type, Project Element...... 18 All Races, Both Sexes, All Ages (1981-2003, Place of Accident = Home) ...... 26 Table 3-1. Residential Fire Deaths by Demographic Category ...... 29 Table 3-2. Functioning Residential Smoke Alarms by Demographics ...... 36 Table 3-3. Leading Causes of Fatal Residential Structure Fires with Working Smoke Alarms ...... 37 Table 3-4. Human Factors Contributing to Fatal Injury in Residential Structure Fires with Working Smoke Alarms, 2001-2004 (percent) ...... 38 Table 3-5. Comparison of Fire Fighters, Population and Area of Industrialized Countries .. 41 Table 3-6. Fire Protection as an Average Percent of GDP (2000-2002) ...... 42 Table 3-7. Fire Fighting Organization Cost as an Average Percent of GDP (2000-2002) .... 43 Table 3-8. Ratio of Number of Fire Deaths to the Percent of GDP Spent on Fire Protection 44 Table 4-1. DHS Funding for Homeland Security and AFG...... 61 Table 4-2. The Value of Changes in Fire Departments, NFPA Assessments of 2001 and 2005...... 65 Table 5-1. Major National Studies of the Fire and EMS Problems ...... 80 Table 5-2. Stakeholder Matrix: Viewpoints, Alternative Visions, Measures of Success ...... 84 Table 6-1. Target Capabilities for the DHS National Preparedness Plan Most Relevant to Fire Departments and EMS ...... 100 Table 6-2. Sample of DHS Response Critical Tasks, Measures and Metrics ...... 101 Table 7-1. Department of Homeland Security’s Current (2004) Strategic Plan ...... 113 Table 8-1. Incident Reporting Data Sources ...... 121 Table 8-2. Survey Data Sets ...... 123 Table 8-3. Strategies, Proposed Measures, Logic Model Notes ...... 132 Table 8-4. Other Descriptive Data Elements ...... 135 Table 8-5. Recommended Available Data for National Outcome Measure Baselines ...... 136 Table 9-1. Prevention vs. Response Performance Measures ...... 142 Table 9-1. Prevention vs. Response Performance Measures ...... 142 Table 9-2. Recommended Selection Criteria ...... 145 Table 9-3. Sample of Metrics from DHS Target Capabilities List ...... 146 Table 9-4. Social Equity Performance Measures ...... 150 Table 9-5. Regional Cooperation Performance Measures ...... 152 Table 9-6. Emergency Medical Performance Measures ...... 153 Table 9-7. Recommended Measures of Strategic Management Objectives ...... 154 Table 10-1. Phased Approach to Implementing an Evaluation Framework ...... 176
xi xii ACRONYMS
Academy National Academy of Public Administration AFG Assistance to Firefighters Grants BCEGS Building Code Effectiveness Grading Schedule CAAS Commission on Accreditation of Ambulance Services CBRNE Chemical, Biological, Radiological, Nuclear and Explosive CDC-CIP Centers for Disease Control—Center for Injury Prevention CPSC Consumer Product Safety Commission CPSE Center for Public Safety Excellence CRA Community Risk Assessment DHS Department of Homeland Security DOT Dept. of Transportation/National Highway Transportation DUNS Data Universal Numbering System EMS Emergency Medical Services EMT Emergency Medical Technician FEMA Federal Emergency Management Agency FP&S Fire Prevention and Safety GMS Grant Management System GDP Gross Domestic Product GPRA Government Performance and Results Act HIPAA Health Insurance Portability and Accountability Act HSPD Homeland Security Presidential Directive IAFC International Association of Fire Chiefs ICARIS Injury Control and Risk Survey ICS Incident Control System IOM Institute of Medicine ISO Insurance Service Office NEISS National Electronic Injury Surveillance System NEMSIS National Emergency Medical Services Information System NFIRS National Fire Incident Reporting System NFPA National Fire Protection Association NHTSA National Highway Transportation Safety Administration NIMCAST National Incident Management Capability Assessment Support Tool NIMS National Incident Management System NIST National Institute of Standards and Technology NPG National Preparedness Goal NPP National Preparedness Plan NRP National Response Plan NVSS National Vital Statistics System NREMT National Registry of Emergency Medical Technicians PART Program Assessment Rating Tool PPC Public Protection Classification [ISO] PPE Personal Protective Equipment Safety Administration RCT Randomized Controlled Trial
xiii SAFER Staffing for Adequate Fire and Emergency Response [grant] SCBA Self-Contained Breathing Apparatus SRP Strategic Review Panel TCL Target Capabilities List USDA U.S. Department of Agriculture USFA United States Fire Administration UTL Universal Task List WISQARS Web-based Injury Statistics Query and Reporting System
xiv EXECUTIVE SUMMARY
The Assistance to Firefighters Grant (AFG) program of the Department of Homeland Security (DHS) asked the National Academy of Public Administration (the Academy) for help in assessing its strategic position and how it could plan and manage based on performance. This report is the product of the Academy’s effort to help identify potential new strategic directions for the AFG program and to provide it with advice on how it could most effectively plan, manage, and measure its accomplishments.
The AFG program, authorized five years ago, has been examined in context. The rationale for federal investment in basic firefighting and emergency medical services (EMS)—traditionally functions supported almost entirely by local communities—is not immediately clear. Moreover, even a federal program that has spent well over one-half billion dollars annually and has awarded grants to thousands of local organizations represents such a small fraction of State and local budgets that the impact on outcomes of interest to the public and Congress is likely to be modest. Setting new strategic directions for the AFG program requires a sorting out of responsibilities and priorities. The program’s role and contributions must be understood in relation to the nature of the problems it was created to address, how it can complement the historically important roles played by local communities and organized volunteers, and how it fits into a broader set of national policies.
The AFG program’s administration has been respectful of the history and culture of firefighting and EMS in the U.S. The fire service culture values volunteerism, team work, individual courage, and rapid response. However, fire departments have proven in recent decades that they can take on new roles and functions for their communities. They now face new challenges. Voices within the fire service and outside recognize the need for changes in methods and strategy. The AFG program is in a strong position to foster and support changes that will make fire departments and EMS operations more effective in dealing with a constantly evolving set of challenges.
To date, the AFG program has used most of its resources to enhance basic fire and EMS response capabilities, often in smaller communities that could not afford such improvements otherwise. In many places such investments may have less value in the future than alternative uses of the same resources.
After sifting available evidence and expert views of the program, the problems, and the institutional context, the Academy study Panel recommends that the AFG program’s leadership consider new strategic directions summarized below. If the program moves in these directions, the Panel believes it will increase its impact on public health and safety outcomes central to its legislative purpose:
• National Preparedness. It is impossible to ignore the shift of national priorities toward preparedness for catastrophic events in the wake of the 9/11 attacks. DHS has been directed by Congress to align all of its efforts to an all-hazards perspective on large-scale disaster risks within the U.S., but preparedness is not currently the primary consideration
xv for the program’s application selection process. This change of directions would require prioritizing capabilities specified in the National Preparedness Plan most relevant to local fire departments and EMS and considering AFG applications in regard to their likelihood to reduce high-priority risks.
• Prevention vs. Response. In many cases, at the margin, increased emphasis on proven prevention approaches will have greater benefits for fire safety than further improvements in fire response capability. The program should also define prevention efforts to extend beyond the fire prevention mission to include programs designed to prevent incidents of all types including EMS. The program should encourage and prioritize high-quality applications for the funding of mitigation capabilities, including targeted public education with an emphasis on gathering and publishing reliable national information about the efficacy of prevention approaches in cooperation with other federal agencies, nonprofit organizations, and program stakeholders.
• Social Equity. As the Nation improved its fire safety record in recent decades, some communities and populations lagged behind for various reasons including a lack of fiscal capacity. The AFG program has focused its funding substantially on communities that demonstrate financial need. It can further increase its effectiveness by targeting places and people at greatest risk from fire. By encouraging the use of community risk assessments, and over time, prioritizing awards based in part on whether they address a community’s major risks, the program can maximize its effectiveness in improving fire safety. The program has the ability to change behavior of its applicants. It should promote the use of community risk assessments the same way it successfully influenced the adoption of incident command systems and the reporting of incident data into the National Fire Incident Reporting System (NFIRS).
• Regional Cooperation. Beginning in 2005, the AFG criteria for award have promoted applications that adopt a regional approach. However, the Panel believes that still greater emphasis on region-wide capabilities would increase benefits from AFG spending both to national preparedness and to local public safety. AFG should work with the Department and its partners on the best way to use its funds to build a national network of response capabilities that can be quickly deployed where needed.
• Emergency Medical Response. While the number of fires in the U.S. has been slowly declining of late, many fire departments have taken on new roles, including emergency medical services. In many departments, EMS calls are the majority of calls. Their volume may rise even faster in coming decades due to an aging population. The AFG program should continue to permit applicants to emphasize either fire or EMS capabilities, depending on their local risk assessments and sense of priorities while the federal government and others consider how best to organize responsibilities for this function. It would be useful for DHS, HHS, and DOT/NHTSA to jointly consider how the federal government can best encourage a rationalization of first responses to medical emergencies and coordinate their development of incident data bases and research.
xvi A great strength of the AFG program is its close working relationship with the fire service and other program stakeholders. The challenge for the AFG program will be to support a gradual shift in direction without losing major strengths of its current management approach—including industry driven priority setting and its well-respected peer review process.
The AFG program conducts its strategic planning by assembling a group of acknowledged fire and EMS experts, however, it currently lacks a fully developed ongoing strategic planning process, a set of long-term, ambitious targets and measures against which to measure its progress and full integration with Department-level planning. Therefore, the Panel recommends that the AFG program:
• Establish a continuous strategic planning cycle that is integrated and synchronized with similar processes at the Department level;
• Set a small number of quantified multi-year targets for fire safety and improved EMS performance that are realistic, ambitious, and consistent with both its statutory mandate and mission guidance from the Department;
• Work with its partners in and outside the federal government to build a knowledge-based consensus on long-term national goals for addressing the fire and EMS problems; and
• Use its strategic planning process to identify opportunities to maximize its influence on how others—especially communities—use their resources to address national goals, consistent with community and regional risk assessments.
Current AFG program metrics focus on inputs, financial management, and characteristics of the grant recipients. The program should take a systematic approach to selection of appropriate metrics related to success in accomplishing long-term goals or the achieving intermediate improvements in public safety. While a variety of incident-based and survey based data sources can be drawn on to describe national trends in outcomes related to response, protection, and prevention of residential fires, each source has significant limitations. Outcomes of EMS activities are even more difficult to measure due to organizational hand-offs in the patient’s continuum of care and regulations covering the sharing of patient data. Therefore, the Panel recommends that the AFG program:
• Systematically select measures at various stages of the program’s logic model that pertain to the chosen strategic direction, so that when the measures and their interrelationships are analyzed, progress, and barriers to progress, toward program goals will be apparent, understandable, and potentially correctable;
• Capture additional data elements during the grant application and closeout processes to establish baselines of data related to capabilities and
xvii intermediate outcomes that are not readily available from any other source; and
• Work with its partners to systematically improve reporting completeness and quality for NFIRS and other sources, including closer coordination between the departments responsible for the development of NEMSIS and NFIRS to either integrate or ensure that they complement each other in a meaningful way. Consider requiring EMS grant awardees to use NFIRS and/or NEMSIS to track EMS data and encourage adoption of the standard EMS reporting system in the future.
• Shift in the near term attention for the setting of strategic objectives to the intermediate steps of the logic model until the front-end of the preparedness taxonomy is understood and the program criteria are changed to reflect the recommended strategic directions (the Department also needs to execute on the evaluation framework offered in Chapter 10);
• Revise its grant application, performance reporting and close-out guides so they are consistent one with the other and produce information and data appropriate to the agencies strategic, evaluative, and management needs without unnecessarily burdening the grantees—a focus on the intermediate logic steps should mean that grantees should for the most part only need to report on capabilities, activities, outputs, and outcomes they propose to accomplish (and the statutory and HSPD requirements), and
• Work with its partners to systematically improve the quality of data sources; this includes closer coordination among federal government agencies responsible for the development of NEMSIS and NFIRS to either integrate these approaches or ensure that they complement each other in an interoperable manner.
Performance-driven management allows managers to gauge their progress in achieving each strategic goal. This leads to a different way of approaching basic management tasks to maximize the program’s effectiveness as measured by progress toward its goals. Therefore, the Panel recommends that the AFG program:
• Actively consider strategies to encourage high-quality applications for the funding of prevention and mitigation capabilities for all hazards based on best-practice models and in conjunction with the Prevention and Protection parts of the preparedness taxonomy forthcoming from DHS;
• Use the DHS lists of critical capabilities and metrics to establish priorities for funding and determining overall program performance;
xviii • Consider a linkage from grant application to the community or regional risk assessment and/or a NIMCAST assessment as a way to assign additional positive weighting to applications that address prioritized community risks;
• Define a DHS-wide method of assessing whether lack of community capabilities pose a risk to a wider region; address such risks through the AFG program by raising the capabilities of “weak link” communities or by using logistics to ensure there is a region-wide plan in place to adequately protect residents and critical infrastructure.
• Adopt additional performance measures to promote and measure regional collaboration, thus encouraging relationship-building among the emergency response organizations in neighboring communities, even for those grants not awarded;
• Establish performance measures related to EMS and focus on increasing the number of EMS-related applications where it appears relatively small grants could show local differences in the conditions in which patients arrive at the hospital relative to their starting condition; and
• Seek and promote best practices throughout the operations and management of fire departments, EMS and other emergency responders by identifying candidate practices, assessing them, and recognizing those deemed most valuable and widely applicable.
The AFG grant management system can be modified to support improvements in the grant selection and management processes. Community risk assessments can be used to link the priorities of each community with the priorities of the AFG program. With regard to the grant management process, the Panel recommends that the program:
• Tie performance reports more directly to the application in terms of the major outcome goals, like public injuries and deaths. Applicants should supply both current values in their community for targeted metrics as well as the expected and actual impact on those metrics anticipated from the grant award, thereby demanding that applicants examine the value of the requested use of funds relative to alternative uses;
• Modify the grant application to include elements that demonstrate the applicant has an active and current program of community outreach and education;
• Convert the substantive portion of the performance report should to a readily analyzed format that ties back to the data supplied in the original application;
xix • Better explain to the applicants how to submit a budget for the grant project and how to portray their organization’s fiscal condition in a way that enables panel reviewers to confirm relative financial need; and
• At the time of panel reviews in the selection process, sort applications into groups from the same geographic region and provide information about past awards from that region for reference to help the Panel determine if the grants are serving regional needs in a rational way.
The AFG program currently gathers little data from which to draw conclusions about which uses of its funds have the largest influence on public health and safety or other outcomes. There is no prescribed way for AFG grantees to analyze or report whether they have realized their expected outcomes for the use of the grant funds in their own community. Various intermediate factors will affect program outcomes, but measures of these are not being gathered or analyzed, clouding an understanding of impact that AFG grants have on improving community capabilities. At a national level, the money is too spread out and too small by itself to produce major improvements in fire safety or emergency medical services. This suggests that reporting and evaluation focus on local effects and on identifying which particular funds’ uses are likely to have the greatest effects in particular circumstances. In the immediate future, evaluations should be process-oriented and focused on measuring intermediate outcomes. Periodic rigorous independent evaluations should be undertaken to validate causal relationships forming the program’s logic model to the extent possible and to assess the program’s contributions.
For testing specific new interventions, such as new firefighting equipment, new EMS or firefighting training programs, comparison group designs and randomization with matched pairs are likely to be feasible, enabling the AFG to identify improved practices that it can then help to disseminate more quickly. Therefore, the Panel recommends that the AFG program:
• Be represented, along with the U.S. Fire Service in general, as DHS continues to refine and update the elements of the National Preparedness Plan most relevant to the AFG program to ensure that the plan covers the broader needs of homeland security;
• Develop a model that establishes a baseline for measuring and predicting improvement in outcome measures attributable to the awarding of each category of grant and a target for continuous improvement, once results data are requested from grantees;
• Modify the performance reporting requirements for grantees to be one year after grant money is fully distributed rather than the current timeframe: one year after the award;
• Add an analysis of the actual outcomes reported in the previous years’ performance reports as a significant input for the annual prioritization and criteria setting session;
xx • Request applicants to describe whether the proposed prevention or mitigation program is based on a model recommended by a nationally recognized organization;
• Revise the closeout questionnaire to focus on the outcomes that the grant was originally intended to affect to enable analysis of grantee success as well as cost-benefit comparisons across categories of applicants;
• Capture additional data elements during the grant application process to establish baselines of data related to capabilities and intermediate outcomes that are necessary to establish correlations, and eventually, measure the impact of the AFG program; and
• Analyze communities awarded a large piece of equipment or vehicle in the earlier years of the program to see if they are in fact allocating money in their budget to replace the item at the end of its amortization schedule.
The Panel shares the prevailing view of those most familiar with the AFG program that its success to date can be attributed in large part to the direct engagement with representatives of the fire service in setting program priorities and in judging the merits of thousands of applications annually. In the Panel’s view, regardless of the extent to which it directs funds to national priorities, the program can retain its character as a locally driven, competition reflecting the best thinking of community-based fire services and other emergency responders.
These recommendations should be viewed as the beginning rather than the end point of strategic planning. The report describes how the program can use a continuous planning, self-assessment, and evaluation process to constantly improve and multiply its effectiveness.
xxi xxii CHAPTER 1 INTRODUCTION
THE ACADEMY STUDY
The Department of Homeland Security’s (DHS) Assistance to Firefighters Grants program (AFG) asked the National Academy of Public Administration (Academy) to conduct an independent study that would provide the program with a performance assessment framework.
Program managers told the Academy Panel and study team they wanted the Academy’s assessment to focus on the program’s strategic position and how it could plan and manage based on performance. Specifically, the Academy’s work is intended to help program managers and stakeholders improve the program’s future use of resources by:
• defining strategic directions—consistent with the program’s legislative mandate and the strategic goals of DHS—that would permit the AFG program to use its grants and administrative resources most effectively to improve fire safety and emergency medical services in the United States
• identifying key performance measures and showing how these may be used to help drive program improvement
• identifying the appropriate scope and general design of a possible future program evaluation
• assessing the performance of the AFG program to date using data available or readily attainable within the funding constraints of the project
OVERVIEW OF THE FIRE GRANTS PROGRAM
Recognizing the increasing responsibilities of fire departments and seeing a need for additional support of their efforts to improve fire safety and provide emergency medical services (EMS), Congress established the AFG program in 2000 through an amendment to the Fire Prevention and Control Act of 1974, in the FY 2001 National Defense Authorization Act (P.L. 106-398). This Act established a new office in the Federal Emergency Management Agency (FEMA) to administer a program of competitive grants directly to fire departments. The Act authorized assistance for hiring and training firefighting personnel, acquisition of firefighting vehicles and other equipment, prevention, public education, improved EMS capabilities, and a variety of other uses.
The same Act required the U.S. Fire Administration (USFA), in conjunction with the National Fire Protection Association (NFPA), to conduct a study to determine the needs of local fire departments. This study, published in January 2003, found that many fire departments— particularly volunteer companies in rural communities—report shortfalls in facilities, equipment
1 and training of personnel. These findings informed the focus of the AFG program for fire service and emergency response activities for state and local communities.
Among the AFG’s future challenges is identifying a clear strategic focus for targeting its grants. Since it was established in FY 2001, by design, the AFG program has been driven in its priority- setting by input from organizations representing the U.S. fire service and emergency response. It has directed many of its grants to smaller, volunteer, and rural departments, enabling them in many ways they otherwise could not afford.
As the federal government confronts increasing fiscal pressures and reviews its spending priorities, all federal programs must demonstrate that they meet clear national purposes. The AFG program is no exception. It must demonstrate that it is using its funds effectively not only to enhance fire safety and emergency medical capabilities locally and nationally but to enhance national preparedness for major catastrophic events.
STUDY METHODOLOGY
The Academy used a variety of methods to gain insight into the strategic challenges facing the AFG program and techniques that the program could use to assess its performance and improve its effectiveness. These included:
• Structured interviews with major program stakeholders, such as representatives of associations of firefighters and fire chiefs, providers of equipment and training to fire services and EMS, and local and state officials (such as city managers) with broader responsibilities for public safety and disaster preparedness;
• Interviews with experts in fire safety, EMS, and related disciplines, including specialties such as community risk assessment and fire prevention;
• Structured interviews followed by a one-day intensive facilitated conversation at the Academy with a Strategic Review Panel named by the AFG program about future directions for the AFG program;
• A literature review, including major national studies of the fire problem and emergency medical care, as well as selected international and specialized research reports on these issues;
• A review of congressionally-mandated reports on the needs and capabilities of local fire services;
• An analysis of data from the program about distribution and uses of grant funds and about grantee performance;
• An analysis of Census and other statistics regarding trends in fire safety, emergency medical care, and spending on fire and EMS in the U.S. and other industrial countries;
2
• Interviews with state emergency management officials;
• Surveys of documented approaches to performance measurement, accreditation, and rating of fire services, including incident reporting systems;
• Interviews with officials and staff of DHS who deal with the AFG program from the broader perspective of national preparedness, as well as those who address the fire problem from other perspectives, such as staff of the U.S. Fire Service;
• Field visits to the National Fire Academy in Emmitsburg, Maryland, and the National Fire Protection Association in Quincy, Massachusetts; and
• A review of federal government reports that include goals for or related to the AFG program and assess the program’s performance and management.
Analysis synthesized this information around two main issues: (1) how to determine the most productive near-term strategic directions for the AFG program; and (2) how to identify and use performance information to guide, manage, and evaluate the program.
A Panel consisting of Academy Fellows and others—including experts in the fire service, local government management issues, and performance-driven management and evaluation—guided the study and have reviewed and approved the methodology, findings, and recommendations.
ORGANIZATION OF THIS REPORT
The remainder of this report is organized as follows:
Chapter 2 summarizes the AFG program’s legislative and funding history, how it operates, and how it has been used to date. Important features of the program’s operations include regular stakeholder input on priorities and the strong relationships with the fire service and its representatives.
Chapter 3 analyzes the primary problems the program is intended to address: fire safety, improved emergency medical services, and preparedness to deal with a range of large-scale disasters and hazards. It highlights the recent trends in deaths and injuries resulting from fires. Also, international comparisons regarding both fire deaths and public spending for fire services provide a context for U.S. efforts to address fire safety.
Chapter 4 places the AFG program in its broader organizational and financial context. It notes the U.S.’s continuing heavy reliance on volunteer and community support for its fire and emergency medical services. Local and state governments are the primary public funders of fire and EMS services, although the latter receive substantial revenues from the federal Medicaid and Medicare programs for certain services. The AFG program is a new source of support for local fire and EMS, with a distinct mission that reinforces the expanded scope of local fire
3 departments, particularly those that have limited local resources. It also has the potential to substantially change the traditional focus and culture of the fire service. This chapter also points out how the federal government has yet to fully define and coordinate its own contributions to the rapidly evolving organization of EMS.
Chapter 5 draws together a wide range of views expressed by AFG program stakeholders, experts on the fire and EMS problems, historical studies of these problems, and a review panel assembled by the program to comment on possible strategic directions for the AFG program. The range of perspectives is wide and therefore presents the program with a range of possible directions as well as a challenge to integrate varied views and interests.
Chapter 6 includes the Academy Panel’s recommendations for new strategic directions. It draws on preceding analysis and expert views of the fire and EMS problems to identify promising directions consistent with the program’s statutory mandate and mission. The proposed changes of emphasis would increase the program’s effectiveness. They would reinforce and strengthen the program’s contribution to the broader national preparedness mission of DHS. The manner in which the program introduces new priorities or emphases will determine whether the changes are effective, because the program’s success depends on its strong working relationships with local fire departments, stakeholders, and other partners who share common goals. Community risk assessment is seen as an important tool, integrating local assessments of risk and capability needs with national preparedness goals and funding priorities.
The chapters that follow describe in some detail how the program can focus its efforts through the setting of long-term strategic goals consistent with the recommended directions, the selection of performance measures, and the use of data and information on performance. These chapters include specific recommendations from the Academy Panel as noted below.
Chapter 7 outlines a possible strategic planning cycle that would enable the program to tap the expertise of the fire service and EMS communities, set multi-year strategic objectives against which to assess its progress, integrate its planning and direction with department-level planning and budgeting, and provide a focus for joint planning with other federal and non-federal partners who share common goals. This process would allow the program to continually learn from experience and, over time and in consultation with its partners, increase its already substantial contribution to improved public safety and emergency medical response.
Chapter 8 summarizes the Academy Panel’s broad review of possible performance measures, discusses its recommended approach to selection of key performance indicators, and identifies issues regarding their practical use and proper interpretation.
Chapter 9 describes how the AFG can be managed in a way that drives toward improved performance by looking at processes and intermediate outcomes—the earlier stages of the logic model. It illustrates how recommended performance metrics can be used to better target funds and reward communities that use funds most effectively. Recommendations are made for changes to the grant application, selection and monitoring processes intended to sharpen the focus on key aspects of performance. The Panel recommends that performance relate to national preparedness and the recipient communities’ own assessments of their risks and capabilities.
4
Chapter 10 outlines a framework for periodically evaluating the AFG program’s contributions to public safety and to the broader goals of public protection against varied hazards and medical emergencies. It describes and recommends a practical approach to evaluating the program’s impact—the final stage of the logic model—recognizing that in many instances an AFG grant is a modest contribution to a larger marshalling of resources and volunteer efforts. It is often part of a community strategy to better protect both the public and first responders against the dangers of fire, other hazards, and medical emergencies.
5
6 CHAPTER 2 THE AFG PROGRAM AND ITS USES
This chapter is presented in two parts. The first part provides an overview of the AFG program. It describes its history, the award selection process, and its strategic direction-setting relationship with its partner organizations. The second part examines uses of AFG funds in recent years, including a more detailed look at FY2005 awards by applicant categories and how recipients used their grants.
OVERVIEW OF THE AFG PROGRAM
The AFG program provides project grants directly to fire departments and non-affiliated Emergency Medical Services (EMS) organizations to enhance their abilities with respect to fire and other hazards covering vehicle needs, and operational needs, including training, equipment, and a long list of applicable purposes.
According to the 2006 Program Guide, this program “seeks to support organizations that lack the tools and resources necessary to more effectively protect the health and safety of the public and their emergency response personnel with respect to fire and all other hazards.” The statutes require the agency to consider financial need of the applicant. The statutes as amended also require the agency to make separate awards for fire prevention purposes using no less than five percent of the amounts appropriated (approximately 5.3 percent in 2005). The Panel’s study and recommendations do not focus attention on this specific set-aside for fire prevention and safety grants, contracts and cooperative agreements, but do address the question of what priority should be placed on fire prevention.
The study focuses on the main AFG competitive grant program, addressing purposes that the program refers to as Operations and Support, including vehicle purchase. The AFG program has determined that these grants should be awarded with a 12-month period of performance. Grantees provide matching funds for these grants that, by statute, must equal five percent of total costs for jurisdictions covering less than 20,000 people, 10 percent for those covering 20,000- 50,000 people, and 20 percent for those covering populations over 50,000.
Eligible recipients of AFG grants include fire departments and nonaffiliated EMS organizations, defined as follows, operating in any of the fifty states, the District of Columbia, or the five U.S. territories:
• A “fire department” is defined as an agency or organization that has a formally recognized arrangement with a state, territory, local, or tribal authority (city, county, parish, fire district, township, town, or other governing body) to provide fire suppression to a population within a fixed geographical area on a first-due basis.
• A “nonaffiliated EMS organization” is defined as a public or private nonprofit emergency medical services organization that provides direct emergency medical services, including
7 medical transport, to a specific geographic area on a first-due basis, but is not affiliated with a hospital and does not serve a geographic area where emergency medical services are adequately provided by a fire department. Rescue squads that provide direct emergency medical services, including medical transport, to the community they serve are included in this category of eligibility.1
The agency has determined that each eligible applicant is limited to one application per program year for the AFG program.
History
The AFG program was authorized in December, 2000. Initially, the program was administered by the U.S. Fire Administration, part of FEMA. In 2003, FEMA became part of the new Department of Homeland Security (DHS). In FY 2004, Congress appropriated the funds for AFG to the Office for Domestic Preparedness, and DHS consequently moved the program from FEMA. In subsequent reorganizations, the Office for Domestic Preparedness became the State and Local Government Coordination and Preparedness Directorate, and then the Office of Grants & Training within the Preparedness Directorate. Since 2004, report language accompanying appropriations has required that AFG administration be conducted in cooperation with the U.S. Fire Administration.
The Act initially authorized funding of $100 million in FY 2001 and $300 million for FY 2002. Possibly in support of the firefighters’ critical role as first responders during the September 11, 2001, terrorist attacks, appropriators have since provided increased funding for the AFG program. In 2004, Congress adopted reauthorization legislation extending AFG’s funding authorization through 2009 and expanding the array of authorized uses.
The total of all of the applicants’ requests for funds in the program’s first year, 2001, exceeded $3 billion. The program currently receives requests each year that exceed $2 billion. Congress increased the level of funding for the program after 9/11 and has maintained funding levels and expanded on the uses of the appropriated funds. Congress enacted the Staffing for Adequate Fire and Emergency Response (SAFER) grant program in 2005. Administered by the same staff, this is another source of funds for operations and support available to fire departments, but restricted to the hiring of new firefighters. Note that SAFER is not a focus of the Panel’s study or recommendations. AFG program history includes the legislative milestones shown in Table 2-1 and program milestones shown in Table 2-2.
1 AFG 2006 Program Guidance.
8 Table 2-1. Program Legislative History
Legislation Date Title/Description Reference Federal Fire Prevention and Control Act: Establishes the National Fire Prevention and Control Administration; a National Academy for Fire Prevention 10/29/1974 P.L. 93-498 and Control. Provides an NIH-expanded program of research on burns, treatment of burn injuries, and rehabilitation of fire victims.
Superfund Amendments and Reauthorization Act (SARA) of 1986: SARA reauthorized and expanded the hazardous substance Superfund to address hazardous waste sites and spills. It provides for (1) emergency response to hazardous waste spills; (2) emergency action at hazardous waste sites that pose an imminent hazard to public health or environmentally sensitive ecosystems; (3) long-term planning, remedial design, and construction to clean up hazardous waste sites where no financially viable responsible party can be found; (4) 1986 P.L. 99-499 enforcement actions to require responsible private and Federal parties to clean up hazardous waste sites; and (5) enforcement actions to recover costs where the fund has been used for cleanup. Due to the site-specific nature of the Agency’s Superfund program, site-specific travel is not considered part of the overall travel ceiling set for the Superfund account. Title III mandates training for hazardous material response, and supervisory and command training in incident management systems. Most fire departments accepted this challenge.
FY 2001 National Defense Authorization Act (Title XVII): Amends the Federal Fire Prevention and Control Act to create a new office in FEMA to administer 10/30/2000 P.L 106-398 grants to fire departments and fire prevention organizations, including hiring and training, prevention programs, equipment and facilities and public education.
National Defense Authorization Act of 2002: Increased funding for the AFG 12/28/2001 P.L. 107-107 through FY 2004. DHS Appropriations Act: Relocated the fire grants to Office for Domestic 10/1/2003 P.L. 108-90 Preparedness and provides for the continued operation of the Grant Program with the inclusion of the U.S. Fire Administration. Assistance to Firefighters Grant Program Reauthorization Act of 2004: 10/28/2004 P.L. 108-375 Reauthorized the grant program through 2010 and designated the USFA Administrator as the administering authority of the fire grant program. Homeland Security FY 2006 Appropriations: Specifies that U.S. Fire 10/18/2005 P.L. 109-90 Administration will continue to participate in the grant administration activities, Administration highlights all-hazards to terrorism approach. Homeland Security FY 2007 Appropriations: Makes appropriations for the Department of Homeland Security for FY 2007 including the Office of Grants and Training, which awards grants to state and local governments for terrorism 10/4/2006 P.L. 109-295 prevention activities, firefighter assistance grants, emergency management performance grants, the U.S. Fire Administration, training, and infrastructure protection and information security.
9
Table 2-2. Program Milestones
Year Event 2000 The spending program is authorized by Congress. 2001 Initial grant awards are made. Fire-department-based EMS are made eligible for support as approved by Congress. 2002 The electronic grants management system starts, making online applications possible. Applications can cover only one narrowly-defined purpose or area. 2003 EMS is singled out in its own funding area for applications. EMS is combined back into overall Operations category to foster more applications. 2004 CBRNE (chemical, biological, radiological, nuclear, or explosive hazards) is highlighted, requiring applicants to relate to state plans. 2005 Non-affiliated EMS organizations become eligible per Congressional action. Regional projects by “host applicants” are given higher priority. 2006 CBRNE is prioritized and agency will seek a technical review by State.
Eligible Uses of Funds
There is considerable flexibility for the applicant to develop and seek funds for a project that comprises multiple activities. However, activities must be completed within one year of award under the current rules. Applicants are urged to ensure all projects applied for are consistent with national standards and address interoperability. Eligible program areas are described in Table 2-3 below.
10 Table 2-3. Program Areas Eligible for Funding (2006)
Applicant Type Program Area Eligible Activities Other Notes Training, equipment, May request EMS personal protective training and equipment Operations and equipment, wellness and here if the EMS unit Firefighter Safety fitness, and modifications to falls organizationally Fire Departments fire stations and facilities under the fire department (1 application per program area) Pumpers, brush trucks, May request one tankers/tenders, rescue vehicle per year and Vehicle Acquisition vehicles, ambulances, previous vehicle quints, aerials, foam units, grantees may not and fireboats reapply for a vehicle EMS training, EMS Requests for equipment including equipment, PPE, and EMS Operations and personal protective training for incidents Nonaffiliated EMS Safety equipment, wellness and involving CBRNE are Organizations fitness, and modifications to available
stations and facilities (1 application per program area) Ambulances, transport units Small specialty access EMS Vehicle and rescue squads vehicles are considered Acquisition “equipment”
Regional “Hosts” Training, equipment Hosts cannot apply for (Fire or EMS acquisition and personal other individual needs organizations) protective equipment in addition to the acquisition, other operations regional project (An application here Regional Projects and support purposes would count as the 1 application allowed in another category as well)
As shown in Table 2-3, a regional project is one “hosted” by the applicant in which multiple organizations serving more than one local jurisdiction benefit directly from the activities.2 Hosts cannot apply for other individual needs in addition to the regional project in the same year.
2 This term would not encompass county-wide fire departments, even if they serve multiple municipalities; these can apply on their own for a county-wide project. Any eligible applicant may act as a “host applicant” and apply for large-scale or regional projects on behalf of itself and any number of organizations in neighboring jurisdictions; but the host is responsible for fulfilling all grant requirements (e.g., reporting incident data to a national incident data base, control of and accounting for the funds, and distribution and control of the property).
11
Additional statutory requirements and limits affect award selection, as described below:
• Maintenance of Expenditure Levels—By statute, the agency may not make awards unless it finds that recipients would “maintain in the fiscal year for which the assistance will be received the applicant’s aggregate expenditures for [operations and safety or prevention activities] at or above the average level of such expenditures in the two fiscal years preceding the fiscal year for which the assistance will be received.”
• Matching Requirements—All recipients are required to match federal funds with non- federal funds. In-kind (non-cash) contributions are not eligible. The non-federal share varies with the population of the jurisdiction served: 5 percent share for organizations serving areas less than 20,000 people, 10 percent for those who serve between 20,000 and 50,000 people, and 20 percent for those who serve populations of more than 50,000 people. Note that the re-authorization of the program in 2004 lowered the matching share from a high of 30 percent for the larger jurisdictions.
• Fiscal Year Caps Based on Size of Area Served—If the organization serves fewer than 500,000 people in its jurisdiction, it can receive a federal award of up $1 million; $1.75 million is the limit for organizations serving from 500,000 to one million people, and $2.75 million for an organization serving more than one million people. Note that reauthorization of the program in 2004 raised these award caps.
Program Priorities and Award Criteria Development
Each year, AFG program office managers meet to develop the program’s priorities and its selection criteria for the coming year. As directed by its authorizing legislation, the program assembles a review panel of fire service professionals representing nine major fire service organizations. The goal of the panel is to make recommendations to the program’s management regarding the criteria used for awarding grants and to propose any necessary changes to the administration of the grant program.3
The organizations represented on the review panel include:
• The International Association of Fire Chiefs
• The International Association of Firefighters
• The National Volunteer Fire Council
• The National Fire Protection Association
• The National Association of State Fire Marshals
3 AFG 2006 Program Guidance.
12 • The International Association of Arson Investigators
• International Society of Fire Service Instructors
• The North American Fire Training Directors
• The Congressional Fire Service Institute
The nine associations above are predominantly concerned with fire problems. However, the FY 2007 appropriations act gives guidance to the program to include EMS-focused associations in future review panel meetings.
A major priority change from 2005 to 2006 resulting from the review panel’s annual meeting relates to the need to recognize the importance of projects that are regional in character. According to the Program Guide, other changes for 2006 include the following:
• Applications that relate to CBRNE (chemical, biological, radiological, nuclear, or explosive) hazards are encouraged more strongly.
• The agency will conduct a specific review with the Homeland Security Office of the applicant’s state whenever applications involve CBRNE equipment or training to obtain more consistency with the state’s strategies and avoid duplication of effort.
• Applications involving equipment or training purposes will be scored relative to their ability to address critical infrastructure elements defined by the potential for “catastrophic loss of life or catastrophic economic loss.”
• Applicants “must certify they will comply with all edicts involving” the National Incident Management System per Homeland Security Presidential Directive-5.
• Training proposals addressing “basic capabilities” (not officer-focused or advanced ones) will receive the highest consideration. The highest scores would involve instructor-led, hands-on training that leads directly to a nationally-sanctioned or state certification.
• Basic firefighting equipment will be prioritized with preference given to first-time purchases over replacement and to upgrading or inventory increases for serving existing areas, not new ones.
• Sub-areas: Modifications to Facilities—Proposals should focus on fire and life safety in existing fire department facilities such as the installation of sprinklers, vehicle exhaust extraction systems, and emergency generators. The Agency will favor firefighting facilities over training facilities and those with high occupancy and broad usage.
These modifications to the program’s priorities are then used: (1) to adapt the computer-based logic or software used to filter, categorize and rank the applications initially; and (2) to provide guidance to the peer review panels that evaluate the more highly ranked applications in each
13 category. The number and nature of the changes illustrate the program’s continuous efforts to improve use of its funding and adapt its priorities to changing needs, including those identified by its partners.
Grant Award Process
The AFG program has established an electronic grants management system (GMS) supporting the processes of applying via the Internet, filtering viable applications, the panel-review process, and recipient closeout activities. One of the most frequent comments by those familiar with the AFG program’s operation is expressed appreciation for its user-friendly electronic interface. The system helps the agency filter a large volume of applications (e.g., an applicant filing more than once invalidates both applications; incomplete applications are rejected) and score the valid applications based on program-year priorities and the scoring criteria adopted by the agency.
The Grant Selection Process
The agency selection process has three basic steps: (1) initial filtering of applications into categories and scoring by the grants management system to identify those within the “competitive range”; (2) independent panel reviews of the competitive range applications to complete the scoring; and (3) final agency decision-making considering the panels’ rankings and the requirements to assure an appropriate distribution of the grant awards by type of applicant and community and to comply with other statutory limits.
Once scored by a batch process that applies the program’s automated scoring and prioritization criteria, those applications calculated to be within the “competitive range” (amounting to 200 percent of the available funds in any category, like “vehicles”) are referred to independent review panels to finalize the ranking.
Review panel members who review a “competitive range” application evaluate based on four equally weighted assessments of the applicant’s submission: (1) clarity of proposal, including the budget; (2) financial need of the organization; (3) benefits that would result; and (4) extent to which the grant would enhance daily operations and/or how it will positively impact the organization’s ability to protect life and property.
To facilitate the review panel assessments, applicants are encouraged within their initial submissions to discuss the following matters in order to improve the panel’s ranking of their proposal: (1) how the project fits with regional approaches; (2) how the project contributes to the implementation of equipment/technology interoperability with other first-responders; and (3) how a project relates to an all-hazards response perspective.
The review-panel-ranked applications are then subject to an agency decision-making process where the rules of distribution called for by the statute are applied (e.g., at least 3.5 percent of funds should be awarded for EMS purposes). The GMS also assists the agency in applying the rules related to the distribution of the awards across the types of applicants and across served communities. (The major milestones of the grant selection process are summarized in Table 2-4 with the target duration for each.)
14 Table 2-4. Annual Milestones in the AFG Program Award Selection Process
Target Duration Step Description (in general) Grants are solicited in advance of the application 30 days Advertise period Application Application processing by the Agency’s GMS 30 days Period vendor Ranking the applications numerically for panel At least 6 weeks; review based on money available, running down the typically it ends Scoring dollar amount until it adds up to 200% of the around December available funds, then divides out the 25% maximum that can go to vehicles and any other pre-allocations. Peer review, technical review, panel review, and 90 days until the first state review (state review for those with award is announced Reviews regional/interoperability requests with the states). after the panel completes its first reviews The award selection may continue for a year or One year or more Additional more depending on appropriations being freed up Awards as (e.g., a city council not being willing to authorize Appropriate matching funds after an award is made). Managing the The applicants receive their award and use the One year beginning Award funds. the award date. An interim financial and performance report is Due every six Six-Month required. months after the Report award date Grantee is required to report on the actual funds Due one year after Closeout spent and provide a narrative description of the use the award date of funds. Source: Interview with AFG program management and GMS support contractor.
AFG FUNDING OF AND AWARDS TO FIRE AND EMS
The AFG program tracks the population that is intended to be served by the grants it awards. Table 2-5 and Figure 2-1 below show population covered by grant awards for the years 2002- 2005. While the program does not track the total portion of the population that has been covered by at least one AFG award, it is clear that the overall population benefiting from AFG awards is still growing. In 2005, for example, 2,612 applications out of about 19,000 were submitted by organizations applying for the first time.4
4 E-mail communication from Heber Willis, January 8, 2007.
15
Table 2-5. Population Covered by AFG Grants by Year
Year 2002 2003 2004 2005 Quantity of AFG Awards (Fire/EMS Operations and 5099 8,471 7,880 5960 Vehicles) Annual Value of Awards $323.4M $693.2M $674.9M $605M Average Value of Award $63,400 $81,800 $85,600 $101,500 Population Covered by Awards* Rural 21,447,113 48,797,708 47,951,624 31,548,443 Urban 44,263,965 38,634,334 48,532,592 51,435,643 Suburban 34,726,085 52,055,910 49,892,020 48,718,785 Total Population Covered 100,437,163 139,487,952 146,376,236 131,702,871
Figure 2-1. Population Covered by AFG Grants per Year of Full Operations by Community Type
160,000,000 140,000,000 120,000,000 100,000,000 Suburban 80,000,000 Urban 60,000,000 Rural 40,000,000 20,000,000 0 2003 2004 2005
The apparent decrease in the number of awards and total award dollars in 2005 is attributable to both a smaller overall pool of available funds and a smaller number of high-value awards, as indicated by the increase in the average size of award from $ 85,600 to $101,500. The ratio of the population served to the dollars awarded has been nearly constant for the three years of full operations, at 0.20 for 2003 and 0.22 for both 2004 and 2005.
16 Analysis of AFG AWARDS for FY 2005
From the outset, AFG has been the largest federal financial assistance program for the fire service community. In its first year, the program received over 19,000 grant applications, requesting nearly $3 billion. That year the program awarded 1,855 grants totaling nearly $97 million. In FY 2005, AFG received 20,972 applications requesting a total of $2.9 billion, of which 5,725 were funded, totaling $586 million. Thus, assistance has been distributed widely, reaching a large proportion of the nation’s local fire departments. Most funding has been used for major capital purchases and firefighter equipment, but the range of uses is wide. (See Appendix A for a detailed break-out of the awards made in each category.)
The highest quantity of FY 2005 AFG awards were for equipment and personal protective equipment (PPE) purchases, and the lowest quantity went to EMS equipment and EMS training, as shown in Table 2-4 below. Equipment and PPE were favored (whether among applicants or the program decision makers) for all types of departments. Awards for purposes unrelated to incident-response are a small relative amount (the agency does not disaggregate by this perspective so the exact figures can not be reported). Training, a pre-incidence activity, received just over 5 percent of the funds.
Table 2-6 shows that the majority of awards, by relative quantity, are awarded to all-volunteer fire departments. In fact, awards to all-volunteer departments outweigh the number of awards to all other types of departments combined. Departments that use paid firefighters (all paid/career plus the paid on-call/stipend) received the fewest grants.
Table 2-6. Total AFG FY05 Project Elements Funded by Grants by Department Type (excluding Prevention)
All- Paid On- All Paid/ ITEM Voluntee Combination Call/ Total Career r Stipend Vehicles* 720 104 42 45 911 Equipment 7,007 2,514 1,251 517 11,289 PPE 6393 2,017 699 684 9,793 Training 351 165 85 41 642 Wellness & Fitness 250 370 304 38 962 EMS Equipment 215 109 33 25 382 EMS Training 32 26 12 8 78 Total 14,968 5,305 2,426 1,358 24,057 * Including $23M for ambulances.
While funded vehicles are relatively few in number, the dollar amount for vehicles is among the highest for any category due to the greater average value of these awards (average of $174,000 for vehicles, compared with an average of $88,000 for awards excluding vehicles).
17
Table 2-7 and Figure 2-2 below show the dollars awarded to each type of department.
Table 2-7. Total AFG FY05 Awards by Department Type, Project Element (excluding Prevention)
Paid On- All Paid/ ITEM All-Volunteer Combination Call/ Total Career Stipend Vehicles $116,698,947 $21,951,610 $13,147,275 $7,488,718 $159,286,550 (including $23M for ambulances) Equipment $39,504,606 $33,989,325 $45,751,269 $5,822,253 $125,067,453 PPE $88,081,362 $58,313,673 $31,003,487 $14,253,021 $191,651,543 Training $6,970,073 $5,118,619 $17,239,742 $1,010,035 $30,338,469 Wellness & Fitness $4,671,470 $13,968,520 $30,027,744 $1,058,266 $49,726,000 EMS Equipment $8,262,308 $6,130,928 $1,913,351 $556,944 $16,863,531 EMS Training $192,327 $485,977 $317,544 $207,460 $1,203,308 Total $264,381,093 $139,958,652 $139,400,412 $30,396,697 $574,136,854
More than one-half of all awards (by dollar value) went to all-volunteer departments. Combination departments also have a high percentage of volunteer workforce. As shown in Figure 2-2, paid/career and paid on-call departments, combined, won less than one-quarter of the grant money in 2005.
18 Figure 2-2. Relative FY ’05 Award Dollars, by Recipient Type
Paid on Call/Stipend 5% All Paid/Career 19%
Combination 24%
All volunteer 52%
In addition to the grants available to fire departments in FY2005 through the competitive grant program, DHS set aside no less than $32.5 million of AFG funds for national, state, local or community organizations or agencies (including fire departments) for fire prevention and injury prevention programs. Those awards are not included in the tables in this section.
Each department type had a different distribution of uses for grant funds. As shown in Figure 2- 3, all departments were awarded many more grants for PPE than equipment, except for all- paid/career departments that had more equipment awards than PPE. While EMS training awards consistently accounted for 0-1 percent of each type of department’s awards, EMS equipment accounted for 4 percent of combination departments, but only 1 percent of all-paid/career departments. This could indicate that when career departments need more equipment, they are more likely to buy equipment for EMS purposes. Also, as a percentage of funds, nearly five times as much award money was used for vehicles by all-volunteer departments (44 percent) compared with all-paid/career departments (9 percent). The most disparate numbers are related to the grant funds for wellness and fitness, which comprise 2 percent of the funds for all- volunteer departments and 22 percent for all-paid/career departments.
Figure 2-3 gives a side-by-side comparison of the distribution of grant funding for each category of department.
19 Figure 2-3. Distribution by Award Purpose by Department Type
Combination Fire Department All-Volunteer Fire Department (23%—2005 awards) (60%—2005 awards)
EMS Equipment EMS EMS Training 3% Wellness and Equipment 0% Wellness and Fitness 4% Vehicles Fitness EMS Training 10% 16% 2% 0% Training Training 3% 4% Vehicles 44% Equipment PPE 24% 33%
PPE Equipment 42% 15%
All Paid/Career Fire Department Paid On-Call/Stipend Fire Department (11%—2005 awards) (6%—2005 awards)
EMS Training EMS Equipment 0% Vehicles EMS Equipment EMS Training 1% 2% 9% Wellness and 1% Wellness and Fitness Fitness 3% 22% Vehicles 25% Training Equipment 3% 34% Training 12% PPE Equipment 47% 19% PPE 22%
20 The majority of award money in FY 2005 was used for fire-related purposes instead of EMS. As shown in Figure 2-4, after including about $23 million of vehicle awards used for ambulances, 7 percent of award money went to EMS-related uses (EMS equipment, EMS training and ambulances) based on the agency’s calculations.
Figure 2-4. Total AFG FY05 Awards to All Fire Departments by Use
EMS- Related 7%
Fire-Related 93%
(See Appendix A for a detailed break-out of the awards made in each category excluding prevention by quantity of award.)
21 22 CHAPTER 3 THE FIRE, EMS AND NATIONAL PREPAREDNESS PROBLEMS
The AFG program funds organizations whose mission it is to prepare for and respond to one or more of the major sets of hazards and emergencies examined in this chapter:
• The “fire problem” generally refers to all concerns related to fire incidents.
• The “EMS problem” refers to all concerns associated with the demand for emergency medical services.
• “National Preparedness” refers mainly to the goal established under the Homeland Security Presidential Directive-8 which recognizes that the nation needs to be much better prepared for large-scale incidents from all hazards and risks.
Of the three, the fire problem is best understood, and public policy and administration to address it are relatively well settled. EMS as a distinctive category of public services has grown rapidly and is organized very differently in different communities; public policy and administrative approaches are not as well settled in the U.S. Although national preparedness has been a focus of policy dating at least from the Cold War, its focus has changed following the attacks of 9/11 and again in response to the weaknesses exposed by Hurricane Katrina in 2005. The organizations that receive AFG grants typically are at the center of planning and response to all three sets of hazards and emergencies in their communities.
This chapter provides background information concerning the three problem sets and relates this information to the AFG program. This analysis sets the stage for the discussion that follows about the AFG program’s operating context and possible strategic directions for the program. The analysis in this chapter also provides a possible basis for selecting appropriate goals and performance measures for the AFG program.
The first section of the chapter examines the fire problem by looking at the trends related to each goal of the AFG program as stated in its authorizing statute, including trends in death and injury of civilians, firefighter deaths, and fire-related losses. This section looks at key statistics related to prevention efforts to spotlight evidence about whether such efforts can help eliminate fire incidents or mitigate their consequences. Finally, it analyzes U.S. spending on fire protection and compares U.S. spending to that in other developed countries to assess whether the U.S. is getting the level of service it should expect compared with the resources it invests in fire protection.
The second section describes the EMS problem by profiling EMS in the U.S. and describing the role of EMS in disaster preparedness. It then explores the special considerations for EMS related to rural communities and the effect that the aging U.S. population is expected to have on EMS in the future. It concludes with a discussion of the rate of on-the-job injuries and deaths of EMS personnel. The EMS problem cannot be described in the same way as the fire problem because data are not available in disaggregated fashion, or at all, to give a sense of whether the relatively
23 new U.S. investment in public-agency-delivered EMS is comparable to other countries or cost- effective.
The final section looks at the current state of National Preparedness from multiple perspectives. It first reviews what information exists about the capabilities of fire departments and EMS compared with a standard level of service. Next, it analyzes the response to past national disasters. The chapter concludes with a summary of the self-reported preparedness of states and urban areas for catastrophic events and a basic description of the extensive plans and guidance developed by DHS to help the nation become more prepared.
THE FIRE PROBLEM
In recent U.S. history, a series of major fires each led to greater recognition that an overarching public policy and public administration approach did not exist related to fire. Fires in New York City, Boston, Chicago, San Francisco after the 1906 earthquake, and the increasingly significant wildland fires of the modern period each painted a dramatic picture of economic losses, injuries and public deaths. In 1947, President Truman voiced the nation’s concern and urged a more effective public administration response. As the Truman report illustrates, experts have long recognized that fires can be avoided, suppressed, and their effects on individuals and on society mitigated if we address the “fire problem” in a thoughtful manner. However, many of the findings of the Truman-era report were neglected, and for many years the report was largely forgotten.
In 1973, America Burning re-kindled the flame of concern in the U.S., and elsewhere, as it reported on a serious neglect of the fire problem.5 Public policy makers and administrators heeded the call of this galvanizing report, and the U.S. has been successful to a great degree in responding to its findings and recommendations.
There has been a steady decline in the total number of structural fires in the U.S. since at least the mid-1970s. Most people attribute the declines to more widespread acceptance of smoke alarms, better zoning and building code enforcement, advances in flame-retardant building materials, safer products (like self-extinguishing cigarettes and space-heaters that turn themselves off), gradual replacement of older structures, and improvements in public awareness.
Unfortunately, it appears that the positive, downward trend in the number of fire incidents flattened out beginning in the mid-1990s. It is not clear why the rate of improvement slowed so rapidly after decades of progress. It also is possible that the National Fire Incident Reporting System (NFIRS) has simply improved the flow of data, making it appear that the rate is staying much the same while improvement continues, or that the apparent improvements were overstated (see Appendix B for a discussion of NFIRS data collection). While more research would be required to fully understand these dynamics, the Panel sees the recent flattening of this trend line and some other evidence as suggesting we may be at a point of declining marginal returns from the current public policy and administrative approach to the fire problem. The Panel observes that the emphasis of current spending policies—including use of the AFG program—has been on
5 National Commission on Fire Prevention and Control.
24 building response capabilities. The AFG may need to consider a change to its strategic direction if returns from investing in response capabilities are, in fact, diminishing. This and other potential strategic directions for the program will be discussed more in Chapter 6.
Figure 3-1 shows the total estimated number of fires by type in the U.S. since 1977. The total number of fires is less than one-half of that in 1977 and the number of structure fires is nearly one-half according to these estimates.
Figure 3-1. Estimate of Fires by Type
Source: NFPA Survey of Fire Departments for U.S. Fire Experience (1977-2005) in Karter, Michael J. Jr., NFPA Fire Analysis and Research Division. Fire Loss in the United States During 2005—Full Report. Quincy, MA: National Fire Protection Association, 2006. (Available at http://www.nfpa.org/assets/files/PDF/OS.fireloss.pdf, accessed 10/13/2006).
Trends in Death and Injury of Civilians
People are less likely to die in a fire today than 25 years ago. Figure 3-2 shows the trend in the number of residential deaths annually for each 100,000 population. Other fire-related data series (death by burn and by flame outside of home) changed to add new categories after 1998, so they could not be used to establish a meaningful trend over the same time period. It is clear, however, that there has been steady improvement in fire safety over the last 25 years. The rate of deaths per fire has been more than cut in half, even as the number of fires dropped in half themselves. However, the improvement appears to have slowed or ended beginning in the late 1990s.
25 Figure 3-2. U.S. Residential Fire/Flame Deaths and Rates per 100,000 All Races, Both Sexes, All Ages (1981-2003, Place of Accident = Home)
2.5
2
1.5
1
0.5
0 1980 1985 1990 1995 2000
Centers for Disease Control and Prevention, National Center for Injury Prevention and Control. WISQARS (Web-based Injury Statistics Query and Reporting System) WISQARS Injury Mortality Reports. (Available at http://www.cdc.gov/ncipc/wisqars/default.htm, accessed 10/13/2006). Figure 3-3 shows the absolute number of civilians killed by fires in the home. These data are not adjusted for population changes, but show a trend similar to the one above.
Figure 3-3. Number of Civilian Fire Deaths in the Home (1977-2005)
Source: NFPA Survey of Fire Departments for U.S. Fire Experience (1977-2005) in Karter, Michael J. Jr., NFPA Fire Analysis and Research Division. Fire Loss in the United States During 2005—Full Report. Quincy, MA: National Fire Protection Association, 2006. (Available at http://www.nfpa.org/assets/files/PDF/OS.fireloss.pdf, accessed 10/13/2006).
26 Figure 3-4 shows a comparison of fire deaths per 100,000 population in several countries around the world. Notice, however, that 2,791 deaths from the September 11, 2001, attacks are included in the U.S. number. Adjusting the value by removing the 9/11 deaths produces a rate of 1.41 for the U.S., eighteenth best among the countries listed instead of twenty-fourth.
The statistics produced by the Geneva Association in Figure 3-4 do not agree with the data provided by the Centers for Disease Control (CDC) (see Figure 3-2), which have U.S. fire/flame deaths per 100,000 people ranging from 1.05 to 0.99 in recent years. This is because the CDC uses death certificate data, which routinely omit some categories of fire deaths including vehicle fire deaths and the arson fire deaths categorized as homicide or suicide.6
Figure 3-4. Population Comparisons for Fire Deaths in Select Countries
Source: The Geneva Association (2005). World Fire Statistics. International Association for the Study of Insurance Economics. Note: The arrow graphic and associated 9/11/01 calculation have been overlaid on the figure by Academy staff.
A decline in the rate of public, non-fatal injuries related to fires and burns occurred during the limited period of time from 2001 to 2005, as shown in Figure 3-5, coincident with the operation of the AFG program. Further investigation is required to understand the factors involved in this positive trend.
6 E-mail communication, Tony Paish, World Fire Statistics Centre, December 14, 2006.
27 Figure 3-5. Overall U.S. Fire/Burn Nonfatal Injuries and Rates per 100,000 (2001-2005)
185 180 175 170
165 160 155 150 145 2001 2002 2003 2004 2005
Source: Centers for Disease Control and Prevention, National Center for Injury Prevention and Control. WISQARS Nonfatal Injuries: Nonfatal Injury Reports. (Available at http://www.cdc.gov/ncipc/wisqars/default.htm, Accessed 10/13/2006).
There are dramatic differences in the rate of death across demographic categories, as shown in a CDC study of the data from 1998-1999 (see Table 3-1). For example, African-Americans are three times more likely than whites to die in residential fires. Males are 78 percent more likely to die in residential fires than females. Those with less than a high school education are five times as likely as those with some college education to die in a residential fire.
28 Table 3-1. Residential Fire Deaths by Demographic Category
Residential Fire Deaths Total Population, 1998 (Rate per 100,000) TOTAL 1.2 Race and ethnicity American Indian or Alaska Native 2.1 Asian or Pacific Islander 0.3 Asian Data are not collected Native Hawaiian and other Pacific Islander Data are not collected Black or African American 3.0 White 1.0 Hispanic or Latino 0.9 Cuban Data are statistically unreliable Mexican 0.9 Puerto Rican 1.2 Not Hispanic or Latino 1.2 Black or African American 3.0 White 1.0 Gender Female 0.9 Male 1.6 Education level (aged 25 to 64 years) Less than high school 2.0 High school graduate 1.2 At least some college 0.4 Select populations Persons aged 4 years and under 1.6 (not age adjusted) Persons aged 65 years and older 3.2 (not age adjusted) Black or African American 3.0 Females 2.2 Males 4.0
Note: Age adjusted to the year 2000 standard population. Source: Centers for Disease Control and Prevention, National Center for Health Statistics, Healthy People 2000 Review, 1998–99. (Available at http://www.healthypeople.gov/Document/HTML/Volume2/15Injury.htm, Accessed 10/16/2006
Career and Volunteer Firefighters
Most U.S. firefighters are volunteers. While the absolute number of firefighters has increased from 1,088,950 in 2001 to 1,101,250 in 2005, the proportion of career firefighters has also increased (see Figure 3-6). Several explanations for the shift were offered by staff of the NFPA, including:
29
• People have busier schedules with more two-income homes and longer work weeks.
• Suburbanization post-WWII meant that more volunteers did not work where they live, so they are not close to their communities during the day as required to respond to fire calls.
• Additional modern training and accreditation requirements encourages career firefighters while the costs of this discourage volunteerism.
• Unionization encourages career firefighters.
Figure 3-6. Career/Volunteer Firefighters
1,200,000
1,000,000
800,000 822,850 807,150 Volunteer 600,000 Career 400,000
200,000 266,100 294,100 0 2001 2005
Sources: FEMA, U.S. Fire Administration and National Fire Protection Association International. A Needs Assessment of the U.S. Fire Service: A Cooperative Study Authorized by U.S. Public Law 106-398. FA-240. December 2002. U.S. Department of Homeland Security and National Fire Protection Association International. Four Years Later—A Second Needs Assessment of the U.S. Fire Service: A Cooperative Study Authorized by U.S. Public Law 108-767, Title XXXVI. FA-303. February 2006.
Despite the trend, the U.S. depends very heavily on volunteers for fire control. This is especially the case in smaller and rural communities.
Trends in Firefighter Deaths
The number of fire fighter deaths per year has been gradually trending downward for both career and volunteer firefighters since the late 1970s, but trend appears to have leveled off since the mid-1990s (see Figure 3-7).
30 Figure 3-7. Firefighter Deaths per Year—Career/Volunteer 1977-2005
Source: Fahy, R. and P. LeBlanc, NFPA Fire Analysis and Research Division. Firefighter Fatalities in the United States–2005. Quincy, MA: National Fire Protection Association, 2006. (Available at http://www.nfpa.org/assets/files/PDF/osfff.pdf, accessed 10/13/2006).
Although both structure fires and firefighter deaths have declined, the rate of firefighter deaths per fire has remained nearly constant in recent years even as the rate of structure fires has continued to fall.7 However, these data cover the period of time ending before the initial AFG grants were awarded, and they are subject to numerous other influences such as aging trends and related increases in heart disease. The top three causes of firefighter deaths other than heart attacks are smoke inhalation, burns, and crushing injuries (see Figure 3-8).
7 Fahy, 2002.
31 Figure 3-8. Rates of Firefighter Non-Heart Attack Deaths, by Common Cause
Source: Fahy, Rita F. U.S. Fire Service Fatalities in Structure Fires, 1977-2000. Quincy, MA: National Fire Protection Association, 2002. (Available at http://www.nfpa.org/assets/files/PDF/fffstructure.pdf, accessed 10/11/2006).
The leading causes of fatal injuries to firefighters in a structural fire that lead to the deaths shown in Figure 3-8 are: (1) getting lost inside a structure; (2) structural collapse; and (3) fire progress. As a rate of incidence per fire, those three causes have actually increased since the early 1980s. An NFPA study analyzed these data and concluded that more research must be done to understand the cause of the recent rise in the three largest causes of firefighter fatal injuries.8 These data cover the period of time ending before the initial AFG grants were awarded. Also recall that the actual number of firefighter deaths has been declining even as the number of firefighters has increased, lowering the death rate per firefighter.
The leveling off or slight increase in the rate of death/injury for firefighters is another example of an inflection point that may signal a need for a change in strategy. More research is required to understand the causes. For example, it should be investigated whether advanced equipment and apparatus allow more firefighters to undertake higher-risk activities like remaining inside a structure fire for longer periods of time or entering hotter sections of the structure than they could in the past. Such research is necessary because without proper understanding and training
8 Fahy, 2002.
32 on such risks, further investments might lead to more rather than fewer firefighter deaths and serious injuries, a negative return on investment.
Trends in Fire-Related Property Loss
While the number of fires has declined, the loss per structure fire, adjusted for inflation, has increased (see Figure 3-9). This may be partly due to increases in property value, building costs and home furnishings that exceeded the rate of inflation over this period, but more research would be required to determine this conclusively.
Figure 3-9. US Average Structure Loss per Structure Fire (1977-2005)
Note: Does not include the events of 9/11/01. Source: NFPA Annual Survey of Fire Departments for U.S. Fire Experience (1977-2005) in Karter, Michael J. Jr., NFPA Fire Analysis and Research Division. Fire Loss in the United States During 2005—Full Report. Quincy, MA: National Fire Protection Association, 2006. (Available at http://www.nfpa.org/assets/files/PDF/OS.fireloss.pdf, accessed 10/13/2006).
Prevention
Many have argued for a long time that the most effective way to prevent residential injuries and deaths, firefighter injuries and economic loss is to prevent residential fires from occurring. President Truman made this argument in 1947:
The serious losses in life and property resulting annually from fires cause me deep concern. I am sure that such unnecessary waste can be reduced. The substantial
33 progress made in the science of fire prevention and fire protection in this country during the past 40 years convinces me that the means are available for limiting this unnecessary destruction.9
A similar observation was made in 1973 in a report on the fire problem:
Response to important social changes is a key to improving the nation’s record in fire protection. A consideration of equal importance is the need to change priorities in the field of fire protection. Currently, about 95 cents of every dollar spent on the fire services is used to extinguish fires; only about 5 cents is spent on effort—mostly fire prevention inspections and public education programs—to prevent fires from starting.10
Since 1947, has there been a shift from fighting fires to preventing fires? While there have been no recent national studies of fire prevention expenditures, local government budgets often break out prevention activities. The City and County of Denver, for example, reported spending 6.7 percent of their $85 million 2005 fire safety budget on prevention efforts. Perhaps more interesting is that Denver tracks prevention-related, intermediate process outputs using performance indicators such as the number of construction plan reviews performed, the speed of plan reviews, the speed of replying to requests for permits of occupancy, and the number of annual inspections completed, but does not use the number of fire incidents to judge successful prevention programs.11
The study looked for evidence on whether the U.S. is focusing its resources in one way or another compared with other countries, and whether there are differences in the outcomes. The study attempted to look, for instance, for comparisons related to response capabilities and for those related to prevention efforts.
A comparison between the U.S. and Hong Kong is offered by Skyaid.org (2006): “The people of Hong Kong display strong motivations toward fire prevention efforts and accept individual responsibility for fire safety. The Hong Kong Fire Department conducts approximately 140,000 fire inspections annually, with three-quarters of these based on citizen complaints.” With a population of about 7 million, the Hong Kong Fire Services responded to 35,092 fire calls and 20,059 special service calls in 2004 in a city where fires claimed nine lives and caused 405 civilian injuries in 2004.12 Based on this report, Hong Kong had fewer than six non-fatal injuries per 100,000 population of Hong Kong compared with the 156 non-fatal injuries per 100,000 population in the U.S. (see Figure 3-5).
Some factors to consider when comparing the U.S. and Hong Kong are the relatively newer buildings in Hong Kong and the fact that the majority of the population lives in government- owned high-rise apartments. In a 2001 article in the NFPA Journal, retired director of the Hong Kong Fire Services Department, John Tsang Kwong-yu, explains, “The number of major fires in
9 President Harry S. Truman’s address to the 1947 President’s Conference on Fire Prevention. 10 America Burning, 1973, p. 7. 11 City and County of Denver, 2005 Budget. 12 Press release from the Hong Kong Fire Services Department, 2006.
34 new buildings is dropping because of proper fire protection systems. Since the Fire Safety (Commercial Buildings) Ordinance went into effect in 1997, three-alarm fires in commercial buildings have dropped from 10 major fires in 1997 to 3 in 1999.” Hong Kong also has a much denser, more culturally homogeneous, population than many parts of the U.S.
While behavioral observation points to a possible shift in American attitudes about fire prevention, it is hard to draw strong inferences about the reasons for the observed differences that would suggest effective strategies.
One generally accepted prevention or mitigation approach is to emphasize smoke alarms to wake sleeping residents and/or to alert residents of the need to extinguish a fire or exit their home to call firefighters to assist.13
In the U.S., there is not great variation among demographic groups in the proportion of residents with functioning smoke alarms on every floor. Table 3-2 below shows, for example, that Hispanic or Latino residents are 7 percent less likely to have a functioning smoke alarm than other groups. It also shows that those with less than a high-school education are 7 percent less likely to have a smoke alarm on every floor. The relatively small differences in smoke detector usage by demographic category suggest that greater smoke detector use has not eliminated the wide differences in residential deaths by demographic category noted earlier in this chapter (see Table 3-1).
13 Such strategies are sometimes labeled “secondary prevention” to distinguish them from strategies used to prevent fires from breaking out.
35 Table 3-2. Functioning Residential Smoke Alarms by Demographics
Live in Residences With Population, 1998 Functioning Smoke Alarm on Every Floor (percent)
TOTAL 88 Race and ethnicity American Indian or Alaska Native 84 Asian or Pacific Islander 90 Asian 91 Native Hawaiian and other Pacific Islander 88 Black or African American 86 White 88 Hispanic or Latino 81 Not Hispanic or Latino 88 Black or African American 86 White 89 Gender Female 88 Male 87 Education level (aged 25 years and older) Less than high school 81 High school graduate 88 At least some college 90 Note: Age adjusted to the year 2000 standard population. Source: Centers for Disease Control and Prevention, National Center for Health Statistics, Healthy People 2000 Review, 1998–99. (http://www.healthypeople.gov/Document/HTML/Volume2/15Injury.htm).
To increase the number of households with functional smoke alarms, many communities sponsor smoke alarm give-away campaigns. As described in Box 3-1, the effectiveness of such programs has been statistically validated, particularly when targeted to high-risk communities.
36
Box 3-1. Effectiveness of Smoke Alarm Give-A-Ways in Reducing Fire Injuries
Background: Researchers collected data on burn injuries in the 24-square-mile area of Oklahoma City with the highest rate of fire-related injuries from September 1987 through April 1990. The team distributed smoke alarms door to door in the target area and then surveyed alarm use and function in a sample of the homes that had received an alarm, and the rest of the city, before and after the smoke-alarm giveaway.
Results: Before the intervention the rate of burn injuries per 100,000 population was 4.2 times higher in the target area than in the rest of Oklahoma City. An initial survey indicated that 11,881 of the 34,945 homes in the target area (34 percent) did not have smoke alarms. A total of 10,100 smoke alarms were distributed to 9291 homes; 45 percent were functioning four years later. The annualized fire-injury rates declined by 80 percent in the target area during the four years after the intervention (from 15.3 to 3.1 per 100,000 population), as compared with a small increase in the rest of the city (from 3.6 to 3.9 per 100,000 population). There was also a 74 percent decline in the target area in the injury rate per 100 fires (from 5.0 to 1.3; rate ratio, 0.3; 95 percent confidence interval, 0.1 to 0.6), as compared with a small increase in the rest of the city.
Conclusions: A targeted intervention involving a smoke-alarm-giveaway program can reduce the incidence of injuries from residential fires.
According to the U.S. Fire Administration (2006), smoke detectors were present in 60 percent of fatal residential structural fires. And in fatal fires with a smoke detector, they operated 39 percent of the time. The causes of three-fourths of fatal residential fires where smoke alarms operated are smoking, arson and open flame (see Table 3-3). Notice that smoking is twice as likely to be the cause of fatal fires in apartments as in 1- and 2-bedroom homes.
Table 3-3. Leading Causes of Fatal Residential Structure Fires with Working Smoke Alarms
Source: U.S. Fire Administration, 2006 (NFIRS 5.0 data, confined fires are excluded)
Smoking, arson and open flames may cause fires, but why do people die in fires? The U.S. Fire Administration analyzed the human factors that were found to have contributed to residential
37 structure deaths. They found that over one-third of all fatalities, and over 80 percent of child deaths, were the result of sleeping individuals with no known impairments—fully preventable (see Table 3-4). Alcohol impairment and physical disabilities were also major factors.
Table 3-4. Human Factors Contributing to Fatal Injury in Residential Structure Fires with Working Smoke Alarms, 2001-2004 (percent)
Source: U.S. Fire Administration, 2006 (NFIRS 5.0 data, confined fires are excluded).
Another prevention strategy is the use of sprinklers. According to information provided by the Home Fire Sprinkler Coalition (2006) and NFPA’s “U.S. Experience with Sprinklers” (2003), sprinklers reduce chances of dying in a fire and the average property loss from a fire by one-half to two-thirds compared to where sprinklers are not present. Further, the risk of dying in a home fire is reduced 82 percent by having both fire alarms and sprinklers in a home compared with having neither.
In 1999, 34 percent of public assembly properties and seven percent of residential properties where fires occurred were covered by sprinklers. However, despite the evidence showing that sprinkler requirements are cost-effective, some communities face inter-governmental barriers to enacting such regulations, as described in Box 3-2. Similarly, Congress has considered but not adopted legislation such as the Fire Sprinkler Incentive Act, which would encourage developers to install sprinklers by accelerating their depreciation to a much shorter period of time.
38 Box 3-2. Community Sprinkler Efforts Prevented By Mini-Max Building Codes
The rapid growth of one small New Jersey community has stretched its volunteer fire service to its limits. Within the community, there was a preference to address the growth by requiring sprinklers in all new residential developments. This was seen as a way to lower the number of fire incidents, firefighting resources, and civilian injuries/deaths from fire. However, several states including New Jersey have state-wide building codes that are known as “mini-max.” This term means that in addition to adopting state-mandated minimum requirements for the building codes, the local jurisdictions are prevented from enacting any codes that are stricter than the state-wide ones. Ostensibly, these types of codes reduce the burdens on large home builders that would have to comply with a patchwork of local building codes. The opposing argument, however, is that lobbying efforts by these home builders have effectively put up a wall to communities that are trying to take a progressive and cost-effective approach to reducing fire deaths, injuries and economic loss.
The effectiveness of numerous available prevention options has not been established by research. The questions are, in each case: how much public money should be spent on prevention compared with response capabilities; and how can interventions be designed to maximize their effects, e.g., can market-based incentives influence consumers to be more responsive to investments in fire prevention?
While most communities have some sort of fire education program available, the infrequency of fire incidents makes it difficult to draw statistically significant inferences about the effectiveness of each program. Two studies of educational prevention programs are described in Box 3-3 below.
39 Box 3-3. Available Fire Education Programs
An available educational tool is “Risk Watch.” This is a grade-specific curriculum developed by the National Fire Protection Association designed to teach students about injury prevention. A three-year evaluation of the curriculum was completed in 2001. It shows that Risk Watch is an effective way to increase preschool through eighth grade students’ knowledge about safety issues, although the sample size was too small to find a statistically-significant impact on fire incidence or injury. “Nevertheless, in the three years that the Risk Watch evaluation was underway, NFPA documented five cases where students avoided injury or death (or caused them to be avoided by family members) because they put into action the knowledge gained from Risk Watch.”14
To reach children younger than 14 years (considered to be a high-risk group for burn- related and fire-related injuries), two prevention games were developed and distributed to 38 school districts encompassing a total of 164 elementary schools and reaching more than 1,035 children in grades one through four in a two-county community. Before playing each game, the participants completed a multi-choice pretest. Pretest and posttest results indicated students gained and retained significant knowledge about fires and burns.15
Fire Expenditures in Proportion to the Economy
While it appears that the United States spends a higher proportion of its Gross Domestic Product (GDP) on fire protection than most other industrial nations (see Table 3-5), distortions common in international comparisons must be considered. U.S. fire protection expenditures include EMS expenditures for those communities where it is publicly funded and part of the fire department. Volunteer resources, which have been estimated to be as much as $37 billion per year in the U.S., are only included to the extent that they incur direct costs.16
Table 3-5 shows a comparison of numbers of firefighters and the portion that are career (paid/full-time) from a subset of industrial countries. Note that the UK has a centralized career fire service that is separate from the EMS function, while most other developed countries are like the U.S. in terms of using a mix of volunteers and careerists via an intergovernmental delivery system with wide variety in the approach to the EMS function.
14 NFPA, 2001. 15 Mondozzi & Harper, 2001. 16 IFCA Blue Ribbon Report, 2004.
40
Table 3-5. Comparison of Fire Fighters, Population and Area of Industrialized Countries
Subset that Population Area Covered Total Country are Career Served (per (per Notes Firefighters Firefighters firefighter) firefighter)
2 United 300 million 3,718,695 mi 1,101,000 294,000 (27%) States (272.5/ff) (3.8 mi2/ff) Provides limited first- responder EMS, but a 60.6 million 260,558 mi2 France 220,000 31,000 (14%) separate hospital-based (275.5/ff) 2 (1.8 mi /ff) service provides the full EMS.
2 127 million 145,883 mi Japan 1,100,000 154,000 (14%) (116.2/ff) (0.13 mi2/ff) Cities with a population of more than 80,000- 82.4 million 137,858 mi2 Germany 1,384,000 “most” 100,000 required a (59.5/ff) 2 (0.1 mi /ff) professional firefighting force. Has a centralized career England 2 53.4 million 58,368 mi fire service that is and 45,000 33,000 (73%) (38.6/ff) 2 separate from the EMS Wales (1.3 mi /ff) function.
These data show that the U.S. uses five times the number of firefighters as France to protect about five times the population, so very comparable, but the U.S. fire service covers much more geography and U.S. fire departments provide more EMS within these same numbers. The U.S. looks more productive than France in these terms. Compared to Japan, the U.S. covers 2.3 times more population and much more geography with about the same number of firefighters.
The Geneva Association’s World Fire Statistics Centre reports attempt to determine the costs related to fire service, excluding EMS and other non-fire work. According to a spokesperson for the World Fire Statistics Centre, the respondent in each country either uses statistical sources to identify separate figures for fire-related costs (e.g., Denmark) or estimates a percentage adjustment for non-fire work (e.g., ten percent). However, where a country has a partially volunteer fire service, the respondent concerned is asked to report the actual costs incurred by the volunteer organization in fire-related work (including training), but they do not attempt to place a monetary value on volunteers’ time spent in fire work outside working hours, only on actual costs incurred by the volunteer fire agency, including training.17 France, as an example, has about the same number of firefighters on a per capita basis but appears to use many more
17 E-mail communication, Tony Paish, World Fire Statistics Centre, December 14, 2006.
41 volunteers than the U.S. So the different composition of career/volunteer fire services by country requires international cost comparisons in Table 3-6 to be made with caution.
Table 3-6. Fire Protection as an Average Percent of GDP (2000-2002)
Source: The Geneva Association (2005). World Fire Statistics. International Association for the Study of Insurance Economics.
Reported public costs for firefighting organizations vary greatly among industrialized countries, from 0.04 percent of the national economy in Singapore to 0.35 percent in Canada. The U.S. figure, 0.25 percent, is shown in comparison with other industrialized countries in Table 3-7.18 Factors contributing to the U.S.’s apparent higher cost in this comparison include an increased investment in higher response standards and capability (in terms of computer-aided dispatch, E- 911, vehicles, equipment and personal protective equipment), a much larger geography, and wider variation in population groupings (i.e., non-homogeneity of culture) than many other industrialized nations.
18 U.S. fire protection expenditures include EMS in those communities where it is run by the fire department and publicly funded but it excludes resources provided through volunteerism.
42 Table 3-7. Fire Fighting Organization Cost as an Average Percent of GDP (2000-2002)
Source: The Geneva Association (2005). World Fire Statistics. International Association for the Study of Insurance Economics.
Information from the previous tables can be used to approximate the relative productivity of each country’s fire protection efforts. Table 3-8 below shows a ratio of the deaths per 100,000 population divided by the percent of GDP that each of these industrialized countries spent on fire protection. The result shows the U.S. with a relatively poor result. (Note: The U.S. ranking remains unchanged by including the deaths on September 11, 2001, with the U.S. value only changing from 1.6 to 2.0.)
43
Table 3-8. Ratio of Number of Fire Deaths to the Percent of GDP Spent on Fire Protection (higher numbers are more positive)
Inverse of Deaths per 100K / Country %GDP Spent on Fire Protection Singapore 20.8 New Zealand 7.0 Slovenia 6.7 France 6.6 Switzerland 6.2 Czech Republic 5.9 Netherlands 4.9 UK 4.7 Italy 4.5 Sweden 4.2 Japan 3.6 Canada 3.2 Belgium 2.6 Norway 2.2 USA 1.6 Denmark 1.3 Hungary 1.2
Note: US Deaths from September 11, 2001, have been removed before calculating. Source: Calculated based on data from The Geneva Association (2005). World Fire Statistics. International Association for the Study of Insurance Economics.
The U.S. spends 0.25 percent of Gross Domestic Product on fire services, one of the highest reported proportions among industrial nations, even excluding the significant value of volunteer contributions in terms of lost wages. A number of these countries report better performance than the U.S. as measured by fire death rates. Such information, at first examination, suggests that the U.S. may not be using its resources for fire protection as cost-effectively as some other developed nations. However, such inferences are not easily made, given the differences in the cost of meeting the fire challenge that may require greater expenditure in one nation than another. For example, the wide areas of the U.S. where population density is low but basic fire response capability must be sustained add to the cost of fire protection relative to a country like, say, Denmark. On the other hand, the strong tradition in the U.S. of volunteer firefighting and charitable donations to support local fire service tends to reduce public costs in the U.S. relative to some other nations.19 International comparisons, especially on the cost side, are problematic and a detailed international comparison of the relative effectiveness of fire protection and
19 Japan and France use a higher percentage of volunteers than the U.S. France has a better death rate while Japan’s rate is not as good as the U.S. (on a population basis).
44 prevention practices is beyond the scope of this report. However, the wide variation in results among industrial countries indicates that such analysis would likely yield valuable insights.
THE EMERGENCY MEDICAL SERVICES PROBLEM
In the early 1960s, the majority of ambulances in the U.S. were run by mortuary service companies. Emergency medical services have undergone dramatic changes since then. A primary impetus for this transformation was the discovery that CPR, defibrillation and other techniques, if applied quickly, can dramatically increase survival and recovery rates for certain patients. Public policy and public administration have changed remarkably too. Ambulance services went from being privately-provided to the point where nearly one-half are provided by fire departments and another portion is provided by other public means, such as through the non- affiliated EMS organizations that are now eligible for the AFG program.
There are estimated to be more than 15,000 EMS systems and 800,000 EMS personnel in the U.S.20
The skill levels of EMS personnel vary greatly related in part to whether they are volunteers or paid employees. Skill levels have implications for the flexibility of dispatchers to assign teams to various incidents as well as for the quality of care a patient receives once the team arrives. The number of EMS-related incidents varies by geography too. Overall, 57 percent of EMS personnel respond to fewer than 10 calls per week. In rural areas, however, 91 percent face fewer than 10 calls per week.21 Skill levels are also affected by the opportunity to practice the skills and to see a variety of medically related incidents. The National Registry of Emergency Medical Technicians describes the skill level of the volunteer and career workforces.22 (see Figure 3-10.) In this regard, it should be no surprise that volunteers, who dominate the composition of rural services, would possess fewer skills because of the smaller number and limited variety of incidents they face. Unfortunately, this means that, in general, people who live in rural areas can expect to be served by less-skilled personnel.
20 Mears, 2004; McCaig and Burt, 2005; Lindstrom, 2006. 21 National Registry of Emergency Medical Technicians (NREMT), 2003. 22 Ibid.
45 Figure 3-10. Skills Level of EMS Personnel, by Workforce Category
100% 90% 80% 46.5% 70%
60% EMT - Basic 89.5% 50% Paramedic 40% Intermediate 30% 46.3% 20%
10% 5.2% 5.3% 7.2% 0% Volunteers Paid
Source: NREMT, 2003.
While there is a perceived shortage of Emergency Medical Technicians (EMTs) today, the Bureau of Justice and Bureau of Labor Statistics estimates the need for EMTs will grow by 59,000 between 2002 and 2012, largely due to population increases, aging, and urbanization.23
EMS responds to an estimated 16 million annual transport calls.24 This appears to be consistent with the numbers in NFIRS, an incident-based reporting system that is voluntary and used by approximately one-third of fire departments, and which had 7 to 9 million EMS incident entries last year, according to staff of the U.S. Fire Administration.25
As of 2006, nearly one-half of EMS operations were fire-based, meaning their services were delivered through a fire department. But there are a number of other organizational arrangements involving municipal governments, hospitals, for-profit companies and nonprofits. For most Americans, EMS is coordinated and dispatched by E-911 call centers.26
Figure 3-11 shows that there has been a slight decline since 2001 in the number of fire departments that report having EMS duties included in their scope. The “Out of Scope” responses primarily came from the smallest communities responding to the survey.
23 Bureau of Labor Statistics, 2004, as explained in Institutes of Medicine IOM, 2006. 24 Mears, 2004; McCaig and Burt, 2005; Lindstrom, 2006. 25 Personal interviews, October 19, 2006. 26 IOM, 2006.
46 Figure 3-11. Percentage of Fire Departments Reporting EMS to be “In Scope”
36 34 32 30 28 35 26 33
Percentage 24 22 20 2001 2005
Rural EMS
Like fire services, EMS are generally not able to provide a level of service for rural residents comparable to that for urban or suburban residents. An Institute of Medicine (IOM) report provides several reasons for this discrepancy:
• Some 77 percent of rural EMS personnel are volunteers compared with 33 percent of urban EMS personnel.27
• Response times are much greater in rural areas due to the large areas to which each EMS is assigned. Thirty percent of fatally injured crash victims arrive at the hospital more than 60 minutes after the crash, compared with 8.3 percent of urban victims.28
• Rural address-identification systems often prevent effective E-911 locating technology. Further, 4 percent of counties still do not have E-911 at all.
• Long periods of time may pass on rural routes before someone in need of medical care is discovered by a passer-by.
• Advanced life support techniques, such as endotracheal intubation, are not well-provided because personnel face an infrequent need to perform the task, training deficiencies and inconsistent supervision.29
• It is difficult to recruit and retain medical directors and staff with emergency medicine training in rural areas.
Low or inconsistent cell phone coverage in rural areas is also a contributing factor to delays in contacting dispatchers.
27 Minnesota Dept. of Health, Office of Rural Health Primary Care, 2003. 28 National Highway Traffic Safety Administration (NHTSA), 2005. 29 Bradley et al., 1998.
47 According to the IOM report, these issues can be addressed through approaches such as strategically locating available ambulances throughout a jurisdiction; establishing regionally based EMS to gain economies of scale in equipment, training, staffing and dispatch; and implementing programs to increase citizen engagement in CPR and other measures while awaiting EMS arrival.
Aging and EMS
While the number of fires in the U.S. is slowly declining, the number of EMS calls is rising and expected to rise faster in the coming decades. Of patients arriving in emergency departments, 40.9 percent of patients over the age of 74 arrive via ambulance while only 14.2 percent of all patients, regardless of age, arrive via ambulance.30 According to recent census information, 6.9 percent of Americans were age 75 and older in 2000, but 11.3 percent will be by 2040.31 So by 2040, EMS needs to be prepared for a 63 percent increase in the portion of people who are nearly three times as likely to require an ambulance to get medical treatment.
On-the-Job Injuries and Deaths of EMS Personnel
Four in five EMS workers have experienced some kind of injury or medical condition as a result of their work, according to the National Association of Emergency Medical Technicians (NAEMT)32 “Experiences with Emergency Medical Services Survey.” A total of 1,356 NAEMT members participated in the survey, jointly commissioned by NAEMT and McNeil Consumer & Specialty Pharmaceuticals, reporting that:
• 52 percent have been assaulted by a patient;
• 50 percent have been exposed to an infectious disease;
• 47 percent have sustained back injury while performing EMS duties; and
• 21 percent have contracted an illness from a patient.33
In addition to the injuries listed above, there are additional dangers to EMS staff who navigate streets at high speeds without adhering to traffic signals. As stated in the IOM report, the CDC found that there were 300 fatal crashes involving ambulances in the U.S. between 1991 and 2000, in which 275 out of 357 deaths were people outside the ambulance such as other drivers and pedestrians.34 Other studies cited by IOM conclude that training, procedures and inexpensive devices could drastically lower these rates, such as vehicle operator training, using lights and sirens only when necessary, and using harnesses to secure the EMTs in the back of the
30 McCaig and Burt, 2005. 31 American Institute on Aging, 2006. 32 NAEMT is a professional association representing paid and volunteer emergency medical technicians and paramedics nationwide. See www.naemt.org. 33 NAEMT, 2006. 34 CDC, 2003.
48 ambulance while working. Note that firefighter deaths from traffic-related injuries exceed the figures noted by the CDC for ambulance services and may also benefit from similar remedies.
The study did not uncover internationally comparable statistics related to EMS and is unable to calculate a ratio relating EMS deaths to the economic resources attributed to the function (a productivity measure) due to a lack of EMS death-related statistics and data on expenditures for EMS.
The most frequently mentioned performance metric for EMS relates to the survival rate for patients where a cardiac event was witnessed by someone. An often-cited statistic for the survival rate for witnessed-cardiac events is around 6 percent in the U.S.; however, one news story noted a survival rate consistently over 40 percent for witnessed-cardiac events in one Minnesota community that uses the most advanced approaches.35
Based on the findings of the IOM study, the following two questions should be addressed: (1) why is the performance on cardiac events so low, and (2) why do we not find other metrics being used by fire departments for some of the other serious medical diagnoses faced by EMS personnel? The IOM study suggests that much more attention be brought to the policies and public administration of the EMS function within the U.S.
THE NATIONAL PREPAREDNESS PROBLEM
DHS is committed to improving preparedness for natural and man-made disasters by employing an “all-hazards” perspective. DHS has articulated a four-part taxonomy that it is using to analyze and communicate about preparedness: prevention, protection, response, and recovery. DHS officials have characterized their efforts as focused predominantly on response and on terrorism and related threats in its first years. DHS recognizes, however, the all-hazards scope of its mission and has determined that it needs to bring more attention to the prevention and protection aspects of the taxonomy. They have concluded as a policy matter that increasing attention to these parts will lower risks for the nation.
Homeland Security Presidential Directive-8 establishes the National Preparedness Goal. This goal is being operationalized by DHS using the four-part taxonomy and a focus on tasks that would need to be accomplished and capabilities that would be needed to perform these tasks. DHS has published a Targeted Capabilities List (TCL) and a Uniform Task List (UTL) to help guide both the public and private sectors of society toward greater preparedness.
DHS has produced guides that can serve fire departments, EMS functions, and other roles and functions served by the fire departments within the U.S. To understand how well-prepared the U.S. is for a large-scale disaster or other incident of similar significance, DHS wants to measure accomplishments based on the TCL and UTL. The AFG program, as part of DHS and contributor to the preparedness goal, may wish to take a parallel approach to measuring its accomplishments. Several dimensions are particularly relevant for the AFG program:
35 While there is disagreement about the definition of “survival” among EMS professionals, the IOM 2006 study refers to this term as patient’s survival from cardiac arrest or discharge from hospital.
49
Fire Service Preparedness
One dimension of preparedness has to do with the capability of fire organizations throughout the nation. In 2001 and again in 2005, DHS/FEMA commissioned a needs assessment survey in conjunction with NFPA to define the current role and activities associated with the fire services and determine the adequacy of current levels. The DHS commissioned the NFPA to undertake an additional effort to match the AFG grants with the findings from the 2005 needs assessment. This was made possible by matching survey respondents with AFG applicants through NFIRS IDs and/or address information. This analysis indicates that AFG recipients were well- represented within the respondent group. The AFG may, as a result, have made a contribution to any apparent improvements shown (when comparing the 2001 assessment to the 2005 assessment). This is discussed more in the next chapter of the report.
The TCL and UTL draft guidance offers a set of very clear performance measures for firefighting, across all aspects of the taxonomy. For instance, the first Activity for a fire department and one of its first “Critical Tasks” is the need to develop and maintain plans “based on risk and threat assessment” (Res.B2a 1.2.5).36 The Panel believes that the DHS guidance, once complete, can be used by the AFG program as a way to measure the performance of grantees and of the program. Simply put, do grant funds help an organization accomplish the performance measures laid out as a result of Directive-8?
EMS Preparedness
The second dimension of preparedness has to do with the extent to which EMS organizations throughout the nation are prepared for and ready to respond to medical emergencies that arise from large-scale incidents, including those that are regional in scope.
One measure of preparedness is the ability of EMS to coordinate their actions with those of other emergency responders. Despite recent integration in dispatch technology and advances in emergency medical techniques, there is very little coordination of information between the EMTs and other public safety agencies, participants in the continuum of care (e.g., hospitals and emergency rooms) or other EMS agencies.37
The draft guidance for the UTL identifies several critical tasks of clear interest to fire departments and others who provide EMS. For instance, among a long list focused on triage and pre-hospital preparedness are dependencies on plans and systems that:
• “ensure sufficient EMS personnel and resources are available to respond to day-to-day emergencies in the community,” and
36 Department of Homeland Security. Target Capabilities List. Triage and Pre-Hospital Treatment, WMD/Hazardous Materials Response and Decontamination, and Firefighting Operations and Support. DRAFT version provided by DHS. 37 IOM, 2006.
50 • “ensure sufficient EMS personnel, supplies, and equipment are available to respond to and manage a catastrophic incident.”
A key capability required of EMS is surge capacity, or the ability to handle patient volumes in excess of normal rates. The draft UTL guidance includes several items that relate to this, such as mutual aid agreements, recall procedures for off-duty EMS personnel, and dispatch protocols in the case of a large-scale incident. Most EMS directors and city administrators have faced fairly predictable EMS call volume, coupled with tight budgets and other more highly-visible demands on spending priorities. As a result, most communities have arrived at a steady state of capacity through all the organizations responsible for various stages of emergency medicine. The typical capacity allows the system to handle normal call volume with crowded emergency departments, an occasional need to divert ambulances from lower- to higher-priority events, and a sizeable portion of unfilled staffing positions. The implication is that there is little or no surge capacity in most communities. The IOM report also recommends key capabilities needed by EMS for responding to natural or man-made mass-casualty hazards, including specific equipment, training and surge capacity.38
Preparedness for Coordinated Response
A third dimension of national preparedness is broader than the specific roles of fire services and EMS. It is the ability, whether at a community, regional, or national level, to deliver a coordinated response to a large-scale disaster. The main question is whether the emergency planners and administrators of states and major urban areas—drawing on the experience from recent large-scale disasters as well as simulations and drills—have adequate plans, backed with resources and practice.
A congressional committee recently issued a 520-page report assessing the government’s response to Hurricane Katrina. It described a “national failure” in preparing for and responding to the event. “At every level—individual, corporate, philanthropic and governmental—we failed to meet the challenge that was Katrina,” the report says.39 Indeed, there was a series of management lapses, improper staff deployments and disruptions of relief supply distribution that contributed to the failures. However, there were process breakdowns as well that could have been avoided by having well-documented and well-rehearsed disaster plans. In summary, significant faults were found at every level of the government and in every functional area of preparedness and response.
Have the lessons of Katrina been learned since that time? A DHS review of state preparedness programs released February 10, 2006, addressed that question. DHS required all states, territories, and urban areas to respond to questions about whether the jurisdiction’s plan components were consistent with existing federal planning guidance, when they had been last exercised and updated, and whether the jurisdiction was confident in the adequacy of its plan components to manage an actual catastrophic event.
38 IOM, 2006. 39 U.S. Congress, 2006.
51 The results indicate that states’ and urban areas’ plan components are generally consistent with existing federal planning guidance,40 and that the majority of states and urban areas have exercised their plan components within the past two years. Updates to those plan components, however, have not been as consistent.
A key finding from the assessment is that, “States, Territories and urban areas report that current plans are generally consistent with existing Federal planning guidance and voluntary standards. However, many States, Territories, and urban areas expressed considerably less confidence in the adequacy and feasibility of their plans to deal with catastrophic events.” Twenty-nine percent of urban areas reported that their basic plans are not adequate to manage a catastrophic event, but interestingly, only 12 percent report those plans fail to meet federal guidelines and voluntary standards. The fact that many states and urban areas self-report they are not well-prepared for a large-scale event points to the need for further work to build capabilities specified for the National Preparedness Goal.
SUMMARY
The AFG program provides resources to communities to address any or all of the three problems outlined in this chapter: how to protect against fire; how to improve emergency medical response; and how to prepare a coordinated response to a large-scale event likely to involve these and other threats to public safety. To make the proper strategic choices, the AFG program’s leadership must understand the complex nature of these problems and use that understanding to help define the best potential use of its resources to address them, community by community and region by region.
The analysis here is not definitive in characterizing any of these challenges at a national, much less at a community, level largely because data and research are insufficient. That lack of information in itself suggests a need for better data and more research to improve understanding of the problems and thus provide a better guide to policymaking and program management.
In the interim, however, we can draw important inferences from what is known about each problem:
• With regard to the fire problem, these data suggest the U.S. may be reaching a point of diminishing returns on further investment in improved response capability, reinforcing repeated calls by expert panels to pay greater attention to fire prevention. International comparisons, while problematic, raise the question of whether the U.S. could use the resources it spends on fire protection in a more effective manner.
• With regard to EMS, available information highlights the rapid growth of these services and even more rapid growth of public spending to support them. Coupled with a relative lack of information about the most effective ways to organize and deliver these services,
40 Relevant Federal guidance includes SLG 101 [FEMA State and Local Guide (SLG) 101: Guide for All-Hazard Emergency Operations Planning] and voluntary standards such as NFPA 1600 [NFPA 1600: Standard on Disaster/Emergency Management and Business Continuity Programs].
52 this profile points to the need for a broad strategic look at how these services can be delivered most effectively.
• Finally, with regard to national preparedness, recent events have drawn attention to the tasks required at all levels of government to prepare for large-scale events. As first responders, the organizations that receive AFG support are typically at the center of community and regional planning for such events.
53
54 CHAPTER 4 THE AFG PROGRAM IN CONTEXT
A full understanding of the AFG program and its challenges requires an appreciation for the institutional, financial, and national interest context within which it operates. The history of the fire service and EMS in the U.S. influences the way these services are organized today. Administration and funding of these services is largely at the community level. The fire service, especially, has developed a strong and distinctive “culture” that values voluntarism, team work, individual courage, and rapid response. All of this must be taken into account by AFG program managers in their choice of strategy and in the way they operate the program.
This chapter briefly reviews the historic development of firefighting and EMS services in the U.S.; the roles of the three levels of government—and volunteers—in providing and financing these services; and how the AFG program fits into that preexisting intergovernmental framework. It also places the program in the context of broader federal interests, such as homeland security, and budgetary considerations, discussing these from a cost-benefit perspective.
HISTORICAL DEVELOPMENTS
Traditionally in the U.S. public safety and firefighting have been—and largely remain— community responsibilities. Almost all spending for fire protection is at the local level, with unpaid volunteers providing most of the labor for many communities.41 Only within the last 40 years has the federal government recognized a responsibility for fire-related public safety and for supporting improvements in EMS organizations across the U.S. The establishment of the AFG program in 2001 was a further step in this direction, a signal of increased interest by the Congress in direct financial support for improvements in the capacities of local fire departments.
The history of fire departments shows several periods of evolution in their development, as fire departments changed their focus and improved their capabilities. The earliest firefighting organizations were quite entrepreneurial; they were formed to lay claim for insurance company payments whenever they protected an insured property (these properties bore marks so the companies knew which properties to protect and which to ignore). The transition to recognizing the potential “public good” nature of a fire-fighting service is evidenced by the numerous volunteer firefighting organizations that arose early in U.S. history.42 Public-based, career firefighter organizations came into being in the middle 1800s in several major cities. A healthy
41 According to A Second Needs Assessment of the U.S. Fire Service (NFPA, February 2006) 73% of the 1.1 million firefighters within the U.S. are volunteers. Germany, Japan and France have a higher percentage of volunteers than the U.S. 42 Volunteer associations arose in Boston following a series of fires and these were emulated in Philadelphia beginning in the early 1700s. Benjamin Franklin created the Union Fire Company in 1736 in Philadelphia, the first volunteer fire company in America. Voluntary associations are part of the early history of most of the developed countries.
55 fire insurance industry became fully national in character by the 1870s, helping to provide a continuous sharing of successful firefighting practices across all fire departments.43
Fire departments remained focused on fire-fighting tasks as their predominant purpose for the next century or more. A very strong culture evolved around the firefighting task, a culture that is shared across the U.S. This was reinforced by the formation of a strong, organized union movement and other associations representing firefighters. Symbolically, establishment in the late 1970s of the U.S. Fire Service as a federal agency and of the federal National Fire Academy further reinforced the identity of firefighters as a group with a distinctive social mission and supporting norms.
One period of rapid evolution affecting the organization and mission of fire departments unfolded in the 1980s when EMS operations were developed and/or integrated into many departments. Fire-based EMS now covers around one-half the population. For the most part, these publicly organized EMS displaced commercial ambulance services in responding to medical emergencies. Separate commercial or private ambulance services, including volunteer rescue squads, still exist in many places. For example, in Austin, Boston, and Seattle, publicly provided EMS operations are administratively separate from fire departments. A recent study by the Institute of Medicine (IOM)44 recognized the various administrative approaches to EMS but was unable to find evidence that allowed it to comment about whether one way of organizing is more successful than another.45 The AFG program is available to provide financial support both for fire-based EMS and for non-affiliated EMS organizations that can be public or private nonprofits but not affiliated with a hospital.46
The federal government’s interest in fire department capacity was both heightened and altered by the events of late 2001, which highlighted the heroic role of firefighters as “first responders” to acts of terror. Further attention was drawn to federal responsibility for response to major disasters by the Gulf Coast hurricanes in 2005. Congress and others continue to debate and legislate concerning the mission and strategies of the DHS, which administers the AFG, requiring leaders of the Department to continually reconsider AFG’s role and how it can contribute to its broader homeland security mission.47 For some, a greater awareness of large- scale disaster-related risks and a concern about how best to prepare the nation to face those risks48 suggest that there is a stronger federal interest in making fire departments and EMS functions more responsive and capable. But to others the emphasis on preparedness for large- scale catastrophic events implies a shift of attention that would draw limited resources from fire departments to other response capabilities.
43 Baranoff interview, September 2006 44 IOM is a division of the Congressionally-chartered National Academies of Science charged with providing independent, objective evidence-based advice to policymakers. www.iom.edu. 45 IOM, 2006. 46 The 2004 reauthorization of the program in Congress added a new class of eligible organizations, Section 33(d) of the Federal Fire Prevention and Control Act of 1974, as amended. 47 Refer to the statutory changes related to DHS and the call for an EMS needs assessment, enacted as part of the Department of Homeland Security FY 2007 Appropriations, P.L. 109-295. 48 See the Priority Area Commentary, National Agenda for the Support of Intergovernmental Research, 2006; National Academy of Public Administration, July 2006, pp. 26-27.
56 One task facing federal policy makers is to clarify just how the AFG program can best be employed to meet its statutory goals within the framework of new and broader national preparedness goals. At the same time there are recognized needs to improve capabilities in addressing both fire and emergency medical services. The international statistics noted earlier also raise questions. The U.S. appears to invest much more in its firefighters than most other countries but with less success as measured by public rates of death from fire. The international differences are not fully understood; however, they invite us to look more closely at the organizational, cultural, and financial contexts that shape the roles and practices of U.S. fire departments.
FEDERAL, STATE AND LOCAL GOVERNMENT ROLES
The roles played by all levels of government in the U.S. must be taken into account by the AFG program in every aspect of its operations. The federal government is the newest kid on this block.
In the U.S., governance of the fire function and service, whether publicly or privately funded and delivered, historically has resided with local and state authorities. Governance of EMS is almost entirely at the state and local level, although the federal government helps support these services through Medicare and the state-administered Medicaid program.49
The primary role of communities in dealing with fire has been recognized for some time. However, in 1973, a national study of the fire problem, America Burning, galvanized the federal government into taking a number of actions.50 Even as it recommended new federal efforts, the President’s commission reaffirmed the traditional view of federal, state and local division of responsibilities for fire:
We felt strongly that fire prevention and control should remain primarily local responsibilities. Local governments--through codes and fire safety laws, and through heavy investments in fire department personnel and equipment--have shouldered the major burden of protecting citizens from fire and should continue to do so. Those governments appreciate special local conditions and needs more fully than an arm of the Federal Government would be able to do. Roles for the Federal Government, in the Commission’s view, are appropriately limited to lending technical and educational assistance to State and local governments, collecting and analyzing fire information, regulating the flammability of materials, conducting research and development in certain areas, and providing financial assistance when adequate fire protection lies beyond a community’s means.51
49 The National Conference of State Legislatures maintains a running catalog of the state statutes affecting EMS: http://www.ncsl.org/programs/health/ems.htm. Local councils address both fire-based and non-affiliated approaches to EMS on a regular basis. 50 National Commission on Fire Prevention and Control. America Burning, 1973. 51 Ibid.
57 Until enactment of the AFG program, federal involvement in the fire problem was primarily regulatory and focused on such things as safety standards for clothing. Indirect financial support of fire services began in the late 1970s with the first appropriations for the U.S. Fire Administration, the operation of a National Fire Academy, the organization of national data collection, and sponsorship of some research.
The federal government faces some practical and legal limits on its involvement in fighting fires or in the performance of emergency-related services like EMS. The federal government vacated most of the field of fire insurance regulation in the historic McCarran-Ferguson Act, with Congress apparently concluding that state governments were better able to regulate practices in this area. The decline in the number of fire incidents and the stability in the fire insurance industry itself may have reinforced Congress’s assumptions about limits on the federal regulatory role.52 In the special circumstances of wild land fires and especially for fires on federal lands and property, the federal government has assumed greater responsibility. Because federal fire- fighting capabilities are so thin relative to local firefighting capacity, however, intergovernmental cooperation has been needed even in the case of federal lands and property.53
Compared to its focus on the fire problem, federal financial support has been substantially larger for EMS over a much longer period, beginning with the EMS Systems Act of 1973. Congress annually appropriated funds specifically to support the development of regional systems of trauma-related care until 1981, when the budget for EMS Systems Act purposes was subsumed within the prevention block grant program for states. State governments continue to flow financial support to the EMS function with support from this block grant. The IOM’s 2006 study reported, however, that public financing for EMS capabilities declined following this 1981 change in federal policy.
EMS provided by fire departments is funded by public budgets, voluntary contributions, by reimbursements from health insurers, including the federal Medicaid and Medicare programs, and by payments directly from patients whose insurance will not cover claims. Ambulance transport services are paid for on a schedule of fees. Such fees vary, some charging flat rates, others charging by mileage too, while some even charge based on the number of vehicles sent and time on-site. Some public EMS-only services publish their own fee schedules and bill patients whenever they provide the service.54 EMS functions usually provide more than just simple transport, and some provide Advance Life Saving support that can include real clinical treatments. A good deal of such clinical effort is not reimbursable by payers such as Medicare. All the un-reimbursed costs associated with such services and use of supplies and equipment must be financed by other means. For some stand-alone services, like those in Austin, Texas, about one-third of the costs need to be financed by public or voluntary resources.55 On the other
52 For a discussion of the early history of the fire insurance industry see: Fire Insurance in the United States, Dalit Baranoff; EH.Net Encyclopedia, edited by Robert Whaples, October 1, 2004, found at: http://eh.net/encyclopedia/article/Baranoff.Fire.final 53 National Academy of Public Administration, December 2003. 54 For more information, see the Little Rock, Arkansas, Metropolitan Emergency Medical Services (MEMS) at http://www.metroems.com/ 55 See the Auston-travis County Emergency Medical Services website found at: http://www.ci.austin.tx.us/ems/ “The department has a staff of 379, a budget of $27,815,000 and collected revenue for direct services of $17.7 million last year.”
58 hand, Toledo, Ohio, is an example of a locality where EMS, organized as part of the fire department, is seen as a revenue-generator for the department.56 We also lack a clear picture of how much is paid by public means such as through direct taxes, appropriations of general revenues, and reimbursement from public programs billed for services. The wide variety in delivery models for EMS makes it difficult to estimate public and private expenditure levels for the EMS in the U.S.
Overall U.S. public expenditures for fire protection have been on a steady rise for more than 30 years, far exceeding the rate of U.S. population growth (see Figure 4-1). Expenditures for EMS are not separately tracked by the Census or anyone else, so it is not possible to show a similar chart for EMS alone.57 But note, fire-based EMS expenditures are included in this chart, as the Census collects data by organization; and fire operations cannot be disaggregated from the EMS- related operations. The growth in EMS demand may be one of the reasons Census data show a steady steep growth of expenditures for fire protection over the entire period.
State and local spending for this function dwarfs the financial contributions of the federal government by way of the AFG program. In fact, the growth of state and local spending since the AFG program was enacted—which averaged $1.1 billion during 2001-2004—exceeds the level of annual appropriations for the federal program. AFG awards made in federal FY2004 were $675 million, while local governments managed expenditures of more than $28 billion. The additional economic value of volunteers who work within the fire and fire-based EMS operations may actually exceed known public expenditures, with estimates ranging as high as $37 billion annually.58 These additional values are not captured in Census data. Combined, this suggests that the resources managed by organizations eligible for the AFG program may exceed $65 billion annually—more than 100 times annual AFG grant amounts. The provision of fire protection and much of the EMS infrastructure and operations in the U.S. is primarily supported by non-federal revenues.
56 Troy, 2006. 57 Non-fire-based, public expenditures for EMS cannot be isolated either, as these are subsumed within the Census figures for health care delivery, and these cannot be disaggregated. The Census of Governments does not identify private expenditures for EMS or transport-related ambulance services. Studies estimate that 16 million people arrive at emergency departments annually and that there are around 800,000 EMS personnel in the U.S. 58 This calculated figure comes from the Blue Ribbon Report (2004). It may be on the high side. Most volunteers contribute their time without compensation and many also pay for their personal equipment and for much of their training. The calculations used in estimations like this are very sensitive to the assumption used about how much of a volunteer’s effort equates to full-time-equivalent (FTE) employment in a similar job. Volunteer practices are known to vary widely across the U.S. Compared to career-based organizations more volunteers are needed to fill a FTE position and organizational approaches also differ because of the very nature of volunteerism. The skill sets and training levels for volunteers also vary widely across the U.S. with many on a par with career employees, but a substantial number not.
59 Figure 4-1. State and Local Government Expenditures on Fire Protection and AFG Contribution, Compared to U.S. Population Growth
State and Local AFG Contribution 30 U.S. Population 300 28 280 26 260 24 240 s s 22 220 20 200 18 180 16 160
14 140
12 120
10 100 Population in Million 8 80
Expenditures in $ Billion 6 60
4 40
2 20
0 0 1970 1971 1972 1973 1974 1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 Year
Source: Census of Governments, U.S. Bureau of the Census
The AFG program is not the only direct federal support for improvements in fire and EMS organizations. Table 4-1 below lists some other major DHS programs whose funds may be available to support local fire departments and EMS operations. For FY 2004, DHS reported to the Congress a total of $3.67 billion in federal assistance made available for related purposes. Fire departments and EMS functions could be the recipients of funds from all these programs, often via sub-grants by state and local governments. AFG amounts were approximately 20 percent of the DHS total in FY 2004.
60 Table 4-1. DHS Funding for Homeland Security and AFG Federal Fiscal years 2002-2007 (dollars shown in millions)
Total FFY02 FFY03 FFY04 FFY05 FFY06 FFY07 2002-07 Homeland $2,206.90 $1,490.78 $960.30 $919.06 $5,577.04 Security Grant Program Eligible Funding EMS Portion $102.40 n/a n/a n/a $102.40 Urban Area $720.70 $829.66 $710.62 $746.90 $3,007.88 Security Initiative Eligible Funding EMS Portion $30.60 n/a n/a n/a $30.60 AFG Eligible $336.50 $707.20 $745.60 $680.00 $616.03 $628.90 $3,714.23 Funding (all program elements) EMS Portion $3.10 $4.60 14.00 n/a n/a n/a $21.70 DHS Total $336.50 $707.20 $3,673.20 $4,726.90* Eligible Funding $336.50 $707.20 $3,673.20 $4,726.90 EMS Totals $3.10 $4.60 $147.00 $154.70
Source: This chart is reproduced with clarifying edits from: Support for EMS Provided by the DHS Office of State and Local Government Coordination and Preparedness, A Report to the Committees on Appropriations of the United States Senate and House of Representatives, February 2005, p. 3. This report to Congress did not separate out the funding provided for fire departments for non-EMS purposes.
* The total represents the amount available; but at the time of the report DHS had only awarded $320.3 million of this amount.
In the wake of disasters, the federal government’s general policy toward the states has been one of cooperation and collaboration, partly because of the limited delivery capabilities of the federal government and also because the federal government may not be legally empowered to act in the same way as state and local officials can. State and local authorities have rather extensive powers to act in the case of emergencies and community health and safety, a legal situation characterized by the term the “police powers.”59 Large-scale disasters such as Katrina challenge these traditional assumptions for some officials and demonstrate the need for the federal government to take more direct leadership for organizing both response and recovery and preparedness efforts.
59 The term comes from the Greek word “polis” and refers now to political authority however derived. The U.S. Supreme Court has viewed the scope of this for state governments to be quite extensive (and much broader than just law enforcement). For a description, see: http://en.wikipedia.org/wiki/Police_powers
61 If the federal government wants to accomplish certain national objectives, as a practical matter it must cooperate with and seek to influence the local governments who deliver and mainly fund fire and EMS operations. State and local governments generally welcome federal assistance and cooperation to address their fire problems, EMS system shortcomings, and other risks that face their communities. The numbers of grant applications to the AFG program indicate its popularity and importance.
State and local leaders may be less welcoming of federal regulatory efforts to steer them toward national objectives. The AFG program remains a competitive grant program where applicants decide whether or not to apply. It is not intended to regulate fire departments, but the competition for funds provides an opportunity for the federal government to offer incentives through its prioritization of awards. To the extent that the federal program were to use this means to steer local spending toward national objectives, its ability to influence local priorities would be limited by its modest funding relative to the total resources provided for these functions by others. For the same reason, the program’s potential leverage on local priorities will be greater in those places that have a greater need for additional funding.
THE CURRENT STRATEGY FOCUSED ON BUILDING RESPONSE CAPABILITIES
The recommendations of this study take into account the Panel’s understanding of historic patterns and the basic parameters of the federal government’s involvement with fire departments and non-affiliated EMS organizations to date. The AFG program’s administration likewise has been respectful of the history and culture of fire and EMS provision in the U.S.
With strong input from fire services representatives, the AFG program has been focused primarily on building the capabilities of fire departments to respond to a fire. The implicit logic behind this program focus is that helping local fire organizations become more capable at fighting fires will reduce the number of public deaths and injuries from fire incidents as well as the number of firefighter deaths and injuries. Figure 4-2 portrays the basic logic of the federal program as operated from its inception. Only a small percentage of AFG grant dollars can be said to support fire prevention and safety activities; few of the dollars go toward EMS provided by fire departments.
62 Figure 4-2. Basic Fire Related Logic Model—Firefighting and Response
Inputs and Process Changes Outcome Goals
Intermediate Intermediate Results Results 1. Reduction in Firefighter Deaths Increases 1. Improves Fire and Injuries Increased Firefighting Response Spending Capability 2. Improves Firefighting Effectiveness 2. Reductions in 1. Firefighting Equipment Economic Loss 2. Personal Equipment 3. Reduction in Public 3. Personnel Training Deaths and Injuries 4. Other Operations & Safety 5. Fire Department Vehicles
A similar logic model can be deduced for the EMS aspects of fire departments. Activities supporting EMS operations based in fire departments have been eligible for grant support from the beginning. Capabilities more related to incidents that do not involve fire, such as hazardous materials exposures, can be addressed by the program.60 In 2004, Congress made non-affiliated EMS organizations separately eligible for AFG grants but capped the amount of awards to these independent EMS operations at no more than two percent of the amounts available. Beyond this set-aside, fire-based EMS purposes have not attracted many applications. EMS garnered awards amounting to less than one percent of the funds initially, and received only two percent in 2004.
Congress mandated that DHS conduct two assessments in conjunction with the NFPA. These assessments were intended to provide a description of what roles fire departments serve and what capabilities they possessed. Information from the assessments was intended to help the AFG program adjust funding criteria for the program to better address needs for improved capabilities.
A survey instrument of more than 40 questions was sent to thousands of fire departments (15,545 in 2005) asking them primarily about the fire problem but also their other roles, such as EMS, hazardous materials response, rescue, and infectious disease. Some questions were derived from accepted standards. For instance, departments were asked if they had a program of health and fitness as required by the NFPA 1500 standard. A low percentage of organizations responding positively to a given question would suggest that this is an area of under-investment. The assessments highlighted several capability areas that were not well served yet across the U.S., for instance:
60 DHS, AFG Program Guidance, 2006.
63 • “Almost no departments have all those [EMS] personnel certified to the level of Advanced Life Support.”
• In cities of at least 50,000 population, the NFPA minimum of having four trained firefighters available for first-unit arrival is slightly less now than estimated in 2001.
AFG funds are available to support all of the areas touched by these “needs assessments.” Success for the program might be measured, in these terms, by improvements year to year in the percentages of fire departments reporting basic capabilities. The second assessment done in 2005 reported some improvement compared to the 2001 survey.61 An analysis performed by the NFPA comparing the two years’ responses indicated that a substantial number of AFG awardees were included in the group of respondents to these surveys (15,541 surveys were sent out in 2006 and from within this group more than 10,000 grant awards were identified).62
While AFG funds must have contributed in some way to the improvements observed between the two assessments, it is not possible to quantify this contribution. In the initial four years of the program, including the awards made in federal FY 2004, the grant awards amounted to more than $1.7 billion. Coincidentally, building off the data from these assessments, some very rough estimates of the value of the improvements indicate that these may exceed $1.2 billion during a period from roughly the end of 2001 to the end of 2005). Table 4-2 below shows that AFG funds are at least of the same order of magnitude as the costs needed to produce the largest reported improvements.
To put the AFG program’s potential contribution to the observed improvements in perspective, however, over the four years covered by these assessments the AFG program contributed less than two percent of all public funds for fire protection in the U.S. Do the improvements revealed by the needs assessment simply stem from the general upward trend in the budgets for fire departments, or from increased funds coming from a number of sources including AFG funds?
61 NFPA, February 2006. 62 Department of Homeland Security and National Fire Protection Association (February 2006). Matching Assistance to Firefighters Grants to the Reported Needs of the U.S. Fire Service, A Cooperative Study Authorized by U.S. Public Law 108-767, Title XXXVI.
64 Table 4-2. The Value of Changes in Fire Departments, NFPA Assessments of 2001 and 2005*
Estimated Cost Calculation of the Estimated Value Improvement Total Over Four of These Improvements* Years 25,000 more firefighters were certified Additional firefighters with over the 2001 assessment. Valuing the $75 million basic training costs of their training and certification processes at $3000, covering all expenses. 55,000 more firefighters have access to health and fitness programs over 2001. Firefighters able to access Valuing the costs per year of health and $110 million health and fitness programs fitness memberships at $500 per year and assuming that this level of improvement was constant for each of the four years. 16.9 million more people are now served. We will assume that this adds 200 FTEs of More of the population specially trained individuals at a cost of served by the review of $80,000 per year—or $16 million per year. $45 million building plans for fire code Since this is a large change we do not purposes assume that this happened all at once the first year. An overall estimate is provided. 11.6 million more people are now served. We assume that enforcement of testing More of the population requirements adds 100 FTEs of individuals served by the testing of who coordinate the activities at a cost of $15 million sprinkler systems in $60,000 per year—$6 million per year. buildings We estimate a potential value of this over the four years assuming a more gradual adoption of the function during the period. 9.9 million more people are now served. More of the population We will assume that this adds 100 served by approval specially trained FTEs at a cost of $80,000 $30 million processes for building per year—or $8 Million per year. We permits estimate a potential value of this over the four years.
65 Estimated Cost Calculation of the Estimated Value Improvement Total Over Four of These Improvements* Years 16.5 million more people live in jurisdictions that have such programs. We estimate that the coordination of these programs requires personnel time that may More of the population cost $10 million per year. The costs of the served by smoke alarm $100 million smoke alarms and installation and testing programs of them, though donated in many cases, still probably amount to $20 million per year for some 200,000 smoke alarms. We estimate the costs for the four year period. 1500 more stations have backup power. Added stations with backup $1200 estimated cost for the equipment $1.8 million power and its installation. 3000 stations obtained such systems. Added exhaust systems for $4000 estimated cost of the design, $12 million fire stations equipment and installation of the exhaust systems. Estimate over four years shown here uses Added vehicles the federal vehicle grants and then adds $532 million costs for the recipient shares. 9% more departments now have sufficient numbers of radios. Assume this is 3000 Added radios $9 million organizations and the radio costs are $3,000 per organization. 8% more firefighters have access to SCBAs for use on their shifts and many were replaced to bring the number of SCBAs with an age greater than 10 years Added SCBAs $200 million at 25% fewer. We assume this adds 50,000 SCBA packs at a cost per back with training and miscellaneous related gear of $4000. 12% fewer firefighters are lacking a PASS Added PASS devices device per shift. Assume this adds 50,000 $75 million new PASS devices at $1500 per unit. 18,000 more firefighters can suit up in Added personal protective personal protective clothing. Estimate the $54 million clothing costs of this with training and miscellaneous items at $3000.
66 Estimated Cost Calculation of the Estimated Value Improvement Total Over Four of These Improvements* Years 18% more fire departments have arranged access to the internet. Assume that this is 5000 organizations and that this requires a Internet access for fire personal computer at $1500 per unit $15 million departments ($7.5M) and an internet service fee of $400 per year ($2M per year). Overall an estimate is provided. 21% more FDs have these than 2001. More thermal imaging Assume to mean 5000 devices at $3000 $15 million cameras per device. Total Estimated Value of Improvements in the Four $1,287 million Years*
* The assessment report from 2006 is the source of the improvement items and the basic percentages. Improvements reported to have been of a small percentage are not included. Caution should be used because the unit value assumptions are only very rough estimates. The Academy study team performed very little fact- finding to justify the unit value estimates or some of the assumptions. The chart is useful as an illustration and more appropriate to reaching conclusions about orders-of-magnitude. Approximately, 97% of funds appropriated to AFG by Congress have been allocated to successful grant applicants.
Cost-effectiveness of the AFG Program
From a federal budgetary perspective, the AFG program competes with other programs partly on the basis of whether it is the most cost-effective use of limited budget resources. In this context, it is useful to ask: How cost-effective has the AFG program been as used to date? Even though the answer to this question cannot be quantified, it is still useful to consider how one would approach an answer as a way to understand the program’s current position.
From the standpoint of administrative efficiency, there is broad agreement among stakeholders and observers that the program has been well run. It is a positive case study in the management of a grant program by a government agency, although there is as always room for improvement (see Chapter 9).
The AFG program has reached almost all eligible organizations and in terms of sales-penetration has converted most to positive actions, with more than 19,000 applications being generated year after year. It appears, therefore, that most fire departments and non-affiliated EMS organizations actively look at the program offering every year, although many decide for a number of reasons not to apply. This level of market awareness is a positive result. In a commercial setting, such a level and receptivity for an offering, and such high measures of penetration and conversion to action, would be considered an overwhelming success, deserving of case study status in the literature on marketing and sales.
67 But efficiency in a marketing sense is just a starting point. One can be highly efficient at doing something that is unproductive or worse. Ideally, a cost-benefit analysis of the AFG program would assess the relative costs and benefits of alternative uses of these funds in different situations, perhaps characterized by differences in community risk profiles, relative to the measured returns from these uses over time to communities and to the nation. Although such an analysis cannot be conducted as part of this study, it is nevertheless useful to think of the program from a cost-effectiveness viewpoint.
Three main types of costs can be considered, though only two are significant here:
1. The costs of AFG to the federal government are the expenditures made for the AFG program. We can look at the level of appropriations as the measure.
2. Awardees or others contribute matching funds, which are a cost to them. These amounts are represented in the grant applications and are reported by the AFG program based on the award agreements. They amount to around 15 percent of the federal appropriations.
3. Applicants incur some costs in making an application and in performing tasks under the grants. These are relatively small. The real costs of performing under the grant consist of some increment not funded by the grant, such as increased maintenance of new equipment purchased with AFG funds. Considering that most grants are small, the additional unfunded performance costs may be significant for many applicants, but they are not large in the aggregate.
Benefits or effectiveness also must be estimated. It is easier to estimate the program’s costs than to estimate whether the program is effective. One first needs to specify in more precise, measurable terms what the program seeks to influence or accomplish. Only when there is some agreement on the goal or goals and how one can measure progress toward these can one ask whether the program’s costs appear reasonable in relation to benefits that is, does it help to achieve measurable improvement at a reasonable cost?
To date the program has set out to improve the capabilities of fire departments. Such capabilities are believed to contribute to improvements in public safety and firefighter safety. Funds are being used mainly to purchase equipment, vehicles, protective gear, and some training and other services intended to improve incident-response and to make firefighters safer. From this simple standpoint, the program is effective—it has improved the asset bases and skill sets of recipient fire departments and thus improved local firefighting capabilities. It is not as clear, however, whether such improvements reduce public and firefighter deaths and injuries, purposes mentioned in the authorizing statute.
Although these effects cannot be quantified, it seems clear that the capabilities-building strategy employed by the program to date has an influence on the deaths and injuries of firefighters involved in a fire incident. Much anecdotal evidence has been assembled about effective uses of the program to achieve such benefits.63 A firefighter who now uses a Self-Contained Breathing
63 See, for example, “Harry Carter’s 2006 Fire Act Roadtrip” at http://www.firehouse.com/interact/category/harry- carters-2006-fire-act-roadtrip/ .
68 Apparatus (SCBA) is plainly safer than when fighting a fire without an SCBA. A firefighter who is wearing modern turnout gear that properly fits is safer now than when he or she wore older gear. Thermal imaging cameras help to ensure that firefighters or residents are not unnecessarily lost in a large fire or unnecessarily risking their lives on rescue efforts because of visibility problems. While these are likely results of AFG program funding, the performance reports submitted by awardees have not been designed to provide direct evidence about subsequent incidents addressed by the organization using the funds they receive.
One view is that the AFG program has been successful in raising basic firefighting response capabilities in many places that, in the absence of federal assistance, would have been unable to replace aging vehicles and equipment or provide adequate training to their personnel. However, improved capability may or may not lead to greater fire safety. Absent additional data collection and a formal evaluation, the benefits of building these firefighting response capabilities must be inferred rather than measured.
The presumed relationship between improved capabilities and increased fire safety merits more scrutiny, especially given evidence cited in Chapter 3. One chart, for instance, indicates that firefighter injury rates have been increasing during the first years of the AFG program’s operation. Nonetheless, it appears that the value of the improvements seen in fire departments over this four-year period have been significant (see Table 4-2). However, evidence of improved capabilities does not make the case that the program has been optimally cost-effective in its current strategy, embodied in the logic model in Figure 4-2.
An argument for the cost-effectiveness of the program’s current emphasis on fire response capabilities may be made also in relation to concerns about economic losses from fires. The sooner a fire is suppressed and extinguished, the lower the property loss.64 As described in Chapter 2, the rate at which fire incidents occur declined substantially over three decades. Economic growth and the building code movement of the 1950-1970s were probably significant contributors to this decline, as older structures were replaced with better, newer ones. Recently, however, the average amount of property loss per fire has been increasing. If analyses were to show that slower or less effective response accounted for this adverse trend, then it might argue for making improved response capabilities one of the AFG program’s priorities. However, no such analysis has been located.
Regardless of the role of the AFG program to date in improving firefighting capabilities, the next question is whether other uses of the same funds could be more effective in improving public and firefighter safety.
64 The case for government intervention to reduce economic losses rests not only on their scale, but also on whether the private market permits property owners to insure against them. Where risks are not well understood, preventing the private market from offering property owners financial protection, governments may need to act. However, economic losses from fires are easily insurable in the U.S. because the nature of the risk is well understood. This makes insurance easy to price. Losses from fire remain large but the risks can be insured, weakening the case for a large-scale federal effort to reduce these losses.
69 As the AFG program matures, it would reasonable for it to change priorities for use of its grants. Its priorities change every year based on the input of its Criteria Development process where program stakeholders reflect on lessons learned about the program.
One reason to consider alternatives now is that a large number of the fire departments that have the fewest local resources available to them have received AFG awards to help them address their most critical basic needs. Further investments in those places may have reduced marginal benefit. Also note that some places may either have already met their basic equipment and training needs, or have the local capacity to do so without federal financial assistance. Further investments in these situations would have a greater probability of supplanting rather than augmenting local spending and charitable/volunteer contributions, reducing the program’s cost- effectiveness. But this reason for considering alternatives should not be overstated. AFG grantees need continuous training and their equipment does eventually wear out. For example, awards for equipment made in 2001 are 6 years old.
The evidence reported in Chapter 3 suggests declines in public deaths and injuries from fires and in the rate of firefighter deaths and injuries have leveled off recently. It is not clear why this change in trend has occurred, but it does suggest a need to re-examine current public policies. The Nation may have, in the aggregate, reached a point of declining marginal returns with its current policies.
The AFG program’s uses may need to change. The program’s strategy of improving firefighting response capabilities, however effective it is at doing this, may not represent the most cost- effective way to reduce either public or firefighter deaths and injuries.
One argument that has been made forcefully by experts on the fire problem over the last four decades is that dollars used to reduce the number of fire incidents are likely to have greater impact on fire safety relative to their cost than dollars used to improve response to fires when they break out. A series of national studies have made related points:
• In 1973, a Presidential Commission noted that, “currently, about 95 cents of every dollar spent on the fire services is used to extinguish fires; only about 5 cents is spent on efforts- mostly fire prevention inspections and public education programs-to prevent fires from starting (America Burning, 1973, p. 7).”
• In 1987, another national expert panel reviewed and updated earlier work on the problem and observed that “the overall demand for fire suppression services is expected to decrease in the future. However, the demand for fire protection will continue and possibly even increase as the effort is shifting to fire prevention and public education duties (America Burning Revisited, 1987, p. 15).”
• And in 2002, still another federal government report observed that: “The wisdom of these recommendations [referring to the America Burning report of 1973] was acknowledged by the Congress and the Administration in the enactment of the Federal Fire Prevention and Control Act of 1974 (the “1974 Act”). However, FEMA and the USFA have not pursued many of the preventive measures authorized by that statute; the
70 Congress has not appropriated the funds necessary to carry them out; they have not been adequately advocated by USFA; and if implementation is the test, they have not been widely accepted by the fire service-at-large. In addition, FEMA has not applied to the fire problem those lessons which it learned with respect to other natural hazards, including earthquake, flood, and hurricane and has failed to exercise all of its powers under the 1974 Act.”
While the recommendations of these study commissions covered a wide range of measures, they each gave great emphasis to increasing fire prevention efforts and safety education as neglected or underused strategies to reduce the number of public deaths and injuries, as well as improve the safety of firefighters and other emergency responders.65
Of course, public safety is improved by having fewer incidents. On the EMS side, efforts to educate the public about when to call for emergency services are already recognized in the strategic plans of more advanced fire departments.66 Fewer incidents and better public response to those incidents lowers the risks for firefighters and for EMS personnel as well as for the public.
Regardless of the effectiveness of the AFG program’s strategy of emphasizing fire response capabilities over incident prevention or incident control capabilities, are the costs incurred to do this reasonable in light of the expected returns? As reported in Chapter 3, the U.S. spends 0.25 percent of GDP on fire services, one of the highest reported proportions among industrial nations, even excluding the significant value of volunteer contributions, suggesting the current U.S. strategy may not be optimal.
An adequate answer to the question of how the AFG program could address public safety more cost-effectively can only be given after assessing relative returns from alternative uses of the grants and taking into account the influence of other resources dedicated to similar purposes. Right now it is not possible to establish definitively whether another program strategy would be more cost-effective. First, the AFG program must declare what performance measures are to be used and the goals for the program as a point of comparison. Performance reports from AFG grantees do not provide information allowing the program to link funds use to improved outcomes such as reduced deaths. Nor has an independent evaluation been conducted that would assess the program’s contributions while taking into account other influences on such outcomes.
A broader view of the program’s benefits and costs would include institutional and other effects that are harder to quantify but may result from program investments. Here are examples of some of the important benefits and federal interests in the success of the AFG program:
65 The shift to fire prevention and protection activities from a predominant focus on response-capabilities has been adopted in both the United Kingdom and in Ireland following major public studies: Local Government Association (2002); Farrell Grant Sparks (2002) (The New Zealand Fire Service’s shift in 1999 is highlighted in Chapter 6). 66 Tualatin Valley Fire & Rescue Strategic Plan, 2006
71 • Important organizations are improved and made more adaptable.
• Commercial marketplaces that serve these organizations are made more dynamic, leading to faster adoption of new technology and new practices.
• Social equity is improved.
• Intergovernmental cooperation is improved.
• Local EMS-fire service coordination is enhanced.
• Community support for fire and EMS services is increased.
In the long run, these less direct benefits of the program’s operation can be as important as direct benefits and therefore should be taken into account in a full-blown cost-benefit analysis and evaluation.
LEVERAGING COOPERATIVE RELATIONSHIPS
The 1973 America Burning report captured what may now be obvious. The federal interests in the improvement of fire department operations can best be served by augmenting rather than supplanting the traditional commitment of state and local governments to these functions. Especially in rural and smaller communities, federal interests may best be served by reinforcing rather than inadvertently diminishing traditional community support for volunteer and combined volunteer/career fire departments and EMS organizations.
In creating the AFG program, Congress limited the administering agency’s grant-making discretion to help ensure that the AFG program leveraged rather than supplanted local financial support. The statute requires that grants not be made by the agency if current financing is not being maintained or if the agency determines that budgetary allocations are being supplanted as a result of the grants. Less clear is how one would assess, or limit, supplanting of voluntary resources.
The AFG program, because of its matching grant provisions and these grant-making controls, would appear to accomplish one fairly common goal for federal grants; namely, it leverages additional resources that serve the same general purposes. The AFG program leverages more than financial resources, of course. It also helps to bring attention to the capabilities and activities of the fire departments and non-affiliated EMS organizations and how these relate to priorities established by the federal government.
In addition to direct funding support, AFG may increase total resources for fire departments by leveraging additional local and state funding or in-kind efforts, or by encouraging more effective use of existing resources. Without a rigorous evaluation, it is not possible to estimate the extent of influence and leveraging attributable to the AFG program. However, measures of performance success may be associated with increases in overall resources or increased attention
72 to federal priorities. This study was not able to conduct the type of inquiry needed to report about performance on these measures.
Any program or set of programs can increase its influence to the extent that it has communicated to other actors clear, measurable objectives. A clear sense of direction, based on an accurate understanding of the problems and the most effective strategies to address them, can have influence on the behavior of communities, public agencies, and private actors that can multiply the effectiveness of a given funding stream or pool of resources.
Decisions about direction and strategy are implicit also in how the AFG has been used to date. Each year the AFG program sponsors a criteria-development workshop where it seeks the input of a number of stakeholders concerning the priorities for funding. This has led to slight changes in the program’s emphasis from year to year. For the most part the program has stayed on track to fund the building of the type of capabilities needed for fighting fires. The most recent changes to the program’s priorities have, however, reflected the location of the program within a department with a broader mission.
Aligning to the All-Hazards Policy of DHS
The DHS report spotlighted in Chapter 3 provides a clear description of the nature of the U.S. all-hazards preparedness and the inherent difficulty of synchronization across the levels of government and other sectors:
In the United States, planning responsibilities are decentralized and divided among levels of government. Hurricane Katrina demonstrated that catastrophic events require fully integrated intergovernmental actions and combined capacities. Two competing imperatives—decentralization and synchronization— converge at the point where plans are most likely to break. Phase 1 analysis suggests that current nationwide planning does not adequately address synchronization for catastrophic events. Synchronization is more than coordination—it is both a process and an effect. As a process, it identifies gaps and inconsistencies in plans before, rather than during, the event. As an effect, it ensures intergovernmental actions and capabilities are arranged to produce the desired effect—whether evacuation, search and rescue, or the provision of mass care—at the place and time and in accord with the purpose prescribed by our combined plans.67
The DHS report’s conclusion is that there are four essential areas for improvement. The Academy has found that the AFG program can improve all four areas if focused, managed and evaluated in a way consistent with the recommendations in this report:
• Need to employ common planning magnitudes for catastrophic planning.
• Need for shared national planning that ensures the adequacy and feasibility of our combined plans by emphasizing their synchronization.
67 DHS, Phase 1 Report, 2006, p. 14.
73
• Need to strengthen the linkage of federal programming, planning, and budgeting and grant funding to operational needs. Plans represent the point where these resources and those of states, territories and urban areas converge and are translated into action.
• Need for a national measurement system that provides accountability by periodically assessing and reporting the adequacy and feasibility of the nation’s plans, and focusing national preparedness (e.g., training and exercises) on developing the collective proficiency and capacity required by our combined plans.
The approach recommended by this report is to let communities determine their own biggest threats and then use programs such as AFG to fund resource shortfalls related to the highest priority risks and to facilitate synchronization and regional planning.
The best example of new AFG funding priorities is the one established to encourage proposals based on a regional collaboration. The program is, for instance, funding equipment and related project costs associated more and more with the ability of organizations in an area to telecommunicate in an inter-operable fashion. Large-scale disasters are almost by definition regional concerns. The AFG program has already articulated a position of favoring applications that are focused on regional agreements about the use of the funds.
Community-level coordination is expected and common for most of the incidents addressed by fire departments. Fire departments must coordinate their actions with those of other first responders, ranging from major traffic accidents to large-scale acts of terror. Many incidents require coordination with law enforcement. EMS incidents require communications with health care delivery organizations and specialized dispatch functions. Traffic crashes may involve hazardous materials. Any significant incident, particularly a large disaster, also triggers mutual aid actions involving other jurisdictions, and possibly states and the federal government too.
Most fire departments and non-affiliated EMS organizations find their 21st century roles to be “all-hazards” in nature. Hence it is no surprise that the protocols for incidence-response now reflect this viewpoint almost everywhere. Indeed, leaders of the fire service assisted in the reformulation of fire-based command approaches into the National Incident Management and Incident Control System standards adopted into the Homeland Security Presidential Directives. The AFG program began in 2005 to require all grantees to train their personnel on those standards. In early October 2006, Congress enacted changes to the organizing statute for DHS that directs it to align all of its efforts to an all-hazards perspective of large scale disaster risks within the U.S. (P.L. 109-295, Homeland Security FY 2007 Appropriations).
It is a good time for the AFG program to consider whether to retain or further adjust its focus on the building of firefighting capabilities to ensure that the program best fits the reality facing fire departments of various sizes and situations across the U.S.
As noted in Chapter 3, the Department has already moved forward with an all-hazards view in developing the overarching National Preparedness Goal announced by the President in HSPD-8. In an all-hazards context the AFG program’s role is to offer financial support related to fire-
74 related public safety covering fire operations, EMS, and the other functions addressed by fire departments and the non-affiliated EMS organizations eligible under the statutes. Its new priorities for regional collaborations, for interoperable communications, for CBRNE, and for hazardous materials response are consistent with the broader all-hazards concerns. However, the AFG program’s funding currently is focused mainly on the fire problem and on the “response” part of the taxonomy. Applications can be submitted to AFG for non-response purposes and for most of the roles being served by fire departments. It is a Department-level responsibility to specify how each of its programs aligns with its mission and statutory purposes and to help set priorities for the use of AFG funds. The AFG is one of the programs or puzzle pieces the Department must fit together in addressing the national preparedness goal, as represented in Figure 4-3.
Figure 4-3. The AFG is one of the programs, or puzzle pieces, the Department must fit into its most appropriate place
Communities interested in improving their ability to address fire-related risks will turn to AFG as an obvious source of federal support. The program and the Department will need to determine the best fit of the AFG program with other programs, other puzzle pieces, by defining the best potential uses of AFG. It will want to ensure that applicants for funds have a clear understanding about the piece they need to pick up when they want federal aid for some local initiative.
Other Areas of Federal Interest: Commercial Markets, EMS, Social Equity
An all-hazards viewpoint encompasses hurricanes, floods, earthquakes, fires, hazardous materials, and other hazards that are now seen to be an important federal concern. Related to the fire grant program, there are a number of other special areas deserving of federal interest.
Depending on how it is used, the AFG program can reinforce the federal interest in having a dynamic and responsive commercial marketplace serving fire departments and the affiliated EMS organizations. Large-scale disasters, especially, require dynamic and responsive
75 organizations. It is clear from industry’s own surveys and from study interviews that the commercial marketplace is also innovating in its offerings and business practices, adapting to the new demands. AFG funding of local fire department purchases makes for a larger market for new and more modern equipment and services than was the case prior to 2001. Historically, as shown by the regular, slow advance of the expenditure levels found in Figure 4-1, fire departments presented a weak market that attracted few entrants and new product and service offerings—lots of small buyers, little dynamic growth or competitive pressures, and not much change. It is no surprise that firefighting equipment and EMS equipment, for instance, stayed much the same for some time. Homeland security concerns require a much more responsive, flexible, dynamic, and improving set of buyers. There is a clear federal interest in continuing to encourage both the supply and demand sides to innovate and improve. AFG funds can play a positive role with regard to fire departments and non-affiliated EMS buyers of commercial services, products, and goods. A bigger or more efficient market serves the country’s interest in improving capabilities to the extent that it supports more rapid improvement in and adoption of new, more cost-effective technology and training. It can be argued that the AFG program’s competitive grant process reinforces the competitive dynamics that support accelerated progress.
BALANCING FEDERAL WITH COMMUNITY INTERESTS
The AFG program represents a relatively new federal commitment in an area that traditionally has been and largely continues to be a community responsibility. The balance of local and federal interests with respect to the capabilities of community fire and emergency medical services seems to be changing as the public looks to the national government for leadership in preparing for and responding to large-scale disasters and homeland security threats.
One of the most important federal interests providing context for the AFG program and its choice of strategy and performance measures is the program’s potential contribution to the Nation’s preparedness to deal with catastrophic events. From this perspective, increasing the capabilities of firefighters, EMS personnel, and other first responders is a critical component of national preparedness.
The federal government also has a general interest in promoting voluntary and charitable efforts. Preserving and encouraging such efforts, it has been shown, are essential to the delivery of fire and EMS services in the U.S. Likewise, the federal government has a general interest in promoting intergovernmental cooperation, which is essential to the success of the AFG program. The AFG program also supports a general federal government interest in promoting a competitive market of commercial suppliers of emergency and fire equipment and training, helping to promote adaptability, and fuel innovation, change and improvement in technology and in practices.
Already intense pressures on the federal budget are expected to mount as the costs of entitlements such as Medicare and Social Security rise with the retiring of the baby-boomers. All federal spending programs will be pressed to demonstrate that they serve important national interests, they know how to use funds well, and they can measure performance success in a credible way. In the context of these pressures, it becomes even more important to operate AFG
76 in a manner that does not supplant but complements and reinforces responsibilities of the fire departments and non-affiliated EMS organizations while helping to steer local investments toward capabilities that also address the national interest in preparedness for large-scale events.
77
78 CHAPTER 5 ALTERNATIVE VIEWS OF THE PROGRAM
People with knowledge of the AFG program hold a wide range of views, reflecting varied interests and perspectives. They point in a number of possible directions, but taken together they suggest answers to important strategic questions about how the program can best pursue its goals.
This chapter first summarizes various program stakeholder views, including those in local fire services and officials of the general purpose state and local governments, as well as those of experts in federal agencies and non-governmental organizations. They provide different perspectives on the program’s operation and priorities to date and possible future strategic directions for the program. They identify potential obstacles and opportunities affecting the program’s management ability to achieve its goals.
These views are of interest in themselves, capturing a great depth of knowledge and a variety of experience with the fire problem and other hazards and with fire and EMS service administration. They also have helped highlight potential disconnects or tensions between the program’s operation to date and the changing environment in which it operates. . The Panel has reviewed a series of major national studies of the fire and EMS problems in the U.S. (see Table 5-1). Together these illuminate the evolution of the problems, as well as a changing understanding of their nature. A common element of fire studies dating back almost 60 years is the need for more attention to fire prevention. Two studies of emergency medical services highlight the growth and evolution of these services over the last 40 years. These reports contain numerous recommendations for public action. Many of the same themes were echoed by the experts and distinguished professionals interviewed for the present study.
79 Table 5-1. Major National Studies of the Fire and EMS Problems
The President’s Fire Prevention Conference of 1947 Washington, DC (May 1947)
The President’s Fire Prevention Conference of 1947 was attended by a broad-based group of at least 2,000 individuals representing of business, industry, government, military, higher education, and the fire service, in Washington, DC. This was the first time a President of the United States called a meeting to examine issues pertaining to fire prevention. Their report coined the expression “3 Es,” which stands for Engineering, Enforcement, and Education, a model still used in dealing with fire safety today. However, memory of the report itself faded until it was “rediscovered” by staff of the U.S. Fire Administration in the 1990s. Accidental Death and Disability National Academy of Science, National Resource Council (1966)
This report reviewed the status of initial care and emergency medical services available to victims of accidental injury. These included ambulance services, voice communication systems, emergency departments and intensive care units of hospitals, and others. Its recommendations included forming an executive level council to serve as a clearinghouse for information on accident prevention, standardized training for emergency workers and ambulance equipment, enhanced emergency communication, and other policy changes. America Burning National Commission on Fire Prevention and Control (May 4, 1973)
This report was the result of a two-year study by a 24-member commission appointed by President Richard Nixon. Recommendations emphasized fire prevention through the implementation of local programs, built-in fire safety measures to detect and extinguish fires, training and public education, buildings and materials design standards, and research and development. This is the most widely quoted national study of the fire problem, credited with raising national awareness of the need for fire prevention and setting the stage for the creation of the U.S. Fire Administration. America Burning Revisited U.S. Fire Administration and Federal Emergency Management Agency (November 30-December 2, 1987)
The purpose of this 3-day workshop was to review the progress made to combat the fire problem and to redefine new strategies needed to further reduce loss of life and property to fire. The conference was aimed at reviewing the fire program and setting priorities for future direction and activities. Although no major problems were identified, the report points out that there has been significant changes in the fire environment, including expanded fire department responsibilities, increased concern about fire fighter health and safety, and awareness that public fire education needed to be improved, particularly for high- risk populations.
80 America At Risk: America Burning Recommissioned Federal Emergency Management Agency (June 2002)
This report builds on the work of America Burning. It is based on statements, discussions and “Principal Findings and Recommendations” issued on May 3, 1973. This report reaches two major conclusions: • The frequency and severity of fires in America is a result of our nation’s failure to adequately apply and fund known loss reduction strategies. • The primary responsibility for fire prevention, suppression and action on other hazards dealt with by the fire services properly rests with state and local government.
A Needs Assessment of the U.S. Fire Service Federal Emergency Management Agency (January 2003)
This assessment was mandated by Congress in P.L. 106.398 to help identify strategic needs for federal assistance to state and local communities in the fire service and emergency response activities. The study examined existing funding sources for fire departments, personnel capabilities, facilities and equipment needs, communication demands, and other trends for small and large jurisdictions. The study found that many fire departments—particularly volunteer companies in rural communities—report shortfalls in facilities, equipment and training of personnel. The findings are based on information obtained through surveys, a technical advisory group, and from other experts in the field of fire protection.
Emergency Medical Services: At the Crossroads Institute of Medicine (2006)
This report examines current trends in the provision of Emergency Medical Services (EMS), the system’s design and current resource allocations. It points out that fire-based systems across the U.S. are in transition as fires decrease while the number of EMS calls increase. Some of the areas of analysis include coordination between EMS providers, disparities in response times, quality of care, readiness in case of disasters, professional ethos among EMS workers and routine EMS medical practices. The report advances a number of recommendations for workforce training, research, development of national performance standards, and more. Specific recommendations are that Congress establish a federal lead agency for emergency and trauma care and that Centers for Medicare and Medicaid Services should evaluate the reimbursement of EMS and consider issues such as inclusion of readiness costs and permitting payment for emergency services without transport.
STRATEGIC QUESTIONS
Stakeholders hold a range of views about how the AFG program can maximize its effectiveness, reflecting their diverse perspectives on the problems the program seeks to address.
What is it that the program wants to influence or accomplish in order to move toward its goals? What outcomes would constitute success? What strategies would most effectively influence those outcomes, and how does one determine or measure accomplishments? What path is the AFG program on now? Is this the right path? What might the program look like in the future? These and other strategic questions have framed the study’s research and its efforts to identify and understand the views of the stakeholders.
81 Views of Stakeholders
Many individuals, representing the AFG program’s stakeholders, were interviewed in the course of this study. They were asked to give their perceptions of the program, how it has worked to date, and what visions and specific suggestions they had for the program’s future direction. These interviews, combined with a review of program documents and analysis of other reports and evidence, help put the program in perspective relative to the historical and other contexts.
The statute (P.L. 106-398) that authorized the AFG in October 2000 listed its goals as “protecting the health and safety of the public and firefighting personnel against fire and fire- related hazards” and listed a set of eligible uses. One set of purposes largely relates to the capabilities of fire departments to respond to incidents with new vehicles, apparatus and personal equipment.
Another set of purposes cover non-response or pre-incident activities, offering funds for applicant projects to:
• “train firefighting personnel”
• “establish wellness and fitness programs for firefighting personnel”
• “certify fire inspectors”
• “enforce fire codes”
• “educate the public about arson prevention and detection”
• “fund fire prevention programs”
Five years of experience with the program has expanded applicant, program staff, and stakeholders’ insights into how the AFG program can be used to protect health and safety and support other national purposes. In the course of our study, we encountered a wide range of views about how the program could be aimed to best accomplish its statutory purposes.
The matrix that follows presents a summary of the range of views (see Table 5-2). Although many of these can be attributed to a particular class of stakeholders, e.g., those in the fire service or city managers, we have organized the spectrum of views expressed to highlight the main differences about how the program has been and should be operating.
In reading the matrix, one should not attribute a particular viewpoint to all members of a stakeholder class, even where labeled as such for convenience, but rather recognize that the members of some classes are more likely than others to hold a particular view. But also note that the matrix does not in all cases align with particular groups of individuals as defined by their organizational position or by geography. Each of these viewpoints is summarized on the left- hand side, followed by a commentary on some of its implications for managing the program, and how one might be able to define and measure success for this view.
82
The discussion following the matrix may give readers a sense of how easy, or difficult, it is to align the AFG program with the many stakeholder views while remaining consistent with the statutory directions. Stakeholders’ views often overlap, and this can provide a common ground on which the program can build. The Academy Panel’s recommendations are intended to reconcile competing visions and interests as much as is practicable.
83 Table 5-2. Stakeholder Matrix: Viewpoints, Alternative Visions, Measures of Success
Stakeholder Viewpoint #1: Urban Fire Departments Perceptions of Problem and Program Resulting Visions for AFG Defining and Measuring Success Population density and population numbers mean AFG should target its funds more to urban and Population coverage more people can be served per grant dollar. This is suburban areas and their capability needs to cover cost effective and targets homeland security risks more persons with better capabilities. Decline in public deaths and injuries better while having a positive impact relative to the program’s statutory goal of reducing the number of This vision may complement views related to Accomplishment of the Universal Task List public deaths and injuries from fire, and from other prevention and public education, those related to performance measures in the high risk areas large-scale fire-related disasters and incidents. social equity concerns, and an all-hazards mission. Leverage—Increasing total public investment in the eligible organizations
Increase in the number of applications from high- risk areas Stakeholder Viewpoint #2: Rural Fire Departments Perceptions of Problem and Program Resulting Visions for AFG Defining and Measuring Success Rural areas have the least fiscal capacity and this AFG should target its funds to rural areas so it Improvement in basic statistics for questions used means their capabilities will always lag urban and supports areas of greater need and inadequate local in the NFPA needs assessment (does not have to suburban areas unless additional investments are resources (jurisdictions with weaker capability and be grantee-specific improvement) made. The plain facts of rural distances and these less fiscal capacity). capability gaps place persons and important parts of Increase in the number of organizations who the country at a higher risk from fire than is This view is often combined with the view that the meet the Universal Task List performance responsible. program’s goals should be focused on improving measures basic firefighting response capabilities. Volunteers are becoming more difficult to recruit, and Improved statistics on volunteerism (e.g., the nation needs to recognize how this could weaken Could be combined with the prevention and properly trained volunteer hours increase) the capabilities in rural areas unless mitigated by education efforts to improve the public abilities to additional investment. prevent and then respond to fires and other all- More organizations meet coverage standards hazards concerns, and also to assist with the problem comparable to those for suburban or urban areas Rural areas deserve comparable coverage, which they of declining volunteerism. can only meet with additional investment. Social equity concerns apply here too, as rural areas Rural coverage and capabilities serve as backup for have death and injury rates per unit of focus that urban and suburban areas whenever a large-scale exceed those in areas of denser population
84 disaster occurs and this is possible only if more investments are made in rural as well as urban areas.
Firefighter deaths and injuries can be reduced by maintaining funding for rural areas.
Volunteer firefighters contribute to negative statistics more than needed because of under-investment in their training and equipment.
The ratio of public death and injury to population is higher in rural areas than in other areas, requiring more investment in rural areas. Stakeholder Viewpoint #3: Prevention and Protection Efforts are Most Important Perceptions of Problem and Program Resulting Visions for AFG Defining and Measuring Success Public Education, prevention programs, engineering The AFG program should give priority weighting to An increasing number of applications for such and mitigation or code enforcement activities and applications for Operations and Support that would projects can demonstrate increased attention to other non-response activities have a greater positive fund such efforts and ensure a better balance of these purposes. effect on public deaths and injuries. funding, reaching a 50-50% balance in the future. Public attitudes, levels of awareness and The data suggest that continued investments in A community’s risk assessment may describe such education can be surveyed to show positive response capabilities may have a declining marginal efforts in a way that fits best for a region and improvements. National surveying should return in terms of public deaths and injuries; if true, it program funds should be available to support the support local and regional needs for comparative is time to alter the course of the program. A lot of community’s consideration of these prevention and data and the coordinating of opinion-polling federal funds have already gone toward the response protection aspects during the assessment process. efforts. part of the National Preparedness taxonomy. Department grants and contracts involving more Grantees should report instances where public The fewer the number of calls, the lower the risk for nationwide efforts should be expanded and responders were involved in incidents. The both the public and for firefighters and EMS coordinated with the competitive grant process. increasing level of press stories may indicate personnel. successful reach and conversion. This view is similar to the one about emphasizing Public education can prepare the public and public safety (see below). Public death and injury rates should be individuals to be better first responders themselves comparatively studied to determine whether with very positive influence on the public’s health grantee projects are valuable compared to non- and safety (this can be an all-hazards view). grantees, or to compare one type of activity to another.
85 Stakeholder Viewpoint #4: All-Hazards View Perceptions of Problem and Program Resulting Visions for AFG Defining and Measuring Success New legislation re-defines the Department’s mission AFG funds use should reflect and serve overall goals Number of organizations who can meet the to ensure that its programs/efforts all align to all- and strategies pursued by DHS. The Department Universal Task List performance measures hazards view of large disasters. should clarify how AFG fits into broader strategy. Number of community risk assessments Eligible organizations need to fit within an all- AFG funds should not be awarded unless the involving the organizations eligible for the AFG hazards policy within their communities; investments applicant demonstrates consistency with regional program that help them to do this also contribute to their plans and agreements related to risks from an all- abilities to respond to fire-related incidents. hazards perspective. Improvements in mutual aid agreements and regional collaborations (numbers of agreements Regional collaboration should be a priority for Applicants can be funded to lead in building and supportive content of the agreements as funding, as large-scale disasters by their nature regional capacities as defined by each region’s all- measured by a periodic audit by the Department) require such collaboration. hazards view. This view can be consistent with the view of the importance of prevention and protection efforts. Stakeholder Viewpoint #5: Fire-Response Capabilities Basics Perceptions of Problem and Program Resulting Visions for AFG Defining and Measuring Success Response practices and response capabilities related AFG should continue to focus most attention on Improved statistics as measured periodically by to fire incidents form the backbone of all responses applications that would build the basic fire-related the NFPA needs assessment regardless of the hazard and scenario; the needs response capabilities within the U.S. assessment clearly indicates that across the U.S. Improvements in the relative age of equipment, many basic “needs” are not yet addressed; until this is Applicants who are accredited or working toward apparatus and vehicles the case the AFG should be focused on these basic accreditation should have their applications favored capabilities. over those who are not intending on reaching Numbers of personnel trained to certain standards accreditable levels of capability. The purpose of the program is to provide financial Number of applicant organizations working support to eligible entities and not to steer or regulate Also see the stakeholder notes for the rural view and toward and accomplishing accreditations and their behavior; hence AFG should accept each the fire service advocacy viewpoints. national standards applicant’s sense of priorities. Reports by grantees can explain about the use of The original AFG intent was to address the fire these AFG-funded, basic capabilities as it relates problem within the U.S. and the AFG program can to public deaths and injuries serve this purpose while other DHS programs address the other risks.
86 Stakeholder Viewpoint #6: Public Safety More Important than Firefighter Safety Perceptions of Problem and Program Resulting Visions for AFG Defining and Measuring Success Public safety is the first thing mentioned in the statute Funding allocations should shift toward the Public education and awareness levels should be and in the National Response Plans and Preparedness “prevent” and “protection” parts of the National tested to assess their preparation for and response Goal, therefore this should be the primary focus of Preparedness taxonomy. AFG should set aside more to incidents, not just from fire but from all federal investments. funds for FP&S purposes and fund these before hazards Vehicles and other O&S activities. O&S Public safety involves the public in preparedness and applications that serve prevention and protection Reduction in public deaths and injuries from fire response, and this can prevent incidents and provide a efforts should be prioritized over other uses. and other hazards faster and more effective response in many situations. This is very true for fires, but this is also true for an Aspects of the UTL that deal with community risk Reduction in the disparity in the number of all-hazards view. assessments, regional planning, and capabilities incidents across socioeconomic groups and needed for the tasks within the prevention and communities Incident-response capabilities by themselves do not protection parts of the taxonomy should be prevent incidents or help with the public’s ability to supported. Related response capabilities and Accomplishment of the UTL capabilities for respond and over-investment in this misses an projects identified through the risk assessments and prevention and protection opportunity for cost-effective use of public resources. regional plans should take priority in funding of response capabilities (e.g., smoke alarm programs, Increase in reported rates of appropriate public Firefighter and EMS deaths and injuries are better sprinkler programs). responder efforts (e.g., number of times a reduced by preventing incidents from happening than witnessed cardiac event is addressed by a from improving response. Public responses to This view is almost identical to the view that member of the public leading to a positive incidents also mitigate risks and provide positive prevention and protection are the most important outcome; number of fire incidents successfully outcomes of interest to firefighters and EMS parts of the taxonomy to fund now. addressed by a member of the public offsetting personnel. the need for a response from a fire department)
Social equity concerns are addressed substantially by targeting efforts to populations and areas known to be more at risk of fire.
Because state and local governments have always and continue to invest in improved fire response, the federal government does not need to invest in firefighting capabilities and can complement their efforts by supporting prevention and mitigation.
87 Stakeholder Viewpoint #7: EMS-Oriented Perceptions of Problem and Program Resulting Visions for AFG Defining and Measuring Success EMS represents the biggest work load and fastest AFG funds should prioritize EMS-related Improved death and severity-of-injury statistics growing area of demand, and a critically important applications, especially those that clearly fit an all- aspect of all the large-scale disasters yet this function hazards view Increase in the number of EMS-related is poorly funded and ignored in comparison to applications for funding firefighting. There is a plain need for more Regional EMS approaches should be prioritized over investment in improvement of this function (see the those that are focused only on one jurisdiction. Accomplishment of the UTL capabilities for recent report from the Institute of Medicine). EMS
There is little attention paid to EMS-related statistics Improved numbers of well-trained EMS that relate to deaths and the severity of injuries and personnel the need for management response (e.g. the survival rates for witnessed cardiac incidents are very poor).
Fire departments do not pay enough attention to this function and treat it as a second priority, therefore federal funds should be used to alter this behavior. Stakeholder Viewpoint #8: Social Equity Perceptions of Problem and Program Resulting Visions for AFG Defining and Measuring Success Certain geographic areas and socio-economic groups, The AFG should draw on epidemiologic statistics to Improve death and injury statistics including low-income and minority communities, are establish a priority for applications that demonstrate at much greater risk of death or injury from fire and they will serve communities shown to face higher Increase in the number of applications for such other hazards. Therefore these should be the focus of risks purposes AFG funds. This priority is most consistent with AFG statutory goals. As a condition for funding, AFG should insist that applicants demonstrate they have effective Investments made by applicants who target improved prevention and public education programs in place capabilities to at-risk geographies and socio- (e.g., anti-smoking programs, public response economic groups will have larger impact than others. educations efforts, code enforcement and inspection) or equivalent other investments designed to serve areas and communities at risk (e.g., maintaining stations who can respond to these areas, expanding their response capabilities).
88 Stakeholder Viewpoint #9: Fire Isn’t the Problem Perceptions of Problem and Program Resulting Visions for AFG Defining and Measuring Success The federal government’s role in subsidizing services AFG should shift its priorities to funding matters Increased accomplishment of the UTL measures related to fire-incidents is not obvious, and other, that are consistent with the other roles the eligible more important federal interests need these resources. entities serve such as EMS or hazardous materials. Increase in a capability’s measure for serving CBRNE, HAZMAT, Search and Rescue, other Fire safety is an area where state and local Applications for vehicles and equipment or for special functions (other than typical fire governments are very responsive to public interests operational matters such as training, planning and so incidents) and have demonstrated support in their fiscal and forth that focus on EMS or other non-fire hazards regulatory postures, increasing attention because of and risks would be a higher priority than improved EMS-related statistics on performance should homeland security. fire response capability. improve (draw these from IOM and NHTSA)
The number of fire incidents has declined. Federal Applicants that demonstrate that firefighting is the funds are subsidizing the profitability of property and main problem area in their community and that there casualty insurers related to fire incidents at a time is no capacity (fiscal or otherwise) to serve this when the industry is quite comfortable with fire as an problem could still be funded. insurable risk.
Federal assumption of additional responsibility for fire safety may supplant local and volunteer support for fire safety with little or no net gain resulting. Stakeholder Viewpoint #10: Creating a “National Fire Service” Perceptions of Problem and Program Resulting Visions for AFG Defining and Measuring Success Advocates want to establish an organizational AFG applications that support coming into or Accomplishment of uniform personnel practices connection among all fire departments within the maintaining compliance with NFPA standards, and portability across the U.S. U.S. in order to establish a more uniform fire-related CPSE Accreditation, or similar benchmarks or organizational capability and capacity throughout the standards, would be prioritized over others. Firefighter safety measures improve U.S. Less economic loss per category of fire These stakeholders want a single organization to define standard policies and practices for fire departments (perhaps as exists within the United Kingdom) and would use AFG as leverage to move in this direction.
89
Stakeholder Viewpoint #11: General Governments Perceptions of Problem and Program Resulting Visions for AFG Defining and Measuring Success The program needs to ensure public support is AFG funds should only be awarded to applicants Leverage—How much more investment in the maintained for eligible entities and that federal funds who can demonstrate that the proposed uses of funds eligible organizations is being made by the do not displace such support. are consistent with community and region-wide general governments (based on representations plans established by general purpose governments. made by the general governments)? The federal program should seek as much as possible to leverage the fiscal and legal capacities of state and Number of organizations who can meet the local governments. The federal interests are not Universal Task List performance measures served by displacing public support for these organizations. Number of community-risk-assessments and completion of the tasks identified by these Fire departments and other eligible grantees need to assessments align to community and regional plans.
Fire departments should proactively engage with their general purpose local governments and foster an open debate about how these grant funds should be used in their community and region.
90 A program cannot be all things to all people. It cannot readily impose nationally-uniform capability and performance standards on thousands of communities given their diversity of size, fiscal capacity, history, and risk profiles. It may not be able to increase financial support for emergency medical services without reducing financial support for fire operations. It cannot continue emphasizing fire response capabilities and at the same time provide greater priority to prevention-related capabilities and public education as means of protecting health and safety. An effort to raise diverse communities to nationally-recognized standards ought not displace or damage community or volunteer support of the fire departments and non-affiliated EMS organizations.
Disconnects and Strategic Choices
Some viewpoints are in direct conflict with others at several points. This both opens the door to and requires strategic choices.
What is the best way to approach such choices? A first step would be to identify the apparent tensions or contradictions between the way the program is currently focused and some aspects of the problem and the environment within which AFG is operating.
In the course of this work, a number of disconnects were uncovered that need to be reconciled before deciding on a strategic direction for the program. Many are evident in the range of stakeholder views above.
Here are some of the key disconnects and tradeoffs that need to be addressed:
• The President’s Homeland Security directives regarding preparedness for and response to large-scale disasters should focus federal efforts on large concentrations of population, BUT over 70 percent of AFG funds go to rural areas.
• The Prevent and Protect segments of the Federal National Preparedness Goal focus on reducing the number of incidents, i.e., prevention, which has no apparent relationship to response capabilities, BUT AFG mainly funds vehicles, equipment, and activities related to response capabilities.
• For most entities eligible for AFG, EMS is the major demand and growth area, BUT the program has mainly addressed fire response capabilities where demands are generally on the decline.68
• Basic fire incidents are usually well-handled in the U.S. and have been for some time, whereas larger-scale, complex incidents are less well addressed and usually require
68 It is important to remember that fire departments, by statute, are a priority. Additionally, EMS receives funding from user fees, health insurance, medicine, and Department of Transportation grants, not available to the fire service.
91 cooperation of organizations and across jurisdictions;69 BUT the AFG program mainly funds local entities and isolated projects not tied to improving regional capabilities.
• Fire and EMS capabilities—whether basic, specialized, or for large-scale events—depend mainly on resources provided by general purpose local government, with the AFG program being only a minor additional contributor; BUT the AFG program has not made connections with these governments an important part of the program.
These disconnects arise to a large degree from changes in the organizational and policy contexts. To what extent can these disconnects be addressed and perhaps reduced by choosing a different future direction for AFG? In some part the answer to this question may involve large policy choices that are the province of the Administration and Congress and therefore beyond the scope of this study. However, this study can help in two ways: (1) by showing how changes in the priorities and administration of the program within its current legislative mandate can lead to better outcomes; and (2) by framing larger policy choices and their likely effects on the program’s effectiveness.
Views of a Strategic Review Panel
In the fall of 2006 the AFG program asked a small group of experienced fire chiefs and others with substantial knowledge of the AFG program to serve as a Strategic Review Panel (SRP) and meet with the study project staff for more in-depth discussions about the program. The AFG program’s managers clearly understand the need to address the strategic contexts and questions.
In the course of its work on the study, the Academy first interviewed members of this Panel and then invited them to gather at the Academy and discuss the program’s future on October 6, 2006.70 Their insights and judgments, which were shared with the program, formed part of the basis for the recommendations of the Academy as presented in its final report.
SRP participants included some of the most well-known, experienced, and thoughtful individuals within the stakeholder community:
• Harry R. Carter, Ph.D., is a contributing editor to Firehouse.com and Chairman, Board of Commissioners for Howell Township District #2, State of New Jersey.
• Ronny J. Coleman, formerly California State Fire Marshall (and author of more than a dozen books on the administration of fire departments).
• Mary Beth Michos, Chief, Prince William County Department of Fire and Rescue, Commonwealth of Virginia, past-Chair of the IAFC Emergency Medical Services Section.71
69 For additional information, see National Agenda for the Support of Intergovernmental Research, 2006 (Academy, July 2006). 70 Discussion was facilitated by the Academy Panel chair, Academy Fellow Jim Kunde. 71 Chief Michos also served as a member of the Subcommittee on Prehospital Emergency Medical Services for the study done by the IOM.
92
• Ernest Mitchell, Jr., Fire Chief Pasadena, California, formerly Fire Chief for Pawtucket, Rhode Island, and past President IAFC.
• Dennis Stark, Fire Marshal, City of Alexandria, Minnesota, Chairperson MNFCA Public Education Committee.
Some of the key points voiced by this group are also represented in the preceding matrix. Major points raised by some (but not necessarily all) members of the SRP include:
• Much more emphasis should be applied to prevention and protection including public and community education.
• There needs to be more regional coordination across the responder functions. The AFG program should continue to encourage applications that support local and regional collaboration.
• The locally-driven competitive process is a key to the successful involvement of the fire departments and non-affiliated EMS organizations.
• The applicant pool should be encouraged to develop a more “cooperative consistency” with the general government leaders in the areas they serve.
• Fire departments and the non-affiliated EMS organizations “are homeland security,” and it is important that they support the homeland security mission including an all-hazards view.
• Community risk assessments (CRAs) provide a link between the general goals of fire departments and the National Preparedness Goals and plans.
FRAMING THE STRATEGIC OPTIONS
The diversity of views about the AFG program sets up a range of possible strategic choices that the program and the Department must address—whether explicitly and self-consciously or not. Each choice implies particular changes in grant priorities and other aspects of the program’s operation, including its choice of strategies, performance measures and evaluation plans.
These strategic options are discussed in Chapter 6. Looking at the broad collection of alternative views, the following questions seem to frame one set of important choices:
• How can the AFG program best align itself within the set of other federal programs and the Department’s all-hazards, risk-based mission?
93 • To what extent and how should the program encourage more applications that address the prevention and protection aspects of the Department’s preparedness taxonomy established resulting support of the National Preparedness Goal?
• How and to what extent should the program encourage more regional collaboration and other efforts in support of national purposes while maintaining the community energy and diversity of approaches that are strengths of a decentralized, locally driven delivery system combined with a national grants competition? And,
• How can the program best use its resources to leverage local resources and provide national leadership, in cooperation with its many partners, to improve the effectiveness of the Nation’s fire departments and non-affiliated EMS organizations?
The next chapter offers some tentative answers to these questions.
94 CHAPTER 6 STRATEGIC DIRECTIONS
The AFG program, like most others, can pursue a range of strategies consistent with its legislative mandate and goals. Its stakeholders have disparate views about how the program can maximize its effectiveness, reflecting their diverse perspectives on the fire and EMS problems and on the program. A limited base of knowledge about what works best to address the program’s goals complicates the task of recommending new strategic directions for the program. Nonetheless, the weight of expert and stakeholder views points in certain directions. This offers the possibility that the program can increase the effectiveness with which its limited funds are used and, moreover, leverage its funding and leadership to improve the way others use their resources.
A set of strategic questions is addressed here:
• What does the program intend to accomplish?
• What outcomes would constitute progress and success?
• What strategies would most effectively influence those outcomes, and how does one determine whether the result is worth the investment of resources?
• What new directions, consistent with its legislative mandate, should the program consider in order to maximize its contributions to improved public safety and other national goals?
The alternative views discussed in Chapter 5 suggest a range of program strategies. As noted, some views are in conflict, and many overlap. The Academy Panel sought to reconcile and balance conflicting views, weighing them in light of evidence about the nature of the problems and expert knowledge of how they can be addressed most effectively.
The Panel believes that it is important to build on the program’s strengths, including a stakeholder-driven priority-setting and competitive award process to help target grants to places that need additional resources and will use them effectively. However, the Panel also believes the AFG program can be made more effective by adjusting its strategy based on new circumstances. Foremost, the Panel believes that the program should define its success by its contributions to public health and safety, consistent with the National Preparedness Goal and the most recent congressional guidance.
This chapter proposes fundamental but measured changes in the AFG program’s strategic direction. In all cases, these are seen by the Panel as consistent with the program’s history and context, congressional intent, and the broader DHS mission. The chapter’s concluding section provides the Panel’s recommendations.
95
HELPING FIRE DEPARTMENTS AND EMS ORGANIZATIONS ADAPT
How should the AFG program think about its role in relation to the changing roles of local fire departments, the rapid growth and uncertain evolution of EMS, and new priorities for national preparedness? One possible role for the program is as a catalyst for constructive adaptation by community fire and EMS services to these changes.
To be optimally effective, the additional funds provided by the AFG program ought to improve the capabilities of the recipients and in doing so perhaps alter their behavior and practices. With the role of fire departments expanding, with new homeland security demands and concerns, and with accelerating growth in emergency medical service calls, fire departments are faced more than ever with the need to adapt and respond to changing circumstances.
Fire departments have proven in the last several decades that they can take on new functions and roles for their communities. Investments by the AFG program may be the assistance needed— especially in financially strapped communities to support needed improvements and to adapt to changing local and national demands.
Large-scale disasters require great adaptability72 in addition to specialized capabilities and coordination with other emergency responders. In support of the National Preparedness Goal, the federal government has an interest in improving the capabilities of fire departments and non- affiliated EMS organizations to be more responsive to whatever major incidents may occur in or near their locations. This interest fuels an expectation that organizational approaches, practices, and capabilities change and improve more rapidly. The AFG program is well positioned to help.
AFG AND THE NATIONAL PREPAREDNESS GOAL
As it considers broad strategy, the AFG program is guided by two sets of goals. These goals, which overlap, derive from statutes and from published DHS strategy statements. The goals provide the general starting point for developing new program management approaches and will provide the basis for a performance evaluation framework that the program may use to judge the program’s contributions (see Chapter 10).
The DHS National Preparedness Plan embodies one set of goals, aiming to strengthen preparedness for terrorist attacks, major disasters, and other emergencies. The Fire Act embodies another set of goals, to reduce firefighter deaths and injuries, assist fire prevention efforts, and to reduce public deaths and injuries. These two sets of goals overlap, as shown in Figure 6-1.
72 Dynamic Networks: Modeling Change in Environments Exposed to Risk, Louise K. Comfort, Milos Hauskrechtm and Jeen Shang Lin, University of Pittsburgh, October 2005.
96 Figure 6-1. Intersection of the DHS National Preparedness Plan and Fire-Related Aspects of All-Hazards Management
Fire-Related Goals: Reduction in Firefighter Aspects of Deaths and Injuries; reductions in Local/Regional Economic Loss from fire; All-hazards reduction in Public Deaths and Injuries. Concurrent Goals of Management
the Fire Act and DHS All-Hazards Management Goal: To most effectively and efficiently strengthen DHS National preparedness for terrorist Preparedness attacks, major disasters, Plan and other emergencies.
Some strategic choices will increase the overlap, i.e., the extent to which the AFG program serves both sets of goals. Other choices could reduce the program’s contribution to one set of goals or the other. An aspiration frequently expressed by the program’s stakeholders is for the fire service, supported by the AFG program, to play a central role in the nation’s preparation and response to major incidents arising from a broad array of threats and hazards. For example, one participant in the October 6th meeting of the program’s Strategic Review Panel expressed his view of the ideal role for the fire service by saying, “We are homeland security.” He summarized the view that a well-equipped, well-trained, regionally-integrated fire service in as many communities as possible is a critical component of national preparedness and one that the AFG program should support.
TARGETING COMMUNITY AND NATIONAL RISK
Over its first five years, the AFG program has supplied many fire departments and non-affiliated EMS with tools and training they need to improve their capabilities in managing fire-related risks. When the DHS all-hazards preparedness perspective is overlaid on those fire-related risks after 9/11, it adds a level of complexity. As discussed above, one way to manage this complexity is to align the program’s priorities to the intersection of both perspectives, which will require prioritizing the AFG program’s uses and leveraging of other resources to improve capabilities that simultaneously address the most important risks in each community and the most important risks to the nation.
The initial reaction of many stakeholders we interviewed to a tight coupling between the AFG program and the National Preparedness Goal was that such an approach will only be effective if
97 the National Preparedness Goal is not confused with an anti-terrorism goal. Soon after the creation of the DHS, protection from terrorism as its primary mission has been somewhat tempered by the all-hazards lessons of Hurricanes Katrina and Rita, which occurred in regions of the country not typically assumed to be terrorism risks. Those experiences demonstrated that threats to homeland security come in many forms and from many sources. As a result, the DHS National Preparedness Goal is a balanced approach to matching community emergency management capabilities with the roles communities need to play in national preparedness.
On February 24, 2005, the International Association of Fire Chiefs convened a summit meeting of 18 major fire service organizations that concluded, among other things, that OMB and DHS should recognize and designate the U.S. Fire Administration and the National Fire Academy as being “homeland security critical” in the federal budgeting process.73 Aligning the use of AFG funds with homeland security goals can help the program maximizes its contribution to this goal while remaining true to its own legislative mandate and focus on fire and EMS capabilities.
DHS National Preparedness Plan and the Universal Task List
HSPD-5 identifies steps for improved coordination in response to incidents. It requires the DHS to coordinate with other federal departments and agencies and state, local, and tribal governments to establish a National Response Plan (NRP) and a National Incident Management System (NIMS). In late 2003, Homeland Security Presidential Directive 8, the “National Preparedness Goal,” was issued to “establish policies to strengthen the preparedness of the United States to prevent and respond to threatened or actual domestic terrorist attacks, major disasters, and other emergencies by requiring a national domestic all-hazards preparedness goal, establishing mechanisms for improved delivery of Federal preparedness assistance to State and local governments, and outlining actions to strengthen preparedness capabilities of Federal, State, and local entities.”
The National Preparedness Goal is designed to guide federal departments and agencies, state, territorial, local and tribal officials, the private sector, non-governmental organizations and the public in determining how to most effectively and efficiently strengthen preparedness for terrorist attacks, major disasters, and other emergencies. According to a Homeland Security “Fact Sheet,” the goal establishes readiness priorities, targets, and metrics to enable the nation to answer three key questions: “How prepared do we need to be?” “How prepared are we?” and “How do we prioritize efforts to close the gap?”74
DHS states that the required capabilities may be delivered with any combination of properly planned, organized, equipped, trained, and exercised personnel that achieve the intended outcome. However, DHS does not expect every entity to achieve the required capabilities for every area of risk. Therefore, it has developed a proposed structure that “groups similarly situated entities into tiers based on risk factors such as total population, population density, and critical infrastructure.” It goes on to say that the “tiers system is designed to strengthen regional planning, coordination, and resource sharing to prepare for incidents of national significance.”75
73 Kruger, 2005. 74 DHS, 2006. 75 DHS, 2006.
98
The Interim National Preparedness Goal was released on March 31, 2005. Figure 6-2 describes its capability-based planning approach at a high level.
Figure 6-2. Capability-Based Planning
Scenarios Tasks Capabilities
The National Planning The Universal Task List (UTL) The Target Capabilities List Scenarios highlight the provides a menu of tasks from (TCL) provides guidance on scope, magnitude and all sources that may be specific capabilities and complexity of plausible performed in major events levels of capability that catastrophic terrorist attacks, such as these illustrated by Federal, State, local and tribal major disasters, and other the National Planning entities will be expected to emergencies. Scenarios develop and maintain
• 15 Scenarios •Prevention • 36 Capabilities that each include: • Chemical, Biological, Nuclear, •Protection • Capability Description Explosive, Food and • Response •Outcome Agriculture, and Cyber •Recovery • Relationship to National Terrorism. Response Plan • Natural Disasters • ESF/Annex • Pandemic Influenza • Groups of Tasks Performed with the Capability • Associated Critical Tasks • Performance Measures and Objectives • Capability Elements • Linked Capabilities • References
Source: Homeland Security Office of Grants and Programs
The strategy DHS uses to implement its National Preparedness Goal (NPG) is to measure progress in terms of having the capabilities to perform certain tasks on the Universal Task List (UTL). This UTL reflects the common and critical tasks identified in 15 scenario-based assessments related to homeland security, many of which require fire and EMS organization involvement. According to Patricia Malek, Chief, Policy Analysis Branch, DHS, the NPG was initially focused on response but is now looking to complete the parts of the taxonomy that cover the prevent and protect tasks and capabilities. Performance metrics related to the UTL are being devised and detailed still, but the drafts recognize a role for these basic capabilities.76
The Panel recommends that these metrics be used, where appropriate, to estimate the extent to which AFG funds succeed in reducing community and National risk. As discussed, the recommended role for the AFG program is to support capabilities of fire and EMS departments that fall into the area of overlap supporting both: (1) the National Preparedness Goal by increasing the capabilities and
76Personal interview, September 20, 2006.
99 influencing the metrics proposed by DHS in the NPG; and (2) each community’s documented risk assessment.
While the mandate of the AFG is not identical to that of the NPG, it clearly overlaps with regard to the following three sets of capabilities. Table 6-1 lists the primary overlaps between the DHS Target Capabilities and the fire- and EMS-related aspects of all-hazards management by showing the capabilities related to firefighting operations, triage and pre-hospital treatment and hazardous materials response. Each of the capabilities in Table 6-1 is described in terms of outcomes, preparedness tasks/measures/metrics, and performance tasks/measures/metrics as described in the Target Capabilities List for the National Preparedness Plan (as of September 1, 2006).
Table 6-1. Target Capabilities for the DHS National Preparedness Plan Most Relevant to Fire Departments and EMS
Firefighting Operations/Support Outcome Dispatch and safe arrival of the initial fire suppression resources occurs within jurisdictional response time objectives. The initial arriving unit initiates the incident command system (ICS), assesses the incident scene, communicates the situation, and requests appropriate resources. Firefighting activities are conducted safely and fires are contained, controlled, extinguished, investigated, and managed in accordance with emergency response plans and procedures. Firefighting Operations/Support Activities Develop and Maintain Plans, Procedures, Programs and Systems (Note: Includes public fire education and life safety campaigns, building plan reviews and building code inspections) Develop and Maintain Training and Exercise Programs Direct Firefighting Operations and Support Tactical Operations Activate Firefighting Operations and Support Tactical Operations Activate Firefighting Operations and Support Size Up (Assess Sites) Contain and Control Conduct Overhaul Operations Preserve the Scenes Demobilize Firefighting Operations and Support
Triage and Pre-Hospital Treatment Outcome Emergency Medical Services (EMS) resources are effectively and appropriately dispatched and provide pre-hospital triage, treatment, transport, tracking of patients, and documentation of care appropriate for the incident, while maintaining the capabilities of the EMS system for continued operations. Triage and Pre-Hospital Treatment Activities Develop and Maintain Plans, Procedures, Programs and Systems Develop and Maintain Training and Exercise Programs Direct Triage and Pre-Hospital Treatment Tactical Operations Activate Triage and Pre-Hospital Treatment Triage Provide Treatment Transport Demobilize Triage and Pre-Hospital Treatment
100 WMD/Hazardous Materials Response and Decontamination Outcome Hazardous materials release is rapidly identified and mitigated; victims exposed to the hazard are rescued, decontaminated, and treated; the impact of the release is limited and responders and at-risk populations are effectively protected. WMD/Hazardous Materials Response and Decontamination Activities Develop and Maintain Plans, Procedures, Programs and Systems Develop and Maintain Training and Exercise Programs Direct WMD/Hazardous Material Response and Decontamination Operations Activate WMD/Hazardous Material Response and Decontamination Identify and Evaluate Scene Conduct Rescue Operations Conduct Mitigation Activities Conduct On-Site Gross Decontamination Conduct On-Site Technical Decontamination Demobilize WMD/Hazardous Material Response and Decontamination Source: The Target Capabilities List for the National Preparedness Goal (as of September 1, 2006)
Each of the activities listed above has corresponding Critical Tasks, Measures and Metrics. A sample of the Critical Tasks, Measures and Metrics for the activity called “Develop and Maintain Plans, Procedures, Programs and Systems” to support the “Fire Operations/Support” capability is provided in Table 6-2.
Table 6-2. Sample of DHS Response Critical Tasks, Measures and Metrics
Critical Tasks for “Develop and Maintain Plans, Procedures, Programs and Systems” Res.B2a 1.1 Develop plans, programs, and agreements on fire-related public safety protection activities, including region-wide or interstate automatic and mutual aid response protocols Res.B2a 1.2.5 Develop firefighting capability needed based on risk and threat assessment Res.B2a 1.2.2 Develop procedures and protocols for coordinating protective action communications with at-risk population on-scene Res.B2a 1.2.6 Conduct fire code inspections and coordinate with appropriate personnel for building inspections and compliance strategies Res.B2a 1.1.1 Conduct fire education and life safety training and education programs Res.B2a 1 Develop plans, procedures, and equipment guidelines to support firefighting response operations Res.B2a 1.1.3 Conduct building plan reviews to reduce or eliminate hazards Res.B2a 1.2.3 Develop plans for establishing alternative water supply
Each of the measures above has associated metrics that will be discussed further in Chapter 9. Since the NPG framework and associated metrics above undergoing active development and refinement within DHS, it would be impractical and a sub-optimal use of resources to develop duplicative detailed preparedness metrics for the AFG program.
101 Community Risk Assessments
Community Risk Assessments (CRAs) play a critical role in predicting hazards and saving lives empirically by profiling the risks of each community in space and time. To identify the likelihood, magnitude, and scope of each risk, fire department and independent EMS representatives should be involved with the creation and periodic updates to the CRA in cooperation with the local, and often state, government officials.
Although an over-simplification, the basic formula for a risk assessment is that probability times consequences equals risk. It is relatively straightforward to determine the probability of a fire due to the number of structures and fires each year and the current understanding of risk factors. Natural disasters are also associated with meteorological forecasts and hundreds of years of recorded history in the U.S. to provide data. Other risks, such as the probability and the consequences of a terrorist attack, are much more challenging to quantify.
To ensure that the AFG program is used to address both community risks and national risks, communities applying for grants should, to the extent possible, describe their need for federal support using the results of a community risk assessment. Moreover each grant application should identify and explain how the requested use of funds will support a prioritized National Preparedness Goal by reducing their greatest vulnerabilities. And, in monitoring and at closeout of each grant, the recipient should be able to demonstrate how the grant money has improved local capabilities related to the National Preparedness Goal’s Target Capabilities List. More detail on this recommended approach is provided in Chapter 9.
Box 6-1. New Zealand: Prevention Emphasized; Death Rate Declines
The New Zealand Fire Service (NZFS) has a country-wide responsibility and predominantly serves fire problem matters. New Zealand is about the geographic size of the United Kingdom, with 4.1 million people. NZFS contributes response services in the case of motor vehicle accidents, hazardous materials, and other rescue and related medical emergencies. It does not, however, provide an EMS function similar to many fire departments in the U.S. as only 11.9 percent of their incidents are recognized as being rescue/medical in nature (2005). One-third of their incident calls are fire-related.
Figure 3-8 shows New Zealand’s very low per capita death rate (from residential fires) and very low cost of service—putting near the top of this international ranking. The actual services are provided locally, with almost 80 percent of the firefighter resources coming by way of volunteers.
In 1999, the NZFS shifted its orientation toward prevention based on the local community. Although annual fire-incident rates have remained level over the last decade, recent figures for public deaths are on the decline. Public deaths have been “the main focus of the national promotion activities.” NZFS reports that the “overall trend [in public deaths] has been reducing over the last eight years” and the “Commission believes that the results of fire safety education programs are now becoming more evident.”
2002/03 2004 2005 Avoidable Residential Structure Fires Fatalities 25 (0.68) 22 (0.57) 19 (0.47) (fatalities per 100,000 population)
Source: Enhancing Community Security, Annual Report for the Year Ended 30 June 2005, NZFS.
102
Fitting the AFG into the Future
The AFG program, in its short history, is widely viewed as well managed and as having made a successful contribution to improved fire response and EMS capabilities across the nation. With this earlier experience to draw on now, this is a good time for the AFG program to step back and reexamine its original focus on building firefighting capabilities, considering both how the program fits the current reality facing fire departments in the U.S. and how well it supports broader national homeland security and community public safety objectives.
Lacking the depth of reliable quantifiable information about costs and results needed to conduct a formal evaluation of alternative strategies in a cost-effectiveness framework, how should the program approach such reconsideration? The Academy Panel has relied on a consensus of expert opinion where available, especially with regard to the efficacy of specific fire prevention and EMS capabilities. The Panel also recognizes the importance of examining the program’s contribution to broader national preparedness objectives that received new priority following the attacks of September 2001 and reinforced the Gulf Coast storms in the fall of 2005.
The Panel believes that the strategies chosen by the AFG program must reflect the way the U.S. has organized its fire and EMS responsibilities historically. Any set of national priorities must be implemented at the local level taking proper account of wide variations in the way fire services and EMS are organized and in the risk profiles of communities who are potential recipients of federal assistance. Above all, given the nation’s heavy reliance on local community support for fire departments and other emergency services, the AFG program must continue to give great deference to local assessments of risk and assistance needs, provided they are based on an adequate community risk assessment methodology.
Here, then, are five strategic directions that the Panel recommends be considered by the AFG program and its leadership. These are offered with the understanding that the program operates in a rapidly evolving context and must therefore refine, adapt, and redirect them over time, through a continuous strategic planning process of the sort recommended in Chapter 7:
• National Preparedness. Recognizing that the AFG program had its legislative origins prior to 9/11, it is impossible to ignore the subsequent shift of national priorities toward national preparedness to deal with catastrophic events. Nor does the Panel see a contradiction, in most cases, between building capacity to address such events and bolstering local fire department and EMS capabilities. The firefighters of New York and Arlington, VA, were the core of the first responses to attacks on the World Trade Center and the Pentagon, respectively. If the fire services have been less central to planning at the national and sometimes at the regional level for emergency response, their contributions will nevertheless be central to an effective response to future catastrophic events. In early October 2006, Congress enacted changes to the organizing statute for the Department that will require it to align all of its efforts to an all-hazards perspective on
103 large-scale disaster risks within the U.S.77 Although not singled out, the AFG program is included in this enactment, signed by the President in October, 2006. The program and the Department must therefore work to fit AFG into federal response planning. This can be done by prioritizing those capabilities specified in the National Preparedness Plan that are most relevant to local fire departments and EMS. (The details of how this can be accomplished are presented in Chapter 9.)
• Prevention vs. Response. With regard to the relative emphasis given to prevention and response capabilities, the Panel must follow the lead of national panels of experts and the advice of others interviewed during the course of this study. Those who know most about the fire problem have repeatedly emphasized that, in many situations, at the margin, increased emphasis on proven fire prevention approaches will have greater benefits for fire safety than further improvements in fire response capability (see also the New Zealand example, in box above). The program should also define prevention efforts to extend beyond the fire prevention mission to include programs designed to prevent incidents of all types including EMS. This requires helping applicants understand that public communications and education efforts for health promotion purposes and to prepare the public to be better first-responders, including elder care programs, CPR classes, child safety seat installations, and blood pressure checks, are appropriate uses of funds.
The program should consider how best to encourage and prioritize high-quality applications for the funding of mitigation capabilities, including public education in fire and EMS safety that is targeted at those groups and places where risks are highest. However, the Panel does not recommend setting a target percentage of program funding to be awarded for such purposes until more reliable information is available about the efficacy of particular prevention approaches for particular populations and risk situations. The pace and extent of a shift toward prevention also will depend in part on the priorities established by each community through its own risk assessments. As more communities bolster their basic response capabilities with AFG grants and other funds, more will find that prevention strategies tailored to the particular risks they face are the best use of AFG funds. As it moves incrementally in the direction of supporting more prevention, the program should work with other federal agencies, nonprofit organizations, and program stakeholders to identify the most promising practices and to monitor and assess the ability of communities to use these to reduce fire risks. The program also should work with these partners to encourage greater recognition by local fire services and the public of the potential value of prevention strategies, including secondary prevention aimed at reducing the spread and cost of fires when they occur.
• Social Equity. As the nation improved its fire safety record in recent decades, some communities and populations lagged behind. These include smaller places with limited fiscal capacity that simply lack the resources necessary to adopt modern equipment and
77 P.L. 109-295, Homeland Security FY 2007 Appropriations. The AFG has historically been focused on the response segment and mainly on fire incident response. The AFG is restricted to its eligible organizations, fire departments and non-affiliated EMS organization, while the UTL covers all hazards across all segments (prevention, protection, response, recovery).
104 provide training in best practices. They also include places where people are concentrated in older buildings that burn more easily and lack sprinklers or other protective systems. Statistically, low-income, immigrant, and minority communities are at most risk from fire. The AFG program can increase the cost-effectiveness of its grants by targeting their use to places and people at greatest risk. Doing this effectively will require communities to conduct community risk assessments and use these in preparing their applications. The quality of these local assessments will, in turn, be improved over time as research being conducted under a grant from the Fire Grants program by the National Institute of Standards and Technology (NIST), in conjunction with the International Association of Firefighters and others, comes to fruition. The aim of this research is to provide a scientific basis for community risk assessment and strategic emergency response system design based on an original analysis of empirical data analyzing determinants of the effectiveness of actual responses to real incidents. Over time, the program’s selection process should be adjusted to use the information thus provided to prioritize awards based in part on whether they address a community’s major risks.
• Regional Cooperation. Some observers most familiar with AFG’s program’s effects to date stress the extent to which it has brought fire departments from neighboring jurisdictions into closer regular contact. Beginning in 2005, the AFG criteria for award gave a priority to applications that adopt a regional approach.78 However, the Panel believes that still greater emphasis on funding region-wide capabilities could increase returns from AFG spending both to national preparedness and to local public safety. For example, smaller communities outside major metropolitan areas could be encouraged to think of developing specialized capabilities—such as those for fighting wildfires—that would be needed not only locally but in response to a major catastrophe affecting a large population center and possibly far away. Because the geography of future catastrophic events is unknown, it is advantageous to develop portable capabilities in many different parts of the country. AFG should work with the Department and its partners to think through the best way to use its funds to build a national network of such response capabilities that can be quickly deployed where needed. In many cases, such a focus on logistics may be more cost-effective than bringing all communities up to a certain level of capability. AFG also should continue to reward local fire departments and unaffiliated EMS in major metropolitan areas that participate fully in the development of regional response capabilities. Regional cooperation and planning must have an all-hazard perspective that assesses the EMS risks and capabilities and includes the EMS providers in the planning effort.
• Emergency Medical Response. The Panel notes that the way EMS is delivered, and in particular the relationships between local and regional views of a fire service and EMS, vary widely in the U.S. The best organizational solution and the most cost-effective way to deliver EMS services are now subjects of intense debate. It is at best premature for the AFG program to in any way favor one local organizational model over another. Until more is learned about the best way to organize EMS and fire in various size communities,
78 Department of Homeland Security, 2005 AFG Program Guidance available, at http://www.firegrantsupport.com/docs/2005AFGguidance.pdf.
105 we recommend that the grant award process be neutral with regard to modes of delivery and permit applicants to emphasize either fire or EMS capabilities depending on their local risk assessments and sense of priorities. In the near future, it would be useful for DHS, HHS, and DOT/NHTSA to jointly consider the issue of how the federal government can best encourage a rationalization of first responses to medical emergencies and coordinate their development of incident data bases and research.
Can AFG reconcile certain tensions within its legislative mandate and set a direction that is consistent with the best expert views of the nature of the fire problem and how it can be addressed most cost-effectively, is practical in terms of the program’s limited scale, and is generally consistent with the views held by most of its stakeholders?
As emphasized in the preceding discussion, the Academy Panel shares the prevailing view of those most familiar with the AFG program that its success to date can be attributed in large part to the direct engagement with representatives of the fire service in setting program priorities and in judging the merits of thousands of applications annually. In the Panel’s view, regardless of the extent to which it directs funds to national priorities, the program can retain its character as a locally-driven competition reflecting the best thinking of community based fire services and other emergency responders. Many communities need federal assistance to improve their basic capabilities to address the program’s fire safety and EMS goals.
The nation needs local fire departments and EMS to improve and to become more adapted to the risks faced by each community and region in order to strengthen national preparedness. These recommended strategic directions represent the Panel’s best advice regarding how the AFG program can be used for that purpose. However, these should be viewed as the beginning rather than the end point of strategic planning. The following chapter describes how the program can use a continuous planning process to constantly improve and multiply its effectiveness.
106 CHAPTER 7 CONTINUOUS STRATEGIC PLANNING
Effective programs set clear performance objectives, develop realistic strategies for achieving those objectives, and continually review their progress to improve their performance. This requires an ongoing annual planning and review cycle integrated with the agency’s budgeting cycle and with agency level strategic planning processes.
OMB PART
OMB assesses strategic planning for individual programs using its Program Assessment Rating Tool (PART).79 In judging the quality of a program’s strategic planning, the PART assesses whether the program has a limited number of performance measures with ambitious and achievable targets to ensure planning, management, and budgeting are strategic and focused. Sources and evidence for the assessment include strategic planning documents, agency performance plans/performance budgets and reports, reports and submissions from program partners, evaluation plans, and other program documents. While it is recognized that some programs have greater difficulty than others in developing quantitative performance goals, all programs are expected to have meaningful and appropriate methods for assessing their progress and demonstrating results.
Specific Yes or No questions in the PART test whether a given program has an adequate strategic planning process as follows:
2.1: Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?
To determine if a program has long-term performance measures to guide program management and budgeting and promote results and accountability as required by the PART, beginning in 2002, and the Government Performance and Results Act of 1993 (GPRA), OMB assesses whether program measures are salient, meaningful, and capture the most important aspects of program purpose and appropriate strategic goals. To receive credit, a program must have about two or three specific, easily understood long-term outcome measures that directly and meaningfully support the program’s purpose. “Long-term” means a long period relative to the nature of the program, likely 5-10 years, and consistent with time periods for strategic goals used in the agency’s GPRA strategic plan.
The performance measures should focus on outcomes, although in some cases output measures are permissible. A Yes answer can also be given if OMB and the agency have reached agreement on a limited number of long-term measures that will be added to the GPRA strategic plan or performance plan/performance budget in the next budget cycle.
79 For an overview of the PART, see www.omb.gov. Also see Redburn, Shea and Buss, Performance Management and Budgeting (2007).
107 Significant changes to the GPRA strategic plan may require stakeholder consultation. Whenever output rather than outcome measures are proposed, the program must clearly show how such measures reflect progress toward desired outcomes.
2.2: Does the program have ambitious targets and timeframes for its long-term measures? For a program to be judged as having challenging but realistic quantifiable targets and timeframes for the long-term measures, it must meet these tests:
• The program must have specific quantified targets for measures evaluated in Question 2.1. (Where targets are not “quantitative,” they still must be verifiable, e.g., through a clean audit or outstanding ratings by an expert panel).
• The program is strongly encouraged to have clear baselines from which to measure targets and changes in performance for outcome measures.
• The program must have baselines for output measures.
• All targets and timeframes must be ambitious—that is, they must be set at a level that promotes continued improvement within achievable efficiencies.
• Where relevant, the program must define an appropriate end target.
A Yes answer needs to clearly explain and provide evidence of each of the following:
• A limited number of discrete, quantifiable, and measurable annual performance measures have been established for the program.
• Annual performance measures adequately measure the program’s progress toward reaching the long-term goals evaluated in Questions 2.1 and 2.2. The explanation must clearly state how the outcomes help achieve the long-term goals of the program.
• Annual performance measures must quantify outcomes if at all possible.
2.3: Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program’s long-term goals?
To receive credit, a program must have a limited number of annual performance measures that directly support the long-term goals evaluated in Questions 2.1 and 2.2. The measures should be logically linked to the long-term goals in a manner that enables them to demonstrate progress toward achieving those long-term goals.
A Yes answer requires evidence of the following:
• A limited number of discrete, quantifiable, and measurable annual performance measures have been established for the program.
108
• Annual performance measures adequately measure the program’s progress toward reaching the long-term goals evaluated in Questions 2.1 and 2.2. The explanation must clearly state how the outcomes help achieve the long-term goals of the program.
• Annual performance measures quantify outcomes if at all possible.
2.4: Does the program have baselines and ambitious targets for its annual measures?
Purpose: To determine if the program has baselines and challenging, but realistic, quantified targets for the annual measures. To receive credit, a program must meet the following tests:
• Baselines have been established for most of the annual measures evaluated in Question 2.3.
• Specific annual targets, in almost all cases quantified, have been developed for most of the program’s annual measures evaluated in Question 2.3. These targets provide a specific value with which performance can be compared.
• All targets are ambitious—that is, they must be set at levels that ensure continued improvement and realization of efficiencies. They also should be within reason for the program to achieve.
2.5: Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?
Purpose: To determine whether program efforts carried out by program partners also support the annual and long-term performance goals of the program. Credit here requires each of the following:
• Partners support the overall goals of the program.
• Partners measure and report on their performance as it relates to accomplishing the program’s goals.
2.6: Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?
To receive credit, a program or agency must conduct non-biased evaluations on a regular or as-needed basis to fill gaps in performance information. These evaluations should be of sufficient scope to improve planning with respect to the effectiveness of the program.
The appropriate design for an evaluation of the AFG program is discussed in this report’s final chapter.
109
In short, OMB judges the quality of a program’s strategic planning effort by whether it produces realistic but ambitious long-term and short-term goals that permit the program’s managers and others to assess whether it is making progress toward important outcomes. The quality of a strategic planning process therefore must be judged by whether it produces agreement and commitment to such goals. The following section outlines the kind of continuous strategic planning process that the AFG program could follow to meet OMB’s expectations and permit the program to continually review and improve its use of resources in order to maximize its effectiveness in achieving its statutory purposes.
CONTINUOUS STRATEGIC PLANNING
As noted in Chapter 5, the AFG program designated a small group of experienced fire chiefs and others with substantial knowledge of the fire problem and the program to serve in the fall of 2006 as a Strategic Review Panel. In the course of its work on the study, the Academy first interviewed members of this panel and then invited them to gather at the Academy and discuss the program’s future in October 2006. The Strategic Review Panel’s insights and judgments, which have been shared with the program, form one basis for the recommendations of the Academy Panel regarding strategic directions (see Chapter 6).
The convening of the Strategic Review Panel would be a significant and helpful event if done once. However, it will be of much greater value to the program if it initiates an ongoing regular strategic planning cycle. This process should help the AFG program’s managers set—and periodically revise—multi-year goals for improving performance, continually review their progress and performance in relation to those goals, and regularly revisit the problem of how to improve performance based on new insights concerning the problem and what works best to address it. Monitoring and evaluation—as described in Chapter 10—should feed information to this process. The setting of measurable goals resulting from the process and the continuous monitoring of progress toward those goals will guide decisions regarding resources and steer their best use to achieve the program’s aims.
The best-run programs and organizations conduct such strategic reviews regularly and, more importantly, make these the starting point for their annual budget planning and other routine management processes (see Figure 7-1). For example, the AFG program already conducts an annual review of its criteria for grant award involving representatives of its stakeholders. This annual process can be guided by and, in turn, inform a recurring strategic planning process; it should be part of an integrated annual planning cycle. A recurring strategic planning process will enable the AFG program to adjust as needed to changes in its environment and to increases in knowledge about the nature of the problems of fire, EMS and preparedness for major incidents arising from a broad range of threats and hazards, and about the most cost-effective means to address these risks.
110 Figure 7-1. Diagram Showing Annual Strategic Planning Cycle and its Integration with Budgeting
It is important for the AFG program to synchronize and coordinate its planning with the major Department-wide planning processes in DHS. In particular, the program is responsible for contributing to strategic objectives of the Department as presented in its strategic planning and annual performance plans as required by GPRA. In addition, the program is a focus of support for the National Performance Goal and accomplishment of the Universal Task List and must take into account the evolution of those plans in its own planning and management decisions. An effective program-level strategic planning process will ensure that it maximizes its contributions to the broader strategic objectives of the Department.
As has been noted, meeting these public health and safety challenges requires the collaboration of many government and non-government actors. At the federal level, the areas addressed by AFG are shared within DHS between the AFG program in the Office of Grants and Contracts and the USFA. Other federal agencies play important roles in addressing fire safety, notably the Centers for Disease Control’s Center for Injury Prevention and the Consumer Product Safety Commission. EMS issues are addressed by the Health Resources and Services Administration and via the reimbursement policies of the Centers for Medicaid and Medicare at the Department of Health and Human Services (HHS); by the EMS Division of National Highway Traffic Safety Administration; by the research being sponsored by the National Institute of Standards and Technology; and by the Agency for Health Care Quality (also in HHS).
111
The quality of the program’s strategic planning will depend in great part on the appropriateness, timeliness, and quality of the information and analysis available to planners and managers about performance. This is the subject of Chapter 8. Information about performance can infuse and guide every aspect of the program’s management. This is the subject of Chapter 9. Planning also can be supported, as noted in the OMB PART, by conducting an appropriately rigorous evaluation of the impacts that the program, and particular uses of the program and other resources, have had on the fire and EMS problems. This is the subject of Chapter 10.
The Need to Coordinate Federal Fire Safety and Other Emergency Preparedness and Response Planning
In other national forums, goals have been established for fire safety that parallel the statutory purposes of the AFG program. For example, a coalition of federal agencies and non-federal organizations that was brought together in 2002 under the umbrella of a self-designated Public- Private Fire Safety Council, has endorsed the following overly ambitious goal: “Eliminate [U.S.] residential fire deaths by 2020.”80
The Healthy People 2010 plan sponsored by Office of Disease Prevention and Health Promotion, HHS includes the following target and description: 81
• Objective: Reduce residential fire deaths.
• Target: 0.2 deaths per 100,000 population.
• Baseline: 1.2 deaths per 100,000 population were caused by residential fires in 1998 (age adjusted to the year 2000 standard population).
• Target setting method: Better than the best.
Comparable measures from the FY 2007 DHS Performance Budget Overview are shown below. These relate only to firefighters and fire safety. This and other public documents do not indicate whether the Department has set and currently pursues long-term targets for fire safety or for emergency medical services. The Department’s current (2004) Strategic Plan is written at a high level of generality and includes broad goals for national emergency preparedness, response, and recovery (see excerpt in Table 7-1).
80 http://www.firesafety.gov/programs/ 81 National Vital Statistics System, CDC, National Center for Health Statistics, Objective 15-25. http://www.healthypeople.gov/
112 Table 7-1. Department of Homeland Security’s Current (2004) Strategic Plan
Program: Fire Prevention Assistance Performance Goal: Maximize the health and safety of the public and firefighting personnel against fire and fire-related hazards by providing assistance to fire departments and by training the nation’s fire responders and health care personnel to prevent, protect against, respond to, and recover from fire-related events. DHS strategic objectives supported 3.7 and % allocation of activities: 100.0% Fiscal Year2003 2004 2005 2006 2007 Measure: Ratio of on-scene fire Target None None None 3.4 3.4 incident injuries to total number of active firefighters. Actual None None N/A N/A N/A Measure: Percent reduction in Target None None 15% 18% 21% the rate of loss of life from fire- related events. Actual None None None N/A N/A $ Lead Organization: Thous N/A N/A $715,000 $648,450 N/A Preparedness ands FTE N/A N/A None 156 157 Source: The table was reproduced from the DHS Strategic Plan for 2004
Overlapping but differently specified goals set by separate planning processes are documentary evidence of a need at the federal level for greater integration of federal strategic planning for fire safety. The same point could be made about federal planning for emergency medical services, where responsibility and attention are divided among HHS, National Highway Traffic Safety Administration in Department of Transportation, and the AFG and USFA programs in DHS. It is beyond the means of the AFG program alone to ensure better coordination of federal strategic planning for fire safety, but it can reach out to other programs in other agencies to initiate a dialogue about how their separate planning processes could be integrated and thereby ensure a more focused and integrated federal response to the fire safety challenge.
Another opportunity for better coordination of planning processes exists within DHS itself. DHS’s strategic planning processes focus primarily on the Department’s broader mission to prepare the Nation for and to mitigate natural disasters and other major events. To date it has not fully integrated planning for the AFG program with planning for this broader mission. Although the ways the AFG program can maximize its contribution to broader preparedness goals are readily identifiable (see discussion in Chapter 6), a more explicit statement of the expected contributions would provide useful guidance to program managers and stakeholders. Such a statement would be one useful product of integrated planning at the Department level.
Setting Long-Term Performance Goals for the AFG Program
The major task of the program’s strategic planning process should be to establish long-term goals against which it will judge and report progress and performance annually. These must be realistic but ambitious, and consistent with the program’s statutory goals. They should represent
113 a consensus of the program and its stakeholders about what the program expects to accomplish over a specified multi-year period.
More challenging but even better, the AFG program could lead in developing a consensus of federal agencies and its private partners concerned with fire and EMS around long-term objectives for the federal government’s efforts to improve fire safety, emergency medical response, and the role of local fire services in preparing their communities for major incidents arising from a broad range of threats and hazards. Although the program’s resources are modest in comparison to those spent by other levels of government, it has the ability—through its strong working relationship with representatives of the fire service, the way it manages its annual competition, and the way it prioritizes uses of its funding—to leverage substantial changes in the way these other resources are used. That gives the AFG program a potential leadership role in joint planning and consensus building across federal agencies and with other partners and stakeholders.
FINDINGS AND RECOMMENDATIONS
The Panel finds that:
• The AFG program has made an important start at strategic planning by assembling for this study a group of acknowledged experts in the fire problem and EMS to discuss strategic directions for the program. However, it currently lacks a fully developed ongoing strategic planning process.
• The AFG program currently lacks a set of long-term, ambitious targets and measures against which to measure its progress.
• Strategic planning at the Department level is not yet fully integrated with planning for the AFG program.
Therefore, the Panel recommends that the AFG program:
• Establish a continuous strategic planning cycle that is integrated and synchronized with similar processes at the Department level;
• Set a small number of quantified multi-year targets for fire safety and improved EMS performance that are realistic, ambitious, and consistent with both its statutory mandate and mission guidance from the Department;
• Work with its partners in and outside the federal government to build a knowledge-based consensus on long-term national goals for addressing the fire and EMS problems; and
• Use its strategic planning process to identify opportunities to maximize its influence on how others—especially communities—use their resources to address national goals, consistent with community and regional risk assessments.
114 CHAPTER 8 PERFORMANCE MEASUREMENT
The federal government has developed a number of performance management assessment tools in recent years as part of the President’s Management Agenda (PMA).82 One of those, the PART, is used by OMB examiners to assess programs, with particular emphasis on how they measure performance and how well they perform. Agencies are being encouraged more and more to develop a results-oriented performance culture and practice.
Programs are expected to set both long-term and short-term performance targets and to measure their progress against those targets. Agencies can link these goals and measures of performance to expectations for individuals, teams, and agency units as a way to institutionalize the focus on goals and measurable results.
Performance-based management requires available, timely, and accurate data about various aspects of government programs. This chapter describes measurement approaches as they relate to the performance of grantees and the logic of the program. (The program’s current logic model is described in Chapter 4 for the fire problem.)
THE BASICS OF PERFORMANCE MEASUREMENT
According to a report to Congress and the President by the Reason Public Policy Institute on transitioning to a performance-based government:
“Most agencies do not know what strategic rationale underlies their programs. They do not have a sense of what specific contribution each individual program is making to national outcomes. As such, they do not have clear linkages between strategic goals and their individual program performance measures…Some programs are measuring the wrong things, some are measuring too many things…The lack of a real data driven understanding of how what we do on a day- to-day basis contributes to results outside the agency is still a missing gap.”83
In Reinventing Government, authors David Osborne and Ted Gaebler describe the power of performance measurement: “Organizations that measure the results of their work…find that the information transforms them.”84
Years later, agencies, including DHS and the AFG program in particular, are still working to overcome many of these same obstacles to embracing performance-based government.
82 See www.omb.gov. 83 Reason, 2000. 84 See also Redburn, Shea and Buss, Performance Management and Budget (2007).
115 To overcome the obstacles and understand how the AFG program can use performance measurement, it may help to be more familiar with three concepts: performance measurement, baselines, and benchmarks.85
Performance measurement refers to a system of collecting data on all elements of a program’s logic model. It concentrates most on measurement of outputs or outcomes closest to the program’s established goals, but includes measurement of intermediate results expected to be critical to ultimate outcomes. Such measurement can allow an organization to determine whether it is improving, maintaining, or failing in achieving a level of service or producing a certain result related to its goals. Measuring intermediate results as well as ultimate outcomes related to multiple goals provides an organization with a rich set of performance information on which to base planning—its choice of strategies, its management decisions, its priorities for grant award, and so on.
A baseline describes something at some point in time using data. A baseline can describe the level at which an organization, a department, a process, or a function is operating at a given time. When examining operations, some set of data must be established as a means to create a baseline against which to compare or measure changes from one period to another; otherwise, the idea of “performance” is always elusive. To measure future performance, today’s levels and trends should be used to establish the baseline. If an organization were to review operations over a ten- year period, for example, it would be helpful if a baseline had been established ten years ago and performance data collected each year since then. In the case of the AFG program, a baseline about fire departments and non-affiliated EMS organizations may be useful as a point of reference for the grant selection process. The Academy Panel’s focus on national preparedness, for instance, offers one the opportunity to define a baseline.
Data used to establish baselines should be as hard as possible: dollars and how they are used; staffing levels; response times; and so on.
A benchmark is a comparative standard against which the organization can assess its performance. Fire departments and EMS organizations, as service-providing organizations, have acknowledged a number of standards or benchmarks they will compare to their own service performance. A response time target, a level of certification for personnel, or a goal to limit 95 percent of structure fires to the room of origin could all be adopted as standards. Starting from a baseline, an organization can seek to achieve a benchmark through increased funding, increased activity, or changes in the way it operates. In the case of the AFG program, there is an opportunity to compare recipients with similar non-recipients while the focus on national preparedness capability and task list accomplishments offers another benchmarking opportunity.
Baselines exist whether anyone is collecting and using data at present or not. Benchmarks can be sought as references for use in measuring improvement. But note that a benchmark used for service-level agreements or coverage standards is not the same thing as an outcome goal in public policy terms. They ought to be related in practice in some way, but one is used predominately for managing a service-oriented organization; the other in public policy to indicate preferred outcomes. The AFG program needs to arrive at some comfort level about the
85 These concepts are discussed in Hatry (2006).
116 relationship of the activities of the fire departments and non-affiliated EMS organizations to the health and safety outcomes expressed in the statutes and the National Response Plan. The Panel holds the view that these relationships are not clear and may be too ambiguous to be “evaluable” as the program is designed now.
Benchmarks can be used successfully for some purposes but not for others. National Fire Protection Association (NFPA) Standards 1710 and 1720 may have been the first documented national benchmarks that addressed fire department staffing levels and response times. They address both career and volunteer fire departments. But, as described in Box 8-1, even benchmarks produced through a broad-based and collaborative effort may not be an appropriate reflection of the risks in a community.
Box 8-1. Benchmarks and Risk Assessments—NFPA Standards
“Obviously, each type of service provided by a fire department has associated risks which must be considered as a community judges its level of protection and projects the protection level it would like to have. Just as obvious is the knowledge that cost must be considered. This is a major reason why risk management is so closely tied to the assessment of community protection levels. The issuance of these standards triggered an outcry that staffing levels and response times are a ‘local’ jurisdictional prerogative. If that is true, then that level is where the discussion of ‘acceptable level of risk’ needs to occur.”
By Stephen N. Foley, Executive Secretary, Fire Service Section, National Fire Protection Association, from “Evaluating Community Emergency Services—A Public Entity Risk Institute Symposium.”
The relationship between baselines, performance measures and goals is represented in Figure 8-1. A goal or target can be related to some benchmark where one is known or can be developed. Performance measurements provide information required to chart progress toward goals. Organizations would make changes to their practices, behaviors, and activities expecting to see this kind of separation from a baseline.
117 Figure 8-1. Relationship between Performance Measures, Baselines and Goals
Source: Adapted from Friedman, M. Trying Hard is Not Good Enough—Workshop Materials. Fiscal Policy Studies Institute, 2006.
Once strategic directions for a program are set, several questions about performance measures, baselines, and program goals remain to be answered. From AFG’s perspective, these questions are:
1. What data sources currently exist and are needed to establish baselines and to conduct performance-related measurements?
2. How should the AFG program select performance measures based on the goals and logic of the program?
3. How should the AFG program use performance goals and measurements to guide priorities, to make awards, to assess the value of grantee efforts, and to inform strategic planning?
To answer the first question, the Academy’s study team surveyed available data sources relevant to the AFG program and interviewed experts to assess data availability and ascertain their relevance and validity.
The basis for answering the second question was established in part by the program’s logic model and in part by the new strategic directions recommended in Chapter 6. Data sources related to firefighters and EMS personnel are mentioned in the tables that follow, to assist with the review and selection of measures. Selecting the appropriate set of performance measures can enable the AFG program to better focus its relatively modest grant funds.
118
The remainder of this chapter reviews sources of data and assesses the potential value of specific performance measures to the AFG program. The next chapter responds to question three above and specifies a performance-based management approach that would deploy these performance measures.
AVAILABLE DATA SOURCES FOR PERFORMANCE MEASUREMENT
A performance measurement system depends on practical methods of developing and/or collecting data. Availability and cost are major considerations.
For AFG, useful data may come from a number of reporting programs (e.g., NEMSIS and NFIRS) and statistical series and studies. Grant applications and grantee performance reports offer another potential source of useful information and data.
This section describes a variety of incident-based and survey-based data sources that can be drawn on to provide data about public deaths and injuries related to the fundable activities under the grant program (including projects that are fire-only, EMS-only, and other eligible projects).
Incident-Based Reporting
An incident report is a record of a fire or other incident that occurred. According to NFIRS, “It may be as brief as a basic fact statement or as lengthy as an extensive discussion of the incident, supported by photographs, witness statements, and laboratory test results. The length and complexity of the report will depend upon the nature and magnitude of the incident, state and local policies concerning data collection, the need for specific data, and the resources available for obtaining information and completing reports.”86
Three basic incident types cover most fire department and non-affiliated EMS organization activities:
• A fire incident where records are to be produced and often sent to collection points under state laws and local ordinances. The federal government developed and provides modest financial support for a nationwide effort to collect common records about fire and other incidents by way of NFIRS. Note that NFIRS also contains record-keeping guides for a number of other incident types addressed by fire departments and rescue squads within the U.S.
• An EMS incident which can be reported using the NFIRS infrastructure, but it is also reported upon by way of state health care data-collection systems. CDC and the Consumer Product Safety Commission (CPSC) also gather injury-related patient health records from the emergency departments of 100 hospitals selected as a probability sample of the 5,300+ U.S. hospitals with emergency departments. (See Appendix C for an inventory of national injury data systems compiled by CDC.) The federal government
86 NFIRS 5.0 Overview, DHS.
119 has recently established the NEMSIS87 as a nationwide data-collection approach for emergency medical incidents and care.
• Large-scale disasters or other incidents of significance which are the focus of attention under HSPD-8 and the statutory mission statements for DHS. The Department has asked state and local governments to provide them with any number of reports that might identify such incidents or disasters. Some of this is reported by way of law enforcement data collection agreements, and some is collected by special agreements that are less visible to the public. Disasters are, of course, identified in formal terms to federal officials by state and local requests for assistance under terms of the federal Stafford Act. DHS draft guidance on the Target Capabilities List included several yes or no performance questions related to the National Preparedness Goal. The sub-section dealing with national preparedness lists several of these as examples.88
For the most part, fire departments and EMS organizations participate voluntarily in such data collection, although legal sanctions can be brought in some cases as a matter of state law. The federal government has offered its assistance to help develop a common approach and accelerate participation across the U.S. Compliance with any of these nationwide efforts cannot be mandated on state and local governments except as a condition for the receipt of federal assistance. Congress made compliance with NFIRS a condition of the AFG grants, and grantees comply in nearly all cases.
(Table 8-1 provides a description of the most relevant sources of incident-related records or data sets that compile such records. Appendix D contains a more detailed description of these and a more comprehensive set of available data sources related to injuries and deaths.)
87 Information about the NHTSA/Emergency Medical Services Division can be found at http://www.nhtsa.dot.gov/portal/site/nhtsa/menuitem.2a0771e91315babbbf30811060008a0c/ ; on NEMSIS at www.Nemsis.org . 88 Target Capabilities List, DHS Draft, 2006.
120 Table 8-1. Incident Reporting Data Sources
Data Source Description Data Quality Availability Offers 84 types of detailed incident Medium Data are collected in batches from each state data in 11 modules plus narratives. Self-reported, non-validated; many modules collection point, and these are integrated and Although it is optional for states are not required of participants. Consistency managed centrally by DHS. National Fire and/or communities, AFG award of interpretation may be an issue. applicants are required to participate An October, 2006, estimate by the U.S. Fire Incident during the period of performance. The collection process produces a time lag, Administration was that 20,000 fire Reporting which can differ by participating departments are participating out of a fire System The records are sent from responding organization and by state. department population of approximately (NFIRS) organizations in batches to a central 30,000. The adoption rate is considered collection point in each participating Discipline in adhering to the reporting sufficient for the intended sample analysis by state or directly to DHS. standards varies by originating organization the U.S. Fire Administration. and by state. Based on the Accreditation Model, High CPSE is requiring all accredited CPSE/CFAI with a risk assessment focus, it Self-reported with Peer Reviews and on-site organizations to submit data in a common provides access to self-reported data inspections, integrated with ISO data. format derived from the use of their software. Data and related to 108 competencies, across Vision 10 categories, 47 criteria and 254+ Fee-based, these data are available to all Software performance indicators. contributors for benchmarking purposes and for research efforts upon request. Analogous to NFIRS but for EMS Pilot Stage Some states are collecting data from data only as defined by NHTSA. Until the computer-based applications or participating organizations and State systems that manage the EMS function adopt Emergency Management Directors have National Participating states collect records this schema, states and participating apparently agreed that the states should Emergency from participating organizations and organizations may be converting existing employ this infrastructure and schema. Medical forward these data to a NHTSA- records or transforming existing data into Services managed repository (NEMSIS, 2006). this schema for the purposes of reporting to DHS has also made note that participation in Information the national repository. NEMSIS is a yes/no performance measure for System This is a future solution to standardize preparedness purposes. (NEMSIS) state and local interoperability of the Unless functional interoperability is EMS function using an XML-based established by way of this voluntary schema schema. This will allow records to be the nationwide collection system will have shared commonly. many of the problems now found in NFIRS.
121 Survey Data Sets
In addition to incident reporting, several organizations conduct surveys or studies related to fire fighting and disaster response. Table 8-2 presents most of the important study or survey efforts known to the Panel. Note, however, some survey data are not available for use until a significant amount of time after the events being reported occur. A more detailed description of each data set in the table is available in Appendix D.
122 Table 8-2. Survey Data Sets
Data Source Description Data Quality Availability In 2001 and 2005, FEMA/DHS High 2001 data are available; 2005 data commissioned a study in conjunction Out of the 26,354 departments mailed are pending publication. The study with NFPA to: (1) define the current role surveys, 46% responded; NFPA projects is conducted approximately every NFPA/USFA and activities associated with the fire from this large sample. four or five years. Needs services; (2) determine the adequacy of Assessment current levels of funding; and (3) provide These are mostly fire-oriented questions. a needs assessment to identify shortfalls. These Assessments are discussed in the report, Chapter 4. The AFG grants management system Medium-High DHS has eighty preset reports and requires that all applicants answer Self-reported with oversight by regional access to ad hoc reporting through a detailed questions about their jurisdiction representatives for a small percentage. data warehouse. and request. Some of these data are AFG entered via standard web-based electronic There is almost no linkage between Researchers could ask the managers “eGrants” forms. Other information is included as applications and the performance reports that of the GMS database to produce Grant searchable text in the submissions. are submitted in the web-based fashion collections based on searching for Management supported by the GMS. text within applications and System performance reports (for instance a search could be run on the term “risk” to find all text blocks that contained this word). Tracks injuries requiring an emergency High Available for free download (in room visit. An interactive database Uses a relatively large, representative sample Wisqars). CPSC National system that provides customized reports of emergency rooms. Electronic of injury-related data. Less than a year lag (2005 data will Injury Reporting categories are not specific enough be available in October, 2006). Surveillance A direct sample of 100 emergency to identify fire as a cause (there is a code for System departments is extrapolated to national fire and burns, but it would includes scalds). (NEISS) statistics. It supports analysis by category, gender, race, age.
123 Data Source Description Data Quality Availability Cause of death information reported from High Available for free download (in coroners nationwide as a census. Reported from all coroners, based on death Wisqars). CDC National certificates. Vital Statistics State vital statistics agencies forward these on a periodic basis in batch form. Data lags 2.5 years because states report to the National Center for Health Statistics. This source includes heart attacks and High Finalized and put in the public USFA stroke within 24 hours of an incident Data collected from news reports, fire domain by April/May of the Firefighter (while NFPA does not). departments, and states. previous year, followed by year- Fatality to-date revisions each month. Database News reports and other individual reports Comprehensive and reliable. are provided. A count and periodic analysis of the High Available to NFPA members. NFPA leading causes of death among Comprehensive and reliable. US Firefighter firefighters by polling all known fire Fatalities departments. Does not include EMS-only fatalities. Collected most recently in 2003 (done Medium 2003 version has been accepted CDC Injury about once per decade, previous version For example, over ninety percent of people for publication, pending release. Control and in 1994). Includes questions about smoke report they had a working smoke alarm, but Risk Survey alarms; working alarm in the house, on surveys could over-report. (ICARIS) every floor; fire escape plans. Gathers data on fire-related injury risk Medium Available for free download factors, such as smoking, and protective It consolidates analysis of data from several (http://www.firesafety.gov/progr CDC National factors, such as regular maintenance of primary sources. ams/). Fire Risk smoke alarms. It is intended to help state Factor Survey and local fire-injury control programs set priorities and guide assessment of interventions and program performance.
124 Community Ratings Data
There are almost as many ways to assess a community’s capability to respond to emergencies as there are communities, and each of these requires data. The Community Risk Assessment (CRA) described in Chapter 6 is among the most comprehensive descriptive approaches to enabling a community (not necessarily a fire department or EMS organization) to systematically assess what risks it faces and determine how it can mitigate those risks. Incident-related data are often used to inform these determinations.
The three community rating models below focus more on the prescription of policies, procedures, practices and resources based on some benchmark ideas underpinning the approach: (1) the Insurance Service Office has developed a third-party classification approach to the rating of community fire safety used by insurers; (2) the Center for Public Safety Excellence, led by fire chiefs and city managers, has developed an Accreditation Model that combines self- assessment with peer review and extends to all of the roles fire departments serve; and (3) the Commission on Accreditation of Ambulance Services, representing a coalition of professionals, has defined a “gold standard” and accreditation process for ambulance services.
Insurance Service Office
The Insurance Service Office (ISO) supplies data, analytics, and decision-support services to organizations including insurance companies, finance and real estate firms, health services, and government to help them measure, manage, and reduce risk. ISO collects information useful for insurance underwriting, including assessments of public fire protection, flood risk, and the adoption and enforcement of building codes in individual communities. ISO performs the assessments as a service to the insurance industry and does not charge a fee to the communities. One of ISO’s programs, the Public Protection Classification (PPC) program, assigns a numeric rating of a community’s investment in fire mitigation, defined by ISO to include “fire-protection features only as they relate to suppression of fires in structures” and used as predictor of future fire losses.89 PPC covers more than 44,000 fire districts across the United States. Another ISO program is Building Code Assessments (BCEGS), which emphasizes mitigation of losses from natural hazards by assessing a community’s commitment to building-code enforcement. Both PPC and BCEGS are used to help companies and insurance commissions determine appropriate premiums for insurance coverage.
Related to data and performance measurement, most insurance companies use PPC information to help establish fair premiums for fire insurance—generally offering lower premiums in communities with better protection. Many communities use the PPC as a benchmark for measuring the effectiveness of their fire-response services. The main focus of the PPC measures is the capability of the fire service to extinguish a residential structure fire and prevent it from spreading to other structures. The PPC program is also a tool that helps communities plan for, budget, and justify improvements. On average, per $1,000 of insured property, communities in the worst classification had commercial property fire losses more than three times as high as communities in the best classification. And communities in the worst classification had
89 ISO, 2006.
125 homeowner fire losses more than twice as high as communities in the best classification.90 However, the PPC program does not focus on prevention or protection activities that can often have a significant influence on the community’s fire incidence rate.
ISO gives insurers BCEGS classification information for determination of premiums. But, while the PPC score applies to all structures in the community because all are affected by such factors as water supplies and fire department staffing, the community’s BCEGS scores in any given year are permanently assigned to the buildings awarded certificates of occupancy in that year. In other words, if a community had a BCEGS score of 3 in 2002, all buildings awarded certificates of occupancy in 2002 are thereafter assigned a score of 3, even if the community improves to a level of 2 the following year. ISO reassesses each community at least every five years, but classifications for existing buildings will not change as a result of a community’s reclassification.
While much of the data ISO collects is proprietary and for use by paying customers—typically insurance agencies—summary data are available online and reports are provided to each assessed community. (See Appendix E for more information and graphs that describe the distribution of data produced by these ISO programs.)
The “Accreditation Model” of the Center for Public Safety Excellence
The International Association of Fire Chiefs and the International City/County Managers Association created the Center for Public Safety Excellence (CPSE) in 1996 (formerly called CFAI). The CPSE provides a self-assessment process for fire and emergency service agencies to rate themselves by filling out an extensive application. The self-assessment is then reviewed by the CPSE through an off-site peer review and an on-site review. The certificate of accreditation is valid for five years, contingent on annual compliance reports.
CPSE certification is based on three questions: (1) Is the organization effective? (2) Are the goals, objectives and mission of the organization being achieved? and (3) What are the reasons for the success of the organization?
The CPSE assessment covers ten organizational aspects: (1) Governance and Administration; (2) Assessment and Planning; (3) Goals and Objectives; (4) Financial Resources; (5) Programs; (6) Physical Resources; (7) Human Resources; (8) Training and Competency; (9) Essential Resources; and (10) External Systems Relations.91
While the staff commitment to accreditation is a major investment (estimated by CPSE to take 700-1000 staff hours per applicant department), the CPSE contends it pays off by methodically improving the capabilities and professionalism of the department. Its application process is being merged with the ISO application process, eliminating redundant data gathering.
Related to data and performance measurement, CPSE offers software called VISION that allows fire and emergency service agencies to collect data in a single place, enter data online and import existing data from third-party applications. This data-base application allows CPSE to serve as a
90 Ibid. 91 CFAI, 2000.
126 central data base of benchmarking information for fire and emergency service agencies nationwide. The software allows agencies to update and exchange community data with ISO and with all participating organizations. It provides a means to develop a comprehensive community risk hazard analysis, comparative performance data with other accredited communities, and incorporate the data into a GIS-based integrated risk/deployment model.92
The Commission on Accreditation of Ambulance Services
The Commission on Accreditation of Ambulance Services (CAAS) was established to encourage and promote quality patient care in America's medical transportation system. The CAAS Board of Directors includes representation from the American Ambulance Association, American College of Emergency Physicians, International Association of Fire Chiefs, National Association of EMS Physicians, National Association of EMTs, and the National Association of State EMS Directors, with liaison representation from the National Highway Traffic Safety Administration. CAAS provides both a set of standards as well as an accreditation process related to medical transportation services.
The intent of the CAAS is to define a “gold standard” for the medical transportation industry of a higher caliber than is typically required for state or local licensing. These standards are grouped into the following categories: Organization, Inter-Agency Relations, Management, Financial Management, Community Relations and Public Affairs, Human Resources, Clinical Standards, Safe Operations & Managing Risk, Equipment & Facilities, Communications Center.93
The CAAS accreditation process includes five steps: (1) self assessment; (2) application; (3) assessment in which the Commission conducts an off-site review of the submitted application before scheduling an on-site review; (4) deliberation to determine whether the service meets all requirements is made by an independent, impartial Panel of Commissioners; and if successful (5) accreditation.94
Related to data and performance measurement, CAAS focuses on allowing an accredited agency to be confident that it is delivering a high standard of service, and to prove it through its written or documented procedures, systems, and reports. Achieving or renewing accreditation must be performed every three years. This periodic review requirement could be used by the AFG program to produce a metric.
Other than the published list of accredited agencies (approximately 100 organizations nationwide), there is no information provided for use by participating agencies or programs such as AFG to assist them in obtaining normative and comparative data sets. This is most likely due to a current lack of well-accepted medical guidelines for EMS or a known set of performance measures or benchmarks that are risk-adjusted by types of patient and their condition. A performance-driven management strategy for the AFG program could include encouraging more applications that address this gap for EMS.
92 CPSE, 2006. 93 CAAS, 2006. 94 Ibid.
127
SELECTING AFG PERFORMANCE MEASURES
To select appropriate performance measures, one must first identify the behaviors that the AFG program is seeking to influence and then understand how they may be influenced by the program. Those relationships are presented below as the AFG program’s “logic model.”
Because the AFG program is part of a larger array of national and community efforts to address multiple risks, it is important to place it in this broader framework. Finally, it is necessary to understand the program’s strategic direction (see Chapter 6).
The AFG program’s current measurement system is primarily focused on the input phase of the logic model. As data presented in Chapter 3 attest, descriptive metrics produced by program management cover the number of grant applications, grant award distribution patterns, and demographic characteristics of the grantee communities and departments. The recommended, approach is for the AFG program office to gather metrics across the logic model.
Performance Logic Model
“All human services face the tension between craft and science, between wisdom of experienced practitioners and the science that often contradicts that wisdom.”95 The same tendency toward management by wisdom was common in the commercial sector too before modern data management tools and intense competitive pressure forced managers to adopt the most effective tools available to allocate their limited investment dollars.
Any program—at least any well-designed program—embodies an explicit or implicit logic that relates the application of its resources to specific activities and the subsequent intermediate and long-term outcomes expected. It helps when selecting a set of key performance measures to make this logic model explicit. To do this, strategic management can be based on a fundamental conceptual framework, or logic model, about how an organization can succeed in light of its internal strengths and weaknesses, its opportunities and threats and the environment in which it exists. Chapter 6 discussed the strategic choices facing the AFG program. Measures presented in this chapter would allow program managers, policy makers, and stakeholders to assess the extent to which any chosen program strategy is contributing to and helping achieve the organization’s goals.
In the case of AFG, our best understanding of how the program is intended to work is summarized in the Performance Logic Model (see Figure 8-2). This model describes the connection of each component of program strategy to the real world of all-hazards prevention, protection and response as well as the roles and activities the eligible organizations can address related to preparedness. Components of the model help us organize the selection of measures of the success or failure of each element in such a way that when the measures are looked at together, progress (or barriers to progress) toward the goals are likely to be more apparent.
95 Sherman, 2000.
128 The examples shown at the bottom of Figure 8-2 derive from the draft DHS list of Target Capabilities. This initial draft was focused on the response segment of the taxonomy, so these examples relate to response. The Panel is, of course, recommending more attention to Prevention and would have preferred to have used examples from this segment instead. The prevention view for this logic model and its use of the Capabilities and Task Lists should be under development by the Department now. These can assist the AFG program in moving toward new measures of performance success. (Using measures of community capabilities to assess regional risk, for example, is discussed in Chapter 9.)
A logic-model is usually built by working backwards from the end-outcome goals. Reading left to right on the logic model, the AFG program should see its funds (an input) as helping a grantee to have a capability whose activities produce an output that will have some influence on a desired outcome of the grant-funded project.
The last phase or part of the logic model relates to the overall or aggregate view that is primarily the subject of the evaluation program discussed in Chapter 10 and referred to in this figure as being Measures of Strategic Use, or Impact. The Panel believes that the AFG program should concentrate most of its management attention for the near term on the measurements related to the intermediate steps. It is extremely difficult to relate any one grant project or even the aggregate of all the grant projects in one year to the end-outcome goals because of the size of the program and its essential nature of being based on grantee self-determination.
The AFG program can, for each part of the preparedness taxonomy, focus on what activities lead to the best intermediate outputs for accomplishing a desired outcome of a proposed project. The Panel has also recommended some changes in the overall strategies for the program that should influence the type of grant purposes that are expected to have a positive influence on the end- outcome goals.
129 Figure 8-2. Fire and Emergency Response/Preparedness Logic Model
Definition: Definition: Definition: Definition: Definition: Definition: • Resources • The ability of a • The value-added • A tabulation, • An assessment of • Intended direct or (financial, physical, program to work performed by calculation, or the results of a indirect effects or intellectual or raw perform an activity the program recording of program compared consequences materials) that to effect an output resources. activity or effort to its intended resulting from a contribute to a based on that can be purpose. program. program’s capability available inputs, Examples from expressed in a quantitative or to effect a desired skills, capacity DHS Universal Proposed: Proposed: output. and authority. Task List (UTL): qualitative • Civilian Casualty per The Intersection of: • Develop and manner. Resident/Worker • Community Risk Examples: Examples from Maintain Plans, • Civilian Casualty per Assessment • Tax revenue DHS Target Procedures, Examples from Fire •HSPD-8 (“The • Other local gov. Capabilities List: Programs and DHS UTL: • Fire Personnel Goal”) • Time to establish payments (e.g., • Firefighting Systems Casualties per • AFG Legisltive matching grants) Operations/ • Develop and incident Firefighter Mandates command • Other State gov. Support Maintain Training • Fire Personnel payments • Triage and Pre- and Exercise • Time to request Casualties per Fire local mutual aid • Fund raising Hospital Programs • $ Fire Loss per • Volunteerism Treatment •Transport • Time for first fire $ Protected suppression • Federal grants • WMD/Hazardous • Conduct fire • $ Loss per Fire • Donated/used materials education and life resource to arrive on scene equipment response and safety training and Decontamination education programs
Sources: The definitions of Output, Outcome and Impact are from OMB Primer on Performance Measurement; others adapted from Developing Performance Measures for the Community Development Block Grant Program, National Academy of Public Administration, 2005 and the Target Capabilities List of the National Preparedness Goal (DHS, 2006).
130 Measures of Impacts or Strategic Use—Impacts refer to the program’s contribution to important outcomes, including its influence on people, communities, neighborhoods, and community and national preparedness for all hazards. The assumption of Congressional lawmakers and those who administer the AFG program is that it can make an important contribution to public safety and preparedness in individual communities and for the nation as a whole, thereby reducing firefighter deaths, reducing property losses, and reducing death and injury of civilians from fire and other hazards. Impact measurement is more difficult, but more useful to the public policy debates, than output measurement. It requires rigorous, carefully designed research to separate the effects of the program and its uses from other influences on the outcomes being addressed. The possible focus and design of an AFG program evaluation are discussed in Chapter 10.
Outcomes—Outcomes are intended benefits produced for the citizens or other stakeholders of a program. Prevention outcomes include reducing the actual number of fires or medical emergencies (including unreported incidents) in a community over a period of time. Broader or longer-range outcomes that the program seeks to influence include reducing the number of civilian deaths and injuries from fire, reducing firefighter deaths, reducing economic losses from fire and reducing deaths and further morbidity or condition in medical emergencies.
Activity Outputs (Intermediate Outcomes)—An activity is work and it should cause something (an intermediate output) to happen that can be identified often in discrete ways. There is no assessment at this point of whether the output is useful to the public or aligned with a stated goal of the program. Outputs can be counted. Response outputs include response times, time-to- contain, and the number of firefighters who arrive on scene in the first vehicle. Prevention outputs include the percentage of structures that have a pre-fire plan on file, the number of inspections performed, the number of children taught fire safety skills, and the number of smoke detectors handed out.
Activities—Activities leverage capabilities and inputs to serve the public. Response activities include dispatching teams and fighting fires. Prevention activities include developing pre-fire plans, inspecting structures, and handing out smoke detectors. In fact, the Act authorizing the AFG program lists specific activities that may be funded.
Capabilities—The inputs mentioned above have no value without the capability to use them to perform the activities required to serve the public. In that regard, capabilities can be considered an intermediate outcome. For example, fire departments work to increase their response capabilities through training and certification efforts, maintaining vehicles and acquiring equipment; they improve their prevention capabilities by training fire inspectors, developing smoke detector or other prevention programs, and through public awareness campaigns. Accreditation and rating organizations, such as ISO, attempt to quantify or grade capabilities.
Inputs—Fire departments and EMS organizations derive their resources from multiple sources, including local taxes, voluntary contributions, fees, state grants and a variety of federal programs. Inputs also include the intellectual capital of the workforce and the manpower of volunteers. Other significant inputs for emergency responders include donations and real estate as well as equipment and vehicle assets that inherited from other, more well-equipped
131 jurisdictions. The priorities of the AFG program, and strategic approach it undertakes to act on them, should also be considered inputs in the logic model.
Potential AFG Performance Measures
It is difficult to manage what is not measured. The Panel considered a number of ways to measure performance, looking across the logic model.
Table 8-3 presents metrics that were proposed during stakeholder interviews and focus groups and recommended by the Panel for consideration by the AFG program. It also relates these measures to the strategy choices discussed in Chapter 6. Note the many Fire Safety measures found at the end of this table. These have been the predominant focus of the program to date.
Table 8-3. Strategies, Proposed Measures, Logic Model Notes
Basic Strategy Logic Model Proposed Performance Measure Supported Notes (See Chapter 6) Accomplishment of the fire-related UTL performance metrics National (numbers of organizations and number of UTL metrics that are Outcome measure Preparedness positive) Accomplishment of the EMS-related UTL performance metrics National (numbers of organizations and number of UTL metrics that are Outcome measure Preparedness positive) Accomplishment of the highest-priority UTL performance metrics National (as identified by both community risk assessments and DHS Outcome measure Preparedness internal prioritization) Number of community risk assessments (number of AFG National applications that identify a CRA; number of CRAs where a fire Capability Preparedness department is the identified lead organization—should cover all Risk Measures hazards implications including EMS and fire) National Number of AFG applications from high-risk areas as defined Input Preparedness under the Homeland Security Directives National Number of personnel educated and trained on NIMS and ICS Preparedness, Outcome measure Regional Cooperation Accomplishment of more of the prevention and protection aspects Input Prevention vs. of the Preparedness Goal and Uniform Task List (weight Capabilities Response accomplishment of prevention and protection ones higher than Activities accomplishment of the response metrics) Level of public awareness and level of education about risks and Prevention vs. how they should respond to an incident on their own and in Output measures Response cooperation with fire departments (public surveying instruments covering locations where grants were made, and national norms)
132 Basic Strategy Logic Model Proposed Performance Measure Supported Notes (See Chapter 6) Prevention vs. Rate and percentage of fires that are “relatively preventable by End-outcome Response inspection” (Urban Institute & NFPA, 1976) measure Fire rate in inspected households versus the fire rate those same Outcome Prevention vs. households prior to the inspections and those not inspected (UI & End-outcome Response NFPA, 1976) measure Change in fire rate as the time since an occupancy’s last inspection Outcome Prevention vs. increases for occupancies covered by a program of regular End-outcome Response inspections (UI & NFPA, 1976) measure Increase in the number of AFG applications addressing areas of Social Equity Input known socioeconomic or other demographic disparities Reductions in public death and injury metrics for certain End-outcome Social Equity socioeconomic or other known demographic disparities (fire) measure Reductions in public death and injury metrics for certain End-outcome Social Equity socioeconomic or other known demographic disparities (EMS) measure Population covered (higher coverage means funds are being End-outcome Social Equity targeted to a calculation of risk) measure Numbers of mutual aid agreements that are regional in character Regional End-outcome (numbers that contain agreements of a desired type, like medical Cooperation measure surge or dispatch backup) Regional Increased proportion of grant applications that are regional in Input Cooperation character Regional Number of personnel educated and trained on NIMS and ICS (see Cooperation for the National Preparedness Strategy) Outcome National Preparedness Emergency Reduced number of public deaths (EMS) (simple magnitudes and End-outcome Medical Response per incident figures) measures Emergency Number of EMS personnel deaths End-outcome Medical Response measure Reduced number of public injuries and reduced severity of injury Emergency (EMS) (simple magnitudes and per incident figures—need much End-outcome Medical Response more evidence-base and more measures here than “witnesses measure cardiac event) Reductions in metrics on public injuries and reduced severity of Outcome Emergency injury for certain socioeconomic or other known demographic End-outcome Medical Response disparities (EMS) (see Social Equity Strategy) measure Number of EMS personnel injuries, severity of injuries (stratify Outcome Emergency causes, including bystander injuries when at incident site) End-outcome Medical Response measure Response time (EMS) (this should be prioritized more than it is, Capability Emergency but we need more evidence base regarding the top 15 legitimate Activity Medical Response EMS calls) Output
133 Basic Strategy Logic Model Proposed Performance Measure Supported Notes (See Chapter 6) Number of personnel trained to certain standards (EMS) (can Capabilities Emergency emphasize the different levels of training and certification) Activities Medical Response Output Emergency Number of organizations who meet “cover standards” (EMS) (see Capabilities Medical Response response time items listed below) Improvements in Needs Assessment (EMS) (note Congress Emergency Capabilities recently directed the accomplishment of a Needs Assessment for Medical Response Activities EMS; the one done by NFPA is mainly focused on fire) Increase in number of public responder successful interventions Outputs Emergency (EMS) (see the severity of injury and other EMS outcome Outcomes Medical Response measures; Prevention strategy) End-outcome measure Number of organizations working toward or maintaining the End-outcome All Strategies highest levels of accreditation (or number of applicants to AFG measure who are working toward or maintaining this) Leverage of state and local government and other investments— Input All Strategies quantities of added resources both financial and otherwise Capabilities Reductions in the average age of equipment, apparatus and Inputs All Strategies vehicles. Outcomes Improved statistics on volunteerism—(more overall, more of Inputs All Strategies certain types, better skilled, more active) End-outcome measure Increase in number of public responder successful interventions Outputs (fire) (avoided response, also see Prevention) Outcomes Fire Safety End-outcome measure Reduced number of public deaths (fire) (simple magnitudes and End-outcome Fire Safety per incident figures) measure Reduced number of public injuries and reduced severity of injury Outcome Fire Safety (fire) (simple magnitudes and per incident figures) End-outcome measure Number of firefighter deaths End-outcome Fire Safety measure Number of firefighter injuries, severity of injuries End-outcome Fire Safety measure Improvements in Needs Assessment reported change in the Capabilities Fire Safety aggregate (fire) Activities Response time (fire) (note: prevention and protection measures Fire Safety Output ought to be emphasized over this) Fire Safety Number of personnel trained to certain standards (fire) Outcome Fire Safety Number of organizations who meet “cover standards” (fire) Capabilities
134 Basic Strategy Logic Model Proposed Performance Measure Supported Notes (See Chapter 6) Usage numbers reported by grantees related to the purposes of a grant (how many times was this training put to use on an incident, Output Fire Safety how many incidents involved this vehicle—compare these to the Outcome levels proposed in the application)
Table 8-4 presents a list of basic profile or descriptive information and data that could be related to an organization who is applying for an AFG grant that would be helpful in the evaluation of the program or in the study of the linkages and performance across the logic model. Much, if not all, of this is already contained within the grant management system now. Again, this list reflects stakeholder inputs and the analysis of the logic model done as part of this study.
Table 8-4. Other Descriptive Data Elements
Basic Strategy Descriptive Data Element Supported All Strategies The method of staffing All Strategies The types of emergency services being provided All Strategies The types of non-emergency services being provided All Strategies Response time goals All Strategies Size of the area being protected in terms of square miles All Strategies Permanent population of the area being protected All Strategies The number of fire stations in the system All Strategies The number of engine companies in service All Strategies The number of truck companies in service All Strategies The number of rescue squads in service All Strategies Description of level of activity in public education All Strategies Description of level of activity in training and education All Strategies Description of level of activity in fire prevention All Strategies The number of personnel in the table of organization All Strategies Current ISO Class (PPG and/or BCEGS) and dates of assignment All Strategies Current annual workload of emergency calls by type Confirmation of participation in various state and federal fire and All Strategies emergency reporting systems, plus tracking numbers (DUNS, NFIRS) All Strategies Current budget of personnel and capital investment and improvements
Relating AFG Outcome Measures to National, End-Outcome Measures
AFG grants are inputs to recipients and this is seen on the far-left side of the logic model. The grants are small. For most suburban and urban jurisdictions they represent a small input whose contribution alone can not be separated from all the other inputs for the purposes of connecting
135 these to the strategic or end-outcome measures. Nonetheless, the AFG program has a responsibility to collect data that can help them understand the logic model as much as possible and the relationship of the steps along the way to these end-outcomes. The AFG program also needs to adopt targets for end-outcomes to guide the development of the program over time.
Data gathered through the grant application, performance reports, and closeout processes only describe a grantees logic model through its output and outcome steps, if requested to do so by the agency. The metrics and data sources listed in Table 8-5, however, are recommended based on their availability, reliability, timeliness and relevance to the end-outcome goals of the AFG. These recommendations are based on available data sources as of the time of this report; however, because these sources are outside the direct control and resources of DHS, the findings may need to be adjusted if characteristics of these sources change.
Table 8-5. Recommended Available Data for National Outcome Measure Baselines
AFG Goals Metric Data Source Reliability Timeliness Measured Reduce Number of Civilian NFPA Fire Loss in the High Medium Civilian Fire Deaths in the United States During 2005— (Survey) (<1 year Deaths Home Full Report. (Available at delayed) http://www.nfpa.org/assets/fi les/PDF/OS.fireloss.pdf) Reduce U.S. Residential CDC, National Center for Very High Low Civilian Fire/Flame Deaths Injury Prevention and (Census) (1.5 years Deaths and Rates per Control. WISQARS Injury delayed) 100,000 Mortality Reports All Races, Sexes, (Available at Ages (Place of http://www.cdc.gov/ncipc/wi Accident = Home) sqars/default.htm) Reduce Overall Fire/Burn CPSC, National Electronic High Medium Civilian Nonfatal Injuries Injury Surveillance System (Sample) (<1 year Fire- Rates per 100,000 (NEISS) (Available at delayed) Related http://www.cdc.gov/ncipc/wi Injuries sqars/default.htm)
Fire-related data can generally be translated into outcomes, but there are additional challenges in doing so for EMS. While the fire department or non-affiliated EMS organization may be able to report on their portion of the emergency medical care process, the patient-centered outcomes of those actions are not readily available to the EMS organization, for several reasons. First, it should be recognized that EMS personnel face a different situation from emergency departments and, as is true across the clinical community, the diagnosis made by one may not in fact prove to
136 be accurate once further testing and assessment efforts are performed. EMS would need to know such facts to analyze the relationship of what they do to the patient-outcomes. Hospitals track outcome measures for patients (such as the injury data reported eventually into NEISS), but hospitals and EMS organization do not typically exchange data related to the patient and as it relates to the EMS organization’s activities and outputs. Accepted clinical guidelines do not universally exist. Finally, EMS records must be matched with hospital and coroner records to tabulate the outcomes and there can be substantial time-lags while the confidentiality provisions of the Health Insurance Portability and Accountability Act (HIPA) create a hurdle for those who would like to accomplish this view of EMS and the continuum of care. The Institute of Medicine (IOM) recommends that lawmakers “amend patient confidentiality regulations to allow, under strictly defined circumstances, out-of-hospital and emergency department records to be linked with longitudinal data on patient outcomes.”
FINDINGS AND RECOMMENDATIONS
The Panel finds that:
• AFG’s current measurement system is primarily focused on the input phase of the program’s logic model.
• AFG has not used a systematic approach to select appropriate performance measures.
• Data sources related to fire, EMS and national preparedness have limitations and all of these need to be improved to serve the recommended program.
• Patient-centered outcomes of EMS activities are not currently reported in any consistent way for a number of reasons. Development of NEMSIS may make useful data available to the AFG program for its use in developing ways to measure the successful use of the grant funds to improve EMS.
• Outcome goals or purposes of grant-funded projects as represented in the grant applications are not connected through performance reporting guides to permit AFG to obtain information and data on outputs, outcomes, or possible end-outcomes.
Therefore, the Panel recommends that the AFG program:
• Shift in the near term attention for the setting of strategic objectives to the intermediate steps of the logic model until the front-end of the preparedness taxonomy is understood and the program criteria are changed to reflect the recommended strategic directions (the Department also needs to execute on the evaluation framework offered in Chapter 10);
• Revise its grant application, performance reporting and close-out guides so they are consistent with one another and produce information and data appropriate to the agencies strategic, evaluative, and management needs
137 without unnecessarily burdening the grantees—a focus on the intermediate logic steps should mean that grantees should for the most part only need to report on capabilities, activities, outputs, and outcomes they propose to accomplish (and the statutory and HSPD requirements); and
• Work with its partners to systematically improve the quality of data sources; this includes closer coordination among federal government agencies responsible for the development of NEMSIS and NFIRS to either integrate these approaches or ensure that they complement each other in an interoperable manner.
138 CHAPTER 9 PERFORMANCE-DRIVEN MANAGEMENT
As stated in Chapter 6, the Academy Panel believes the AFG program should consider new strategic directions to maximize its effectiveness. The directions recommended for consideration include increased emphasis on prevention over response, facilitating national preparedness for all hazards, improving social equity by targeting those most at risk as determined by each community, promoting regional cooperation, and working with others to support strengthened EMS capabilities and effectiveness. This chapter describes how the program can use performance information, as it moves in new directions, to introduce and manage these changes.
The recommended approach is intended to respect and reinforce community decision-making, which is a basic strength of U.S. fire and emergency response. The challenge for the AFG program will be to support a gradual shift in direction without losing major strengths of its current management approach, including industry driven priority setting and its well-respected peer-review process. The program should also maintain its adaptability to account for wide variation from place to place in operating scale, sophistication, and resources—which affects both communities’ abilities to apply for the AFG program and the most appropriate use of grant funding to meet local needs.
Information on performance can be used to manage in a different way. In the broadest sense, it allows managers to gauge their progress in achieving each strategic goal. This can lead to a different way of approaching basic management tasks. For example, criteria for awarding grants, monitoring and analysis of grantee performance, and other tasks of program administration can be geared toward maximizing the program’s effectiveness as measured by progress toward its goals.
This chapter provides recommendations for how to modify the grant application and selection process to gather and use performance data in a way that addresses community and national risks. It also discusses influencing capabilities and intermediate outcomes—the earlier stages of the logic model—to, over time, improve the ratio of benefits to costs, especially as more is learned about what local strategies are most effective in addressing local fire risks, emergency medical needs, and risks arising from other hazards for which fire departments and unaffiliated EMS organizations have responsibility.
OVERVIEW: A DIFFERENT MANAGEMENT APPROACH
The AFG program currently evaluates grant applications based on four equally weighted aspects of the submission:
139
1. Clarity of proposal, including the budget;
2. Financial need of the organization;
3. Benefits that would result; and
4. Extent to which the grant would enhance daily operations and/or how it will positively impact the organization’s ability to protect life and property.
Current scoring of potential benefits and enhancement to operations and ability to protect life and property are necessarily subjective, relying on the expert views of those with substantial relevant professional experience. The effect is to reinforce established wisdom about what investments will result in greater benefits and improvements. The alternative approach recommended in this chapter would reduce subjectivity and sharpen the focus by tying funding to specific risk assessments, how the grant will address specified risks by improving specific local capabilities, and the applicant’s own assessment of how these improvements will lead to greater protection of life and property. It also would revise reporting requirements in a way that ties back to the applicant’s original statements about expected results.
The recommendation, in brief, is to develop a capabilities- and results-driven system for managing AFG awards that uses DHS lists of critical capabilities and activities to:
1. Establish priorities for funding tied to critical capabilities;
2. Define criteria for ranking applications; and
3. Establish standards against which to judge the program’s performance and assess its results.
As discussed in Chapter 6, a local CRA can be used by the AFG program to link the priorities of each community with the priorities of the AFG program. The CRA methodology need not be demanding for smaller, less sophisticated applicants; the program or others can offer a simple checklist approach that is useable in those places, while others use a more sophisticated approach. This approach allows each community to base its application on its own assessment of where and when it faces its greatest risks. The applicant would then identify which emergency protection capabilities—training, vehicles, equipment, prevention—they need to build or enhance in order to address specific locally-identified risks. Using the same approach, communities that already have basic capabilities and resources can focus on improved preparedness for large-scale hazards or improved regional cooperation.
The grant application could be modified to first ask the applicant whether the grant request is based on a CRA; then if so, ask the applicant to explain how the requested resources aligns with the risks identified for the community. While the program should most likely assign additional positive weighting to CRA-based requests rather than mandating this approach, it would help the program better evaluate the grant’s expected results in terms of mitigating the identified risks in
140 a particular community. The same reported information could then be used by the program at a national level to assess impact relative to broader preparedness as specified in the National Preparedness Plan.
In addition to managing the program to meet its strategic goals, the AFG program needs to measure the effectiveness of various aspects of its management. Measures and their application to management are discussed, including ways to assess success in meeting the program’s data- gathering objectives, management of its grants, and overall efficiency and effectiveness.
The following sections discuss how the program can be managed to address each proposed change in strategic direction.
PREVENTION VS. RESPONSE
The Academy Panel agrees with most experts and stakeholders interviewed that, in many situations, increased emphasis on prevention or mitigation approaches will have greater benefits for public safety than further improvements in fire, EMS and other hazard response capability. Steps can be taken to target and measure the program’s progress in this strategic dimension. Gradually, as information is developed about which prevention approaches are effective under specific circumstances, the AFG program can take a more prescriptive approach to awarding grants—ranking applications higher if they propose a use that has been shown to produce positive results.
Currently, the program operates on the assumption that the individual applicant is best qualified to develop a program that will be effective for its community. According to the 2006 program guidance, “…we are allowing your grant proposal to include a number of related ‘activities’ to address all of your needs within a programmatic or functional area.” However, a more effective approach could be to let program management and the review panel, which together review thousands of applications per year, identify and indicate a funding preference for a much smaller selection of best practices based on the actual results realized by previous grantees. For example, Chapter 3 described a study of a smoke alarm give-away in a targeted area of Oklahoma City that resulted in a 74 percent decline in the injury rate per 100 fires (from 5.0 to 1.3). Would it be reasonable for the AFG program to determine, based on such results, that this is one best uses of funds in communities with demographics similar to those of the targeted area? If applicants instead used program funds in ways that achieved a much smaller reduction in injuries, some would say the flexibility afforded applicants came at too high a cost.
Building inspections are another example of a prevention approach that might be favored in the program’s scoring of applications where there is evidence it is likely to produce good results. In addition to providing information to determine the effectiveness of this and other prevention programs, the mere existence of an inspection program in a community may have positive effects on fire incidence. An Urban Institute and NFPA study of fire protection performance measurement states “…the knowledge that inspections are conducted periodically may produce a continuing level of fire prevention awareness and corresponding actions that does diminish as the
141 time since last inspection increases.”96 This effect should be validated by tracking the incidence rates before and after inspection programs are established; if an effect is found, it suggests some preference be given to communities that do not currently have such a program in place but will use the grant to establish one.
To encourage prevention efforts, the AFG program should consider modifying the grant application to include elements that demonstrate the applicant has an active and current program of community outreach and education. To the extent possible, this information should be reported by the applicant in quantifiable terms, including elements such as the numbers of persons contacted, number of events conducted or the number of media releases provided or articles published. The applicant review and selection process could then favorably weight applicants who can respond to the requested elements in a way that that demonstrates an active prevention outreach program.
The performance measures in Table 9-1 are recommended, and goals based on these measures can be used as targets for continuous program improvement.
The Panel recommends that the program should actively consider how best to encourage behavior targeting strategic objectives among applicants and prioritize high quality applications for the funding of prevention and mitigation capabilities, including public education in fire safety that is targeted at those groups and places where risks are highest.
Table 9-1. Prevention vs. Response Performance Measures
Strategic Objective Performance Metric Status Increase grants applications request Number of prevention Currently available funding for prevention programs applications. Increase prevention applications that are Percent of applications affirm/ Not currently available based on risk from a Community Risk describe how grant supports Assessments CRA. Address prevention and protection Number of applications Not currently available aspects of the Preparedness Goal and supporting prevention and Uniform Task List protection aspects of the Preparedness Goal and Uniform Task List
96 UI & NFPA, 1976, p. 18.
142 Strategic Objective Performance Metric Status Determine and fund the types of fire and Ask the applicant how many Not currently available EMS preventative programs are the most structure fires they had each of successful at influencing outcome the last three years? (Specify if this refers to the entire community or a targeted subsection) and how many structure fires they expect in the year following the award of this grant, sighting results of similar programs if possible. Fire and EMS incidence rates Not currently available reported in the closeout report Identify/create and publicize model fire Complete: yes/no Not currently available prevention programs that are grant Publicized: yes/no Not currently available eligible Percent of awarded grants Not currently available related to model program Increase public education campaigns Level of public awareness and Requires public being performed to increase public level of education about risks surveying instruments knowledge of fire and EMS risks and Knowledge about how citizens covering locations where responses can respond to an incident on grants were made, and their own and in cooperation national norms with FDs
NATIONAL PREPAREDNESS
The appendix of the “2006 Program Guidance for the Assistance to Firefighters Grant Program” states that “HSPD-8 directs the grant program office to assure that Federal resources are directed toward ensuring the Nation is prepared to prevent, respond, and recover from major events, especially the prevention and response to terrorist’ attacks. We believe it is possible to fund fire departments through the AFG in a manner consistent with HSPD-8, i.e., that will ensure that first-responders are prepared to respond to major events.”97 In order for the AFG program to test this proposition, it must establish a system to track grant applications, awards, and subsequent reported performance to the specific critical capabilities and activities the grants are intended to influence.
Such a capabilities-driven system would use DHS critical capabilities and activities to:
1. Establish priorities for funding tied to critical capabilities;
2. Define criteria for ranking applications (e.g., does the proposed project promise to produce a critical capability addressing an identified source of fire risk in the applying community); and
97 DHS, 2006, p. 45, italics added.
143
3. Establish standards against which to judge the program’s performance and assess its results. Each of these uses is addressed below.
The current AFG process sets and adjusts funding priorities through the annual criteria development process described in Chapter 7, involving a panel of designated stakeholders representing fire service organizations (and EMS organizations in the future) as well as DHS officials. The criteria and priorities are based on available data and the expert knowledge of the representatives and their organizations. A missing element in the current process is consideration for how best to equip the applicant fire and EMS departments in a systematic way that best serves the evolving strategic priorities of DHS across multiple DHS programs. For example, if DHS determines that a major threat to homeland security involves derailing mass transit, it could simply elevate the priority of funding to departments intending to affect activities “Res.B2a.2” and “Res.B2a.2.1.3” where the proposed metric to be influenced by the applicant is “Annual training is provided in mass transit fires (departments with subways and /or commuter rail or light rail).” By having DHS funding recipients and DHS administrators speak the same language, the program can enable alignment from DHS strategic goals through to the behavior that it influences among grant applicants and recipients.
To use DHS target capabilities and activities to rank AFG applications, the AFG grant management system must be adapted to link applications to the National Preparedness Goal framework. This should be a fairly straightforward undertaking that could use a set of drop- down choices for applicants to populate based on a four-level tree: Capability, Activity, Critical Task, Preparedness Measure. Each choice would further restrict the list of choices available for the next selection.
For example, an applicant could be requested during the application process to:
1. Select “Firefighting Operations Support” from among the relevant Capabilities (see list of Target Capabilities in Chapter 6); then
2. Select “Direct Firefighting Operations and Support Tactical Operations” from among the associated Activities; then
3. Select “Coordinate fire suppression operations” and “communicate internal incident response information” from among the associated Critical Tasks; then
4. Select the metrics they expect the communication equipment is intended to improve. They could choose “Time to establish incident command for firefighting operations.” Repeat the previous steps for the relevant choices in other activities, such as “Size Up (Assess Site)” and “Contain and Control” that would benefit from the intended use of the communications equipment.
The additional metrics that should be gathered and managed to during the grant selection process are shown in Table 9-2 based on an analysis of the current AFG application process. This table includes recommendations for additional questions to link criteria back to the strategic goals of
144 the AFG program, as well as for use in the performance assessment phase that would occur one year after the grant money is awarded, described in more detail in Chapter 10.
Table 9-2. Recommended Selection Criteria
Strategic Objective Selection Criteria/Metric Status To promote alignment with DHS Number of applications from high-risk Not currently among prioritized areas of national areas as defined under the Homeland available risk Security Directives Improve the numbers of The fire-related and EMS-related UTL Not currently organizations and number of UTL performance metrics listed in Table 9-3 available metrics that are considered below. “positive” Improve UTL performance metrics Number of the highest-priority UTL Not currently identified by both community risk performance metrics improved to above available assessments and DHS internal as threshold levels highest priority Maximize expected outcomes per Have applicants report the top several Not currently dollar, aligning with national DHS capabilities, activities, critical tasks, and available priorities and focusing applicants preparedness measures that this grant is on expected outcomes. intended to improve for your community. (Each choice would further restrict the list of choices available for the next selection.) The following apply to each category of grant: Training, Firefighting Equipment, Personal Protective Equipment (PPE), Wellness-Fitness Programs, Modification of Facilities, Vehicle Acquisition (EMS and Fire), Fire Prevention/Safety Award grants that address high Ask the applicant which high-priority Provide priority risks for each community risk(s) will benefit from this grant dropdown of the five previously entered values. Ask the applicant to discuss how the Add narrative grant supports the highest priority risks section in your community Award grants by considering which Ask the applicant to enter the relevant Not currently will maximize cost-benefit metrics related to the risks they available identified, both current and expected values one year after the requested money is fully spent. Source: Based on Final Report on the Results of the FY 2006 AFG/SAFER Criteria Development Meeting (DHS, 2005).
Each activity in the National Preparedness Plan has associated metrics that the AFG program can reuse. Additionally, the 2007 federal appropriations bill requires DHS is to enter into a memorandum of understanding with the Emergency Management Assistance Compact for the
145 purposes of “developing standards for deployment capabilities” and for the “credentialing of personnel and typing of resources likely needed” to respond “to natural disasters, acts of terrorism, and other man-made disasters.” Those efforts combined should provide a comprehensive view of what are the most important measures of capability and preparedness from a homeland security and all-hazards preparedness perspective. As an example, Table 9-3 gives a list of the measures and metrics related to just one of the key capabilities in the National Preparedness Plan called “Develop and Maintain Plans, Procedures, Programs and Systems.”
Table 9-3. Sample of Metrics from DHS Target Capabilities List
Select Preparedness Measures for “Develop and Metrics Status Maintain Plans, Procedures, Programs and Systems” Plans include special risk requirements (e.g., alternative Yes/No Not Currently water supply) Available in GMS Regional and interstate mutual and automatic aid Yes/No Not Currently agreements have been established Available in GMS Department has specialized equipment for subway Yes/No Not Currently firefighting and search and rescue Available in GMS Department has preplans and maps for subway systems Yes/No Not Currently and standard operating procedures for dispatching Available in GMS companies to subway egress points to assist in search and rescue and evacuation (departments with subway systems) Department has to aerial units for deployment to roofs or Yes/No Not Currently high-rises (departments with high-rises) Available in GMS Department has sufficient stocked and garaged spare Yes/No Not Currently apparatus to surge total number of companies by 25 Available in GMS percent over normal first-line staffing needs (large cities) Department has procedures and protocols for coordinating Yes/No Not Currently protective action communication with at-risk population Available in GMS on-scene Department has procedures in place to ensure safety of Yes/No Not Currently operating personnel Available in GMS
The rating system must also take into account wide variations in the risks facing various fire and EMS departments and communities. Therefore, the next and inter-dependent step would be to link the capabilities that AFG funds are intended to enhance with a CRA that has been performed by the requesting community, described next.
SOCIAL EQUITY AND COMMUNITY RISK
As discussed in Chapter 6, the use of Community or Regional Risk Assessments is a critical linkage between the priorities of each community and the priorities of the AFG program. This approach allows those communities that have lagged behind in basic emergency protection
146 capabilities, training, vehicles or equipment to focus on catching up in the most cost-effective manner. At the same time, those communities that already have basic capabilities and resources can focus on improved preparedness for large-scale hazards or improving regional cooperation.
The program has the ability to change behavior of its applicants. It can promote the use of community risk assessments the same way it successfully influenced the adoption of incident command systems and the reporting of incident data into the National Fire Incident Reporting System (NFIRS).
The Federal Emergency Management Agency within DHS provides a guidebook called the “Multi-Hazard Mitigation Planning Guidance, March, 2004” to help communities and States develop community risk assessments.98 While DHS requires that mitigation plans be completed as a condition of receiving project grant funds under the Pre-Disaster Mitigation program, it is a valuable planning tool for all communities. The guidance documentation explains strategies and provides worksheets for identifying hazards, profiling hazards, assessing vulnerability (by identifying structures, estimating potential losses, and analyzing development trends) and considerations for multi-jurisdictional risk assessments.99
Some of the key factors that should be considered in a CRA include:
• guidelines and best-practice templates for CRAs (available from DHS)
• the community’s actual experience with the particular risk, including trends distilled from the analysis of data
• changes or anticipated changes in population relative to the built environment
• current capabilities of the fire department and EMS
• analysis of selected fire protection and EMS methods
• effect of the building and fire codes, code enforcement, and public education programs in relation to fatalities, injuries, and property losses from fire
• existence or availability of mutual aid, automatic aid, and other agreements with local communities
• defined level of the fire and EMS capabilities expected and the ability to provide expected levels
• ability to provide reliable communications—a prerequisite for the ability to deliver prompt fire, EMS, and technical rescue responses
98 The “Multi-Hazard Mitigation Planning Guidance, March, 2004,” also known as the “Mitigation Planning Blue Book,” is at http://www.fema.gov/library/viewRecord.do?id=1903. 99 DHS, 2004.
147
• identification of hazards that would precipitate a need for technical rescue activities
There are several ways a CRA could be included in the grant selection process, including:
1. Establish a hard rule that there must be a prior, adequately conducted CRA (definition of adequacy to be determined) as part of the initial application filtering step;
2. Give the application additional credit in the scoring process if there is sufficient demonstration that risks have been assessed, perhaps one level of credit for a full-blown systematic CRA and lesser credit for a lesser effort; or
3. Simply suggest that a CRA be used as input into the application process without any explicit credit given in the application assessment.
Because communities and emergency response organizations vary considerably in their available resources, the program would be advised against a hard filtering rule. The consequences of such a rule would likely disadvantage the very communities most in need of resources or inadvertently providing an incentive for cursory CRAs that are not integrated between community organizations and across neighboring communities. Therefore, giving additional weighting to those applications that use a documented CRA as the basis for determining their stated needs is likely the most prudent and effective approach to both encourage the desired behavior and take advantage of the information it provides the program.
Depending on which option is implemented, a linkage from grant application to the CRA could be made by first asking the applicant whether the request is part of a CRA; then if so, asking more detailed questions to assess the thoroughness of the CRA, followed by a question in which the applicant is asked to explain how they determined that the requested resources align with the risks identified for the community. The applicant should be informed that a copy of the risk assessment may be requested as part of the application assessment process (and these should be requested from a sample of applicants).
In communities that have conducted a FEMA National Incident Management Capability Assessment Support Tool (NIMCAST) self-assessment,100 this could be a second and complementary basis for the proposed use of AFG funds that would avoid redundant administrative work. Similar to a CRA, the NIMCAST is a tool “to evaluate [a] jurisdiction's ability to effectively prepare for, prevent, respond to, and recover from domestic incidents, regardless of cause, size, or complexity.”101 This may be an appealing alternative for smaller communities.
A final use of the CRA or other risk assessments described above would be to establish standards against which to judge the program’s performance and whether it mitigated the identified risks in a particular community that has received an award. If the steps described above are taken first,
100 For more information and an example of a completed NIMCAST, see http://www.fema.gov/nimcast/About.do. 101 FEMA, 2006.
148 this assessment is fairly straightforward. In the performance report that the applicant is required to complete one year after award (which suggested to be modified to one year after the funds are disbursed), the applicant could be asked to document the results of the grant online in a standardized format as follows:
• Please indicate whether the following capabilities, activities and tasks have been enhanced as a result of the grant award (provide a list of those associated with the original grant).
• For each enhanced capability, activity and task, provide the updated metric values for the measures included in the original application.
If the reported and verified metric now results in a value greater than or equal to the metric acceptable point in the DHS documentation (e.g., “equipment and apparatus are inventoried less than 30 minutes from the start of demobilization”), that same metric should not be funded again unless it is reported on a future application to be below that acceptable point.
The intended result of such an assessment procedure is to enable the DHS program management to document which critical capabilities were reported as deficient, which of these were funded, and the anticipated and resulting improvements in the metrics that bring fire departments and EMS into accord with the DHS preparedness plan minimum acceptable metrics. Beyond contributing valuable information to a periodic evaluation of the impact of the AFG program, such an analysis if expanded to other DHS programs could be used to compare the cost- effectiveness of AFG with other DHS programs in improving critical capabilities. And further still, if a particular capability increased in national priority for a certain geographic area (e.g., a catastrophic hurricane demonstrates the vulnerability to communities in a flood plains), contributors to elevating that capability should be elevated in priority among all DHS funding programs.
The Panel recommends that DHS use a national risk assessment process that analyzes the top risks reported on AFG applications collectively against DHS prioritized risks. This requires revising the closeout questionnaire to focus on the results that the grant was originally intended to effect and to enable the AFG program to analyze grantee results across categories of applicants and grants. (Recommended considerations and metrics to aid the program in performance- based management related to social equity are listed in Table 9-4.)
149 Table 9-4. Social Equity Performance Measures
Strategic Objective Performance Metric Status Encourage the development of CRAs Ask applicant if the community Not Available and use the CRA information to make performed a community risk grant awards that match high-priority assessment (CRA) and to discuss community needs based on populations how it was conducted. Applicant at risk. should be prepared to supply the CRA if requested. Obtain a list of the top priority risks, Ask the applicant for five highest Not Available matched with UTL, for reporting trends priority risks, in order (choose and matching with DHS priorities. closest items from the drop down list based on the Universal Task List from the National Preparedness Plan). Hold applicants accountable for Ask applicant to provide name, Not Available verifying CRA information and contact information for legislative encourage community planning efforts contact involved with CRA. Increase the number of AFG Devise national priorities for Not Available applications addressing areas where ameliorating disparity and review certain socioeconomic or other each year during the priority demographic disparities are known setting panel meeting Maximize the quantity and portion of Ask the applicant about the Currently population covered by AFG Grants population that the grant would available cover as an approximation that the funds are being targeted based on a calculation of risk
Regional Cooperation
As stated in Chapter 6, the Academy Panel believes that still greater emphasis on funding region- wide capabilities could increase returns from AFG spending both to national preparedness and to public health and safety. Promoting such cooperation is a traditional use of federal dollars and one that has been given increased emphasis in the AFG program in recent years. As suggested during the AFG Strategic Review Panel meeting, these performance measures and the corresponding program guidance will promote collaboration and relationship building among the emergency response organizations in neighboring communities during the application process, even for those grants not awarded.
The program, in conjunction with others at DHS, should further define a comprehensive and consistent approach to regionalism. As it is currently described in the AFG program guidance, any combination of communities can apply for regional AFG grants. This places a burden on the application evaluation panel to determine if communities are requesting grants that do in fact serve a regional interest. One way to formalize such an assessment is to encourage regional grant applications (or all applications) to be accompanied by a Regional Risk Assessment that
150 shows the resources are to be used to meet a broader need that cannot be met by any one community.
The Academy Panel’s recommendations about the use of the Capabilities and Task Lists offer the opportunity for the AFG program to apply this on a regional basis when looking at grant applications. The Panel believes it would be useful for the AFG program to identify applicants and projects that fill the gaps in the region, looking for “weak-link” so to speak. The Department can be proactive here too, marketing and advertising their interest in applicant organizations’ efforts to ensure that they are not the weak link in a region’s network of capabilities. Two ways to address a community with insufficient capabilities using AFG grants are: (1) to raise the capabilities of the “weak link” community, or (2) to use logistics to ensure there is a plan in place, such as a mutual aid agreement or a memorandum of understanding, supplemented by inter-departmental training exercises, to ensure that the residents and critical infrastructure across the region can be adequately protected.
Clearly there are scenarios in which a logistical approach is far more cost effective than supplying all communities with the capability to respond to all hazards however infrequently they occur. As DHS completes the taxonomy of the Universal Task List to support the National Preparedness Plan, the AFG program is encouraged to participate to ensure that the prevention and detection topics adequately provide capabilities, activities and metrics that describe a regional approach to all-hazards response that can be integrated into the AFG application and closeout processes to measure which communities are in fact risks to regional preparedness and to measure the extent to which those risks are mitigated by AFG grants.
To manage implementation of this strategic direction in the near term, several fairly simple modifications to the grant selection and reporting process are recommended (see Table 9-5).
151 Table 9-5. Regional Cooperation Performance Measures
Strategic Objective Performance Metric Status Promote public/private coalitions and Percent of awarded grants include Currently partnerships. “regional” indicator available in GMS Number of other jurisdictions involved in a regional project Structure in a way that brings local Percent of awarded grants that Not available governance and applicant closer include the name and contact together. information for governing bodies’ contacts. Align decision makers with motivated individuals in each community (i.e., provide people with a “hook to people with power” as recommended in the strategic review session). Maximize the numbers of mutual aid Ask applicants or survey regarding Not available agreements that are regional in the number of mutual aid character agreements that are of each desired type (e.g., medical surge or dispatch backup) Promote adoption and consistency of Number of personnel educated and Not available NIMS and ICS integration trained on NIMS and ICS
Emergency Medical Response
Despite the complex web of local and regional fire service/EMS configurations and relationships, funding EMS capabilities represent a significant portion of the AFG program, and needs to be managed in a way that maximizes its use in improving the effectiveness of local EMS operations.
While DHS, HHS, and DOT/NHTSA jointly consider how the federal government can best encourage a rationalization of first responses to medical emergencies and coordinate the development of incident databases, the metrics in Table 9-6 should be gathered to set a baseline for analyzing the relative effectiveness of current and future EMS grants.
152 Table 9-6. Emergency Medical Performance Measures
Strategic Objective Performance Metric Status Target training toward capabilities Number of personnel trained to Not currently certain standards—EMS (can be available used to emphasize the different levels of training and certification as discussed in Chapter 3) Maximize EMS coverage Determine the number of Not currently organizations that meet EMS “cover available standards” Reduce the “needs” in the aggregate Congress recently directed a Needs Not currently and for communities that receive Assessment for EMS be performed; available AFG grants the one done by NFPA is mainly focused on fire Increase the rate of successful Number of public responder Not currently medical interventions successful interventions—EMS (see available the severity of injury and other EMS outcome measures) Minimize EMS response time Develop baseline data regarding the Not currently top 15 legitimate EMS calls available
Program Strategic Management
Apart from estimating the AFG program’s contribution to the broader goals of all-hazard prevention, protection and response, performance-based management of the program needs to answer several questions that will help guide it toward improved effectiveness, efficiency and a greater return on its investment of public resources.
The Strategic Review Panel, at its October 6th planning session, suggested several strategic priorities for the AFG program. Some important questions that a properly designed independent assessment could answer are listed in Table 9-7 with regard to data/benchmarking and the overall grant management process.
153 Table 9-7. Recommended Measures of Strategic Management Objectives
Data/Benchmarking Strategic Objective Metric Status Leverage of state and local Quantities of added resources Not currently available government and other investments both financial and otherwise (See Chapter 10 for additional discussion). Improve statistics on volunteerism. Overall volunteers, more of Additional detail to be certain types, better skilled, more added to GMS active Establish Program Benchmarks for Benchmarks devised and Not currently available participation in the AFG program. published: yes/no Maximize usage numbers reported For example, how many times Not currently available by grantees related to the purposes was training put to use on an of a grant to be compared with the incident, how many incidents levels proposed in the application. involved the new vehicle (provide NFIRS reference numbers) Support research programs within Percent of awarded grants for Currently available in Fire/EMS to support continuous research GMS improvement and science based benchmarks. The Grant Management Process Strategic Objective Metric Status Identify, describe and publicize Publicized: yes/no Not currently available success stories and best-practice uses of the available resources. Improve the feedback loop to Process exists to explain rejection Not currently available unsuccessful candidates to criteria: yes/no establishing an understanding for Increased reapplication rate Available from GMS applicants for why they did not following rejection data receive grant, thereby increasing participation and improving the quality of requested grants. Build in more training on how to Decrease grants unused/misused Available from GMS manage the grant money after the data grant is awarded. Award and distribute grant money Percent of grants awarded per the Available from GMS awarded per the published published schedule data schedule. Award grants based on the set- Set aside requirements met/not Currently available, asides defined in the program met but not an active guidance documentation. management focus
154 Leveraging Best Practices
As discussed above, the program will identify innovative prevention programs and other best practices over time that will be applicable to numerous communities. It was recommended that the program review the outcomes reported in the closeout report to identify and prioritize likely best practices for analysis and publication. The program should seek best practices throughout the operations and management of fire departments, EMS and other emergency responders and encourage the adoption of those practices in other communities. This approach requires, in collaboration with partner organizations, identifying candidate practices, assessing them, and publicizing those deemed most valuable and widely applicable, using both prioritization of applications and other means. These other means include using the program’s resources to disseminate information about such practices and to reward those who adopt them to good effect.
The AFG program may be more effective in promoting adoption of promising practices if it partners with other national organizations. Such partners might be in a good position to offer awards to those submissions deemed to be the most promising. A national partner also might contribute information or help to identify best practices instead of starting new data collection.102 Regardless of which organization manages the repository, the AFG program could contribute by modifying its GMS to requests nominations. This can be done in two ways. First, when a grant recipient is describing the use of funds in a narrative format as required in the closeout process, they could be permitted to nominate the practice described in the narrative, thereby flagging it for review by program staff or others. Second, fire or EMS organizations could be allowed to access the nomination part of the GMS even when they are not in the grant application or closeout process. This would provide additional narratives related to best practices, even if not related specifically to the use of AFG grant funds. The nomination form could be similar to the one shown in Figure 9-1.
102 Although a quick review of the AFG program’s nine partner organizations’ web pages does not show an obvious model for this.
155 Figure 9-1. Sample Best Practice Nomination Form
NOMINATE THIS ACTIVITY AS INNOVATION OR BEST PRACTICE
NOMINATION TYPE (SELECT ONE): Innovation __ Best Practice __
STRATEGY SUPPORTED (SELECT ONE): Fire Prevention/Mitigation __ EMS Prevention/Mitigation __ Fire Response __ EMS Response __ Regional Coordination/Large-Scale Hazard Management __ Other______
NARRATIVE DESCRIPTION OF INNOVATION/BEST PRACTICE: ______
EVIDENCE OF SUCCESS: ______
Once nominations are received, they should be vetted by the program staff (possibly with a partner organization) to ensure clarity, applicability to other organizations and validity of the benefits claimed in the nomination. Follow-up conversations will likely be necessary. The goal is to determine the organizational or community characteristics to which the practice is applicable and provide a measure of its success.
Once a collection of best practices are identified and vetted, they should be publicized by the program and its partners. A future use could be to allow applicants to choose among the best practices as model programs they will enact with AFG funds, presumably associated with additional positive weighting during the application assessment. This is one strategy the program can use to accelerate learning and the adoption of promising innovations without increasing the size or number of its grants. Doing so multiplies the program’s potential influence over improving public health and safety and achieving other objectives.
Managing the Grant Process
The proposed grant selection process would have three basic steps:
1. Initial filtering into categories and scoring by the grants management system to identify those within the “competitive range;”
156
2. Independent Panel reviews of the competitive range applications to complete the scoring; and
3. Final agency decision-making considering the Panel’s rankings and the requirements to assure an appropriate distribution of the grant awards by type of applicant and community and to comply with other statutory limits.
Several areas of the process could be considered for revision. Perhaps the most important improvement for the purpose of strengthening AFG’s focus on performance would be establishing a more direct tie between the information provided in applications and that provided later in grantee performance reports.
Asking the applicant about both current metrics in their community as well as the expected improvements to those metrics as a result of the grant award would force the applicant to examine the relative value of the requested use of funds and allow AFG manager to compare relative benefits among all applicants. Asking those same questions again in the performance report will allow the program to hold grantees accountable for claims made in the application. At a minimum, grantees should be asked to comment on public deaths and injuries avoided as a result of the grant award (the grantee should be able to supply an NFIRS incident number for verification or additional analysis).
To date, the program has generally emphasized workload and financial management of the grant project in its GMS ahead of performance measurements needs or for use in an integrated evaluation plan. The current maturity of the GMS system creates opportunities for improvements in the process that can help the agency with its strategic questions and evaluation framework as discussed in later chapters. For example, other than a detailed financial reconciliation of the grant funds, the substantive portion of the grantee performance reporting requirement is in the form of a free-text narrative currently instead of a readily analytic format tied back to the data supplied in the original application. For instance, an applicant that projects a number for the public that will be served (like public deaths avoided) can subsequently fill in a box with their calculation of the number actually served by the use of the funds over the grant period. The narrative can further explain the evidence used to support the numbers shown and the differences between the proposal and actual performance.
All applicants are required to fill out an inventory of vehicles, but PPE and other equipment inventories not submitted. Adding additional inventory categories would provide a valuable tool for comparing firefighting capabilities and effectiveness based on equipment inventories. Additionally, sorting applications into groups from the same geographic region, as well as providing past awards from that region for reference, would enable the panel to determine if the grants are indeed serving regional needs in a rational way. The purpose would be to avoid, for example, awarding a ladder truck to two neighboring jurisdictions with few tall structures when a logistical mutual aid agreement would more effectively distribute that resource. This type of determination would also benefit from a community or regional risk assessment, as explained earlier in this chapter.
157 Panel reviewers are charged with looking at the supplied budget information from applications to confirm relative financial need of the organization. The applicant submits a “budget” for the grant project and a fiscal portrayal or “budget” information. All this needs much better explanations for applicants. But note that reviewers are not helped with some normative views of the applicant pool to help them assess financial need (e.g., a calculation of a normal budget amount for all applicants with similar call-volumes and type jurisdiction served).
The AFG program also needs to be able to report to Congress with some evidence in hand about the influence on applicants and grantees of statutory conditions such as the maintenance and non- supplantation rules, the matching funds rules, and the rules related to NIMS and NFIRS compliance. The agency should do an assessment in the aggregate about whether the non- supplantation rule is being met (and then move to compliance actions based on its findings and conclusions, if necessary). Also, because applicants are required to meet the maintenance requirements without supplanting local funding, it is insufficient to only require the current year’s budget. This might provide a valuable data set allowing the agency to extract budget data by applicant types year-to-year to see if there are trends and ratios of interest to the AFG’s strategies and selection (and to compare applicant representations from year-to-year too).
Also note that the program needs to focus some management attention on grants management processes. Current market research activities, for example, should be continued to understand recipient awareness of the program, their reasons for and rate of conversion to application action, and whether the program guidance and marketing/advertisement of the program is helping to promote more applications of higher quality in the areas that match the strategic directions chosen for the AFG program. Considering the Academy Panel’s recommendations for changes in strategic direction, observed increasing rates of applications for fire-response-related equipment would, for instance, signal a problem that needs to be corrected.
Finally, while the Academy study team was unable to analyze the software used to calculate the scores for the initial ranking of applications, the program management should undertake such a review periodically to ensure consistency with the statute and with the priorities and criteria published annually in Program Guidance.
CONCLUSION
Recommended changes to program management are not one-time activities. They should continue over time and will be guided by new insights into which uses of grant funds are most effective in addressing the risks identified by the communities as well as the evolving national preparedness priorities. Over time, the AFG program management will become better informed and better able to shape the priorities of the program to realize the strategic goals of the program (reducing public injuries and deaths) as a result of additional performance data.
The performance data will come from three general sources: (1) third-party sources such as the survey data and incident data described in Chapter 8; (2) modifications to the program management and reporting processes and systems to gather additional information during the application and closeout processes; and (3) data generated by future research efforts. For
158 example, evidence and analysis to be provided by the AFG-funded three-year NIST study project, “Vision,” (see Box 9-1) could provide critical guidance to those setting AFG grant award criteria. The project is intended to improve fire protection decisions related to resource allocation and services by performing “scientifically-based community risk assessment, strategic emergency response system design and the local government’s service commitment to the community.”103 Pioneering projects such as this, that enable the fire service to move toward data- driven analysis and decisions, show the potential for developing improved sources of knowledge about what strategies/uses of funds will work best in addressing specific risks; this is the key to success of the Academy Panel’s recommended strategy to redirect resources to more cost- effective uses.
Box 9-1. NIST/CPSE “VISION” Study and Software Project
Background: “RHAVE (Risk, Hazard and Value Evaluation) was developed for the [Center for Public Safety International (CPSE)] as a tool to conduct a fire risk analysis of individual properties within a community. A more advanced community risk assessment software package, known as VISION, is currently under development by [CPSE]. The scientifically based risk assessment process should be compatible with VISION and could be applied to future models and processes.”104
Goal: To “enable fire departments and city/county managers to make sound decisions regarding optimal resource allocation and service based upon scientifically-based community risk assessment, strategic emergency response system design and the local government’s service commitment to the community.”105
Phased Approach and Timeline: The 3-year project, funded with a $1 million grant a 2005, is divided into three phases. “In Phase I of the three-phase project, experts are bringing together project technical advisors and a stakeholder group to identify and quantify community risks and necessary resource deployment. In addition, one of the principal tasks of Phase 1 is to evaluate the technical foundation of VISION, a web-based software tool developed by Emergency Reporting in Bellingham, Wash., that enables fire and emergency service agencies to analyze and categorize the risks in their communities … In Phase II, they will collect hazard and response measurements that will form the technical basis for the model. Phase III will be devoted to validating and beta-testing the software for accuracy and ease-of-use. Once complete in 2008, the VISION software will be re-released through [CPSE]. All purchasers of the original version of VISION, available from CPSE for a nominal fee, will receive updated versions as they become available.106
103 NIST project application, 2005. 104 National Fallen Firefighters Foundation, 2005. 105 NIST project application, 2005. 106 CPSE, 2006.
159 Data-gathering and research efforts will contribute to a gradual refinement of targeting that reflects growing community, industry, and federal government understanding of what works to increase local fire safety. This new knowledge base will support changes by informing the program’s and Department’s strategic planning so that they have a better idea of what capabilities are most critical to the program’s goals and the Department’s strategic objectives.
FINDINGS AND RECOMMENDATIONS
The Panel finds that:
• In many situations, increased emphasis on prevention approaches will have greater benefits for public safety than further improvements in fire, EMS and other hazard response capability, yet the AFG program does not currently target and measure its success along this strategic dimension.
• It is possible to fund fire departments and EMS organizations through the AFG program to respond to all hazards in a way consistent with HSPD-8, is consistent across DHS programs and is in alignment with its priorities for increasing the safety of the Nation as a whole.
• Community or Regional Risk Assessments can provide a critical linkage between the priorities of each community and the priorities of the AFG program. Their use will allow communities that have lagged behind in basic capabilities, training, vehicles or equipment to focus on catching up in the most cost-effective manner.
• A valid approach to assessing regional risks is to determine if any subset of communities that would likely be called upon to provide mutual aid has a level of capability so low that it poses a risk to its neighboring communities. Two ways to address a community with insufficient capabilities using AFG grants are: (1) to raise the capabilities of the “weak link” community, or (2) to use logistics to ensure there is a plan in place to ensure that the residents and critical infrastructure across the region can be adequately protected.
• There is consensus that the EMS function has not received adequate attention from the program and its application process, but as of yet there is no agreement on how the AFG program can play its role most effectively to strengthen this function.
• Current program metrics focus on inputs, financial management, and characteristics of the grant recipients. The program does little to measure success in accomplishing long- term goals or achieving intermediate improvements in public safety.
• The AFG grant management system can be modified to support improvements in the grant selection and management processes.
160 Therefore, the Panel recommends that the AFG program:
• Actively consider strategies to encourage high-quality applications for the funding of prevention and mitigation capabilities for all hazards based on best-practice models and in conjunction with the Prevent and Detect taxonomies forthcoming from DHS to support the National Preparedness Plan;
• Use the DHS lists of critical capabilities and metrics to establish priorities for funding and determining overall program performance;
• Consider a linkage from grant application to the community or regional risk assessment and/or a NIMCAST assessment as a way to assign additional positive weighting to applications that address prioritized community risks;
• Define a DHS-wide method of assessing community capabilities that pose a risk to a region; address such risks through the AFG program by raising the capabilities of the “weak link” community or by using logistics to ensure there is a region-wide plan in place adequately protect residents and critical infrastructure.
• Adopt additional performance measures to promote and measure regional collaboration, thus encouraging relationship-building among the emergency response organizations in neighboring communities, even for those grants not awarded;
• Establish performance measures related to EMS and focus on increasing the number of EMS-related applications where it appears relatively small grants could show local differences in the conditions in which patients arrive at the hospital relative to their starting condition; and
• Seek and promote best practices throughout the operations and management of fire departments, EMS and other emergency responders by identifying candidate practices, assessing them, and recognizing those deemed most valuable and widely applicable.
With regard to the grant management process, the Panel recommends that the AFG program:
• Tie performance reports more directly to the application in terms of the major outcome goals, like public injuries and deaths. Applicants should supply both current values in their community for targeted metrics as well as the expected and actual impact on those metrics anticipated from the grant award, thereby demanding that applicants examine the value of the requested use of funds relative to alternative uses;
161 • Modify the grant application to include elements that demonstrate the applicant has an active and current program of community outreach and education;
• Convert the substantive portion of the performance report should to a readily analyzed format that ties back to the data supplied in the original application;
• Better explain to the applicants how to submit a budget for the grant project and how to portray their organization’s fiscal condition in a way that enables panel reviewers to confirm relative financial need; and
• Sort applications into groups from the same geographic region and provide past awards from that region for reference to help the panel determine if the grants are serving regional needs in a rational way.
162 CHAPTER 10 IMPACT EVALUATION
A well-designed evaluation can measure the contribution of a program, or a particular program strategy, to desired outcomes, net of other influences. If carefully designed and executed, it also can provide insights on questions about what elements or uses of a program, in what settings or contexts, are more likely to succeed in producing results. By collecting information on costs as well as benefits, a proper program evaluation can be used to assess the relative cost-effectiveness of different program strategies or uses.
An evaluation program does not concern itself with the financial and compliance performance of any one grantee or grant project. Accountability efforts are part of program management. The AFG program has such mechanisms in place, assisted by the computer-based GMS. However, if the program was to conduct a formal evaluation, the findings of that research could at some future time inform a shift in the accountability approach from one focused just on compliance with rules and regulations to one focused on rewarding results.
The Department may want to seek outside assistance in the design of and detailed planning for an evaluation of the AFG program, based on the evaluation framework presented here.
Based on recommended strategies and statutory goals of the AFG program, the Academy Panel sees central questions for evaluation as (see Figure 10-1):
1. Would a shift to emphasizing prevention and protection efforts by its grant recipients have the expected influence on public death and injury rates? The obverse is also important; that is, would a shift away from the building of response capabilities having a negative effect on public deaths and injuries? Second, what types of prevention or protection activities are likely to have the most positive influence?
2. Would a shift in emphasis to assisting its grantees to meet the measures established under the National Preparedness Goal have a positive influence on public deaths and injuries?
3. Which uses of the grant program are most cost-effective, (i.e., yield the greatest return in improved performance) measured as described in Chapter 8, for the federal grant dollars and total dollars expended?
4. Are regional projects successful in helping those areas meet the Department’s preparedness goals?
5. To what extent is the AFG program leveraging more public investment in fire safety, improved emergency medical response, and other contributions to public health and safety?
6. To what extent is the AFG program responsible for a faster rate of positive change in the practices of fire departments within the U.S.?
163 7. Is the AFG program contributing to improvements in the performance of the EMS function in the funded communities?
8. To what extent, if at all, is the risk to the public of death or injury from fire, other hazards, and medical emergencies, reduced by the AFG program?
Figure 10-1. A Multi-Faceted Approach to Impact Evaluation
Evaluations should be designed and conducted to ensure that the AFG program can separate its influence from other influences on the performance of local fire departments and non-affiliated EMS organizations and on the public health and safety outcomes that are the ultimate target of the program’s efforts. This is always difficult to do in a public administration setting. Fire services, search and rescue, HAZMAT responses, and EMS, for instance, are not experiments that can be set up in well-controlled ways. It is, however, important to investigate whether and to what extent AFG grants are having a positive influence.
Chapter 9 discussed influencing capabilities and intermediate outcomes—the earlier stages of the logic model. This chapter describes and recommends a practical approach to rigorously and independently evaluating the program’s impact—the final stage of the logic model—on a periodic or ad hoc basis to validate the assumptions about causation implicit in the logic model. The first section of this chapter begins by discussing the perspective of the OMB PART on performance evaluation. It then presents an evaluation framework that focuses on the main questions noted above and provides some other commentary related to the management of the program and measurement approaches, recognizing that in many instances an AFG grant is a modest contribution to a larger marshalling of resources and volunteer efforts.
164
OMB PART’S APPROACH TO PERFORMANCE EVALUATION
It is important that the AFG program evaluation satisfy PART expectations. The AFG program was subject to a PART review in 2004 and is scheduled for review again in 2007.
According to the OMB (2004), the PART primarily analyzes program evaluation, answering two questions:
• “[A]re independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?” (Question 2.6)
The purpose of Question 2.6 is to ensure the program (or agency) conducts non-biased evaluations on a regular or as-needed basis to fill gaps in performance information. These evaluations should be of sufficient scope and quality to improve planning with respect to program effectiveness.
• “[Do] independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?” (Question 4.5)
The purpose of Question 4.5 is to determine whether the program is effective based on independent and comprehensive evaluations. OMB has published separate guidance describing what kind of an evaluation would meet its standards for rigor and independence.107
DEVELOPING THE EVALUATION FRAMEWORK
In responding to the PART, the program can look to the evaluation framework recommended in this report. The framework encompasses strategic management of the program and a research strategy addressing the main evaluation questions noted above. The Department will need to collect data, design analytic programs or projects (including simple assessments, market research, and quasi-experimental approaches) and experiments to accomplish a proper evaluation meeting OMB’s recommended standards.
Recommendations in this report assume a number of logical relationships that must be carefully tested. These also should be main attention points for an evaluation effort.
The Department should first take the steps necessary to complete an evaluation program design and execution plan. At the same time, it should modify some data collection and grantee reporting requirements so that they produce information necessary to support an evaluation.
107 OMB, “What Constitutes Strong Evidence of a Program’s Effectiveness,” undated. Available at www.omb.gov.
165
Evaluating Preparedness
The Panel recognizes the National Preparedness Goal and the supporting DHS National Preparedness Plan, described in Chapter 6, as a valid framework for improving preparedness. This implies that the Firefighting Operations/Support target capabilities will, if improved for a community, make the citizens of that community safer from fire-related death, injury, and economic loss, regardless of whether it is part of a national disaster or an individual structure fire. Also, the Triage and Pre-Hospital Treatment target capabilities are assumed to improve the safety of citizens whether they are, for example, victims of a terrorist attack or a heart attack.
It is logical that the skills required for responding to different hazards, while not identical, would have significant overlap. From the Academy’s research, there are relatively few emergency responders at the community level who have a high degree of specificity regarding types of events they are effective in responding to.
There is currently little data available to validate the effectiveness and applicability of the various elements of the National Preparedness Goal/Plan. In fact, the details of the Plan themselves are in various states of completion and refinement. While the high-level goal in relation to all-hazards preparedness is sufficiently broad, the underlying question is whether the NPP is internally valid, in that achievement of the specified levels contributes to achievement of that high-level goal. Answering this question in a rigorous way, due to the hundreds of underlying implied causal relationships, represents a monumental analytical challenge.
Returning to the hierarchy of recommended OMB PART approaches for validating causality, it is unlikely that randomized controlled trials will be the best use of resources to validate, for example, that “Conduct building plan reviews” causes improvements in the “Develop and Maintain Plans, Procedures, Programs and Systems” capability that eventually leads to decreases in economic loss, injury and death from fires. Assuming development of NPP has taken into account available retrospective analyses (e.g., the response to Hurricane Katrina) and expert interviews and secondary research where primary research is practical, the risk to the success of the AFG program if it were to incorporate the assumed causal relationships into its management framework is relatively low. However, the fact that AFG program administrators have had limited involvement with the development of components of the NPP related to fire and EMS may pose a risk of future disconnects, especially if the AFG program becomes more tightly- coupled with the NPP as recommended by this report.
The Panel recommends that as the DHS continues to refine and update the elements of the NPP, it include representation from the U.S. fire service in general and the AFG program in particular, to ensure that the plan covers the broader needs of DHS.
Does the AFG Program Contribute Significantly to National Outcomes?
Several interrelated factors affect the ability of individual fire departments and other emergency responders to fulfill their missions. Complexity confounds the ability to isolate the AFG
166 program’s influence on outputs, outcomes and impacts. Only a rigorous controlled evaluation of the program can identify its separate influence on the outcomes to be affected.
Most significantly, the AFG program contributes only a small portion of fire-related financing each year. Considering the value of volunteerism and the full cost of EMS, the AFG program contributes far less than one percent of fire department and fire department-based EMS budgets, as described in Chapter 4. Therefore, changes in national tracking metrics cannot be used as an accurate gauge of the success or failure of the AFG program without considering the simultaneous impacts of numerous other factors, such as:
• changes in overall spending (including other available state and federal grant programs)
• population changes (e.g., population levels, demographics, shifts from rural toward urban)
• further transformation to a career fire service from a volunteer fire service
• changes to fire risk factors and fire enablers through statutes or education, including:
o heat sources (e.g., self-extinguishing cigarettes, child safety lighters)
o behavior (e.g., arson, safety education)
o fuel sources (e.g., furniture, mattresses, bedding, appliance housings)
o building codes, sprinkler laws and other fire prevention programs
The AFG program goals are given in terms of outcomes, but an implicit goal is that the program should achieve the maximum national impact on those outcomes for each dollar invested by the Federal government. Some AFG program stakeholders have considered assessing grantee performance by determining how and how often the equipment or other items purchased with AFG funds were used by the grant recipient. For example, how often was a thermal camera used in a fire? There are practical problems with this as a research approach. The questions and interpretation of the resulting data would need to be tailored for the nearly two hundred unique uses of funds in 2005. For example, four awards were made for “spark-proof tools.” To determine if that money was well spent would require an estimate by the grantee of the number of situations where spark-proof tools were used when other, traditional tools would not have been sufficient, and what was the impact of those uses in terms of metrics that affect public health and safety. With only four such awards, the responses would too few to draw conclusions about the relative contribution of this use versus other uses of the grants.
Measuring the return on investment in emergency response is not practical because the incident is always unknown until it occurs or is reported. Forecasting incident patterns at a granular level (e.g., time of day, type of incident), as well as establishing regional cooperative programs to absorb the risk of incident rate spikes (also known as surge) or to share infrequently used assets are both approaches to improving the return on the investments of each department. Cross-
167 training personnel on both firefighting and EMS skills is a form of load-balancing to produce more output with a given level of input by minimizing idle workers and maximizing the effective skills of each person who arrives at the scene of an incident. In any case, an evaluation measuring return on investment must be able to determine to what extent observed improvements in outcomes are attributable to grants made by the program.
A special consideration involves the tradeoff of spending more grant dollars for prevention and mitigation instead of protection and response. Ironically, many measures of the effectiveness of a fire department or emergency response capability (e.g., average casualties per fire or average response time) overlook the ability of prevention campaigns to reduce both the numerator and the denominator of those calculations. Some experts interviewed noted that after a certain response capability has been achieved, spending additional money on prevention was likely to be the more cost-effective use of funds to protect citizens from fire-related incidents.
The Panel recommends that when results are requested from grantees, baselines be established by the program for measuring and predicting improvement in outcome measures attributable to the awarding of each category of grant. The outcome metrics should become a target for continuous improvement by the program and an input into its annual priority setting and planning sessions.
Who Should Determine which Uses of AFG Funds Have the Largest Impact?
Separate from the question of the overall impact of AFG grants on national measures is the question of the most efficient use of grants within each category of eligible uses. The approach to performance-driven management described in the previous chapter would allow an objective analysis to determine the uses of funds that are likely to make the largest impact. For example, the grant application would ask the applicant about the current fire incidence in the community and how much they would expect to decrease that instance (overall, or in a targeted subsection of the community) as a result of the AFG grant. That subjective data on its own is of limited value, but it forces the applicant to consider alternative formulations of the proposed project that would maximize the impact on the relevant metric.
The comparative value is derived once the actual results are provided in the performance reporting section of the close-out documentation. If 200 fire prevention grants were awarded this year, it is reasonable to assume that the ten communities that realized the most significant reductions in the incidence of fire are at least worth studying if not emulating.
The Panel recommends that a significant input for the annual prioritization and criteria setting session include an analysis of the actual impacts experienced by the previous year’s performance reports. Those uses that were found to be the most cost-effective in each category should be provided as case studies made available to all applicants (and non-applicants) via AFG publications. There should also be preference given to future applications that mark on their application that they are emulating a published best-practice with demonstrated cost-effectiveness.
168 The Panel recommends that until performance information is collected and analyzed, with regard to prevention programs, the AFG request applicants describe whether the proposed prevention program is based on a model recommended by a nationally recognized organization. The example used here relates to prevention programs because that is an area where there is a particular gap in comprehensive research. However, model programs can just as easily be analyzed and published in areas such as inter-jurisdictional cooperative agreements or training programs.
Do AFG Grants Leverage or Supplant Other Public Funds?
There is an implicit assumption in the AFG program that by requiring local matching funds, the federal share of the funds is being leveraged, or amplified, in its effectiveness. The non-federal share varies with the population of the jurisdiction served: 5 percent share for organizations serving areas less than 20,000 people, 10 percent for those who serve between 20,000 and 50,000 people, and 20 percent for those who serve populations of more than 50,000 people. One implication of this requirement is that the AFG program management reports a small portion of grantees return their grant for redistribution because the matching funds did not materialize in the budget as expected.
There is no conclusive data available to analyze whether federal funds are in fact being leveraged or if they are instead experiencing the exact opposite—they are being used to supplant local funds that would have been spent on the same items. However, anecdotal evidence abounds on this topic.
The theoretical supplantation can take a different form as well. A community may be in need of an equipment or vehicle replacement for goods purchased several years ago (either with an AFG grant or without). Instead of a community amortizing the items by setting aside a portion of their replacement cost each year, the community could apply for an AFG grant each year for the replacement. This would, in effect, be a gamble that an AFG grant will be awarded before the equipment or vehicle is in such critical state of disrepair as to cause a safety risk to its users or the protected population.
• One question that should be addressed is whether the current matching percentages are the optimal values. Perhaps more successful leverage could be achieved by requiring all communities to contribute 20 percent, for example.
• A second question is whether there are effective ways to ensure that federal funds are not supplanting local funding, but rather supplementing it.
• A third question is whether there are other factors that serve to maximize the leverage of federal funds. For example, a reasonable hypothesis is that grants flagged as regional in nature are less likely to be considered a normal budget item for a community in the absence of a federal grant.
169 The Panel recommends that after-the-fact analysis of communities awarded a large piece of equipment or vehicle in the earlier years of the program are in fact allocating money in their budget to replace the item at the end of its amortization schedule. And as mentioned in Chapter 9, applicants should be required to confirm that grants requiring multiple years of budget commitments, such as vehicle maintenance, provide assurance that the obligation will be met each year.
Do AFG Grants Improve Community Capabilities?
A common theme in the opinions the Academy heard from AFG stakeholders was that if the AFG program is intended to improve community capabilities to handle all hazards, and accreditation programs are intended to measure capabilities, there should be way to measure effectiveness of grant money by looking at accreditation. The Academy explored that question and found some apparent synergies but other areas that require caution.
According to Mike Waters, Vice President of ISO’s Risk Decision Services Department, ISO’s property insurance rating programs combine the principles of actuarial analysis with representations of loss mitigation programs that exist within communities.108 Relationships between actual fire loss experience in a state and the relativity’s developed using ISO’s Fire Suppression Rating Schedule evaluation program are foundational to the ISO property rating plans filed in a state.
ISO’s PPC evaluations are centered on three main aspects of fire protection—fire alarm and communications including telephone systems, staffing and dispatching systems; the fire department, including equipment, staffing, training, and geographic distribution of fire companies; and the community water supply system, including condition and maintenance of hydrants, analysis of the capacity and adequacy of the amount of available water for fire suppression.
According to Waters:
“The PPC program recognizes the efforts of communities to provide fire-protection services for citizens and property owners. A community’s investment in fire mitigation is a proven and reliable predicator of future fire losses. The program also helps fire departments and other public officials plan for, budget and justify improvements. In fact, in an independent opinion poll commissioned by ISO and conducted by ORC International in 2001, 97 percent of fire officials surveyed indicated that the PPC program is important in helping the community save lives and property.”109
Waters contends that, on average, communities with “superior” fire protection services as determined by the Public Protection Classifications have lower fire losses than communities whose fire protection services are not as comprehensive. To quantify the relationship, ISO reviewed the cost of fire claims per thousand dollars worth of insured property by PPC for communities across the country. Analysis showed that “for commercial property, on average,
108 personal communication, November 10, 2006. 109 Ibid.
170 per $1,000 of insured property, communities in the worst classification had commercial property losses more than three times as high as communities in the best classification. And for Homeowners, on average, per $1,000 of insured property, communities in the worst classification had homeowners fire losses more than twice as high as communities in the best classification.”110
ISO maintains a database about the quality of public fire protection in more than 45,000 fire districts across the United States. ISO also maintains an outreach program to identify changes that may affect a community’s PPC or the insurance premiums of individual homeowners and businesses. In doing so, ISO updates protection information for over 16,000 fire districts per year regarding changes in fire district boundaries, automatic aid agreements, fire station locations, and access to water. Analysis of the classification review data reflects the fact that in any year, more than half of the communities receive better PPC ratings, with less than 2% receiving worse classes. ISO provides a summary report to communities at the completion of on-site evaluations that include details of the review as well as improvement statements should the community desire to enhance their level of protection.
To answer the question of what impact the AFG program has on community fire protection capabilities, ISO conducted a preliminary comparison of the communities that received AFG money for the purposes of purchasing apparatus and PPC. “While there are many assumptions made in review of this data (e.g., the apparatus was a pumper truck, the fire department name equaled the community name in our database, the purchase was a key ingredient in the PPC number, ISO’s evaluation was completed after the apparatus was put in service, etc.), there is an indication that over 50 percent of the jurisdictions [that received AFG money for apparatus and PPC] improved their classification, and many by two or more classifications. While the enhancements in fire protection may have extended beyond purchase of new fire apparatus, the initial results support further analysis.”111
Although it is virtually impossible to draw a direct correlation between the acquisition of apparatus and an improved grading, there is a potential impact that would benefit from further investigation. It may be that the process of compiling data required for an AFG application emphasized planning within these organizations. Alternatively, the requirement that AFG applicants contribute to the NFIRS fire data system places more emphasis on the use of data in the overall planning and operations efforts of the organization. Irrespective of the cause for the improved capability ratings of AFG grantees, this is an example of the type of data analysis that would be made possible by establishing a comprehensive baseline of intermediate outcome measures as described in Chapter 8.
In a related effort, recently ISO and the U.S. Department of Agriculture have begun comparing data to discern the potential positive effect of Agriculture’s Rural Development Community Facility Direct Loans and Grants for enhancing local fire protection (such as construction of new fire stations, new community water systems, and fire apparatus) on a community’s PPC.
110 Ibid. 111 Ibid.
171
DESIGNING AND IMPLEMENTING THE EVALUATION FRAMEWORK
OMB considers the most significant aspect of program effectiveness to be its impact, defined as “the outcome of the program, which otherwise would not have occurred without the program intervention.”112 There are several approaches to determine the impact of a program, each with a different ability to provide unbiased impact data. However, some approaches are not suitable due to the characteristics of the AFG program.
Where feasible, OMB considers randomized controlled trials (RCT) to be the ideal way to attain unbiased program impact data. A randomized controlled trial is a study that measures an intervention’s effect by randomly assigning, for example, fire departments, into an experimental group, which receives or conducts the intervention, and others into a non-experimental group for control purposes. Data are gathered to identify the difference between the two groups to determine the difference in outcome that is due to the intervention. This is the most effective way to infer causality.
The AFG program is a competitive application process making it almost impossible to construct RCTs in the normal course of the program; it certainly could not be blinded in any way, for instance. Chapters 8 and 9 both discuss data, data set, and data collection or instrument issues and problems related to AFG. By definition the experimental group of applicants would be a convenience sample, which would be difficult to control. It would not be appropriate statutorily to award grants randomly since that approach would be in conflict with the fundamental competitive purpose of the statute. The filtering algorithms and the peer review process based on the annual program priorities are intended specifically to avoid “random” distribution of grant funds. There are “history” and “maturation” issues involved with any of these applicants too (all of whom are aware of the program), making it nearly impossible to construct a proper RCT. This is why the Panel recommends that other funds be used for truly experimental type efforts. Supplemental appropriations may be needed to fund such efforts at an appropriate level.
Quasi-experimental methods may be useful. In the case of the AFG program, some applicants will get funds, some will not. Some grantees will have narrow purposes of varying types. Some eligible organizations will not apply, and some may never apply. These and other characteristics can allow the agency to perform comparative analyses using concurrent or retrospective samples. Evaluation questions noted in the introduction above signal areas where such comparative analyses would be useful. Here are two examples:
• Do jurisdictions where prevention projects operated for some time have better public death or injury statistics than similar jurisdictions who received funds for response- capability purposes?
• Do recipients or their jurisdiction who target the use of funds to serve areas where at-risk populations reside see better statistics for these populations compared to others?
112 OMB, 2004.
172 In conclusion, of the relevant methods endorsed by the OMB for evaluating program effectiveness, the most unbiased feasible method is the quasi-experimental design. The program should also look for opportunities to apply RCT techniques to the secondary questions, such as the one noted above about the best prevention or protection activities that recipients can pursue.
When using a quasi-experimental design for the AFG program, the most relevant comparison would be between a fire department that receives a grant and a comparison group with similar demographic, resource and management characteristics that does not. Instead of using random assignment, the differences between the two groups are determined by the judgment of the evaluator to minimize any differences. Therefore, it may be most appropriate, for example, to compare grant recipients against applicants who were denied grants due to specific categories of factors that indicate as little systemic variation as possible. Another approach would be to look at those applicants slightly above and those slightly below a cutoff point to indicate fewer differences that would confound the results. Great care should be taken to design the comparison groups to minimize the unavoidable risk of misleading results based on factors other than the grant award.
For some metrics, there is no appropriate or currently available source of data to compare both the awarded and non-awarded applicants. For example, there will be no post-grant performance reporting requirement of those applicants who did not receive a grant to compare them with applicants that did. But there are still lessons to be learned from the result that grants had on the recipient departments and communities. For this type of analysis, OMB recommends a non- experimental direct analysis to examine only the intervention subject (e.g., a fire department awarded a grant) without a comparison group. Sometimes known as a “pre-post study,” this format is used by the AFG program today, to some extent, when it asks grant recipients to close- out the application process by describing the impact of the grant one year after its award. By adding some rigor to the current evaluation process, this method may be useful “for examining how or why a program is effective, or for providing information that is useful for program management.”113 OMB also warns against relying too heavily on this approach due to its susceptibility to false conclusions if used to measure program effectiveness.
(See Figure 10-2 for the design determination decision tree provided by the General Accounting Office114 for program evaluations. It gives criteria that can be used to determine the most appropriate designs to consider for a variety of evaluation conditions.)
113 OMB, 2004. 114 Now known as the Government Accountability Office.
173 Figure 10-2. Linking a Design to the Evaluation Question
174 A Phased Approach
As stated above, the OMB considers the most significant aspect of program effectiveness to be its impact, defined as “the outcome of the program, which otherwise would not have occurred without the program intervention.”115 The Panel recommends that such an evaluation be addressed by the AFG program in two phases, both of which are distinct efforts outside the normal program management activities. Phase one should include changes to the application and closeout processes to collect appropriate impact data for phase two evaluation, as well as analysis of impact data that is currently available to the program. Phase two should include analysis of data generated as a result of phase one program modifications as well as other concurrent data collection efforts currently underway.
Chapter 9 included a table of recommended additional application questions based on the Academy’s analysis of existing questions and the logic model presented in Chapter 8. Those modifications should be undertaken in phase one, which will likely take a year before it results in new types of data that the AFG program can analyze. The closeout questions currently asks grantees about the actual funds spent, the equipment inventory and a narrative in which the grantee should “provide a summary of the qualitative accomplishments that your FEMA grant has made during the grant period.” In addition, grantees should be asked to respond with a quantitative or yes/no response to each of the metrics that were originally reported as being expected to change as a result of the grant. (See Chapter 8.)
Given the size and focus of the program, it is most important to focus measurement on the end public safety goals as described in Chapter 8. However, it is also useful to build a baseline for future measurement and trend analysis of capabilities, specific or general (e.g., accreditation, measures of fire fighting capacity), and outcomes in addition to the broader impacts to be influenced. The ultimate goal, once data have been collected, should be to use these intermediate measures to establish a connection with program’s awards to validate the logic model, thereby suggesting new data-driven strategies.
From the currently available data sources listed in Chapter 8, very few cause-effect relationships can be drawn in such a way that would help in the development of criteria for selecting the best use of grant funds. But once the relevant data has been captured for applicants, the phase two effort should be undertaken to assess the actual results after the period of performance expired to gain a better understanding of causality where possible. The AFG program needs a solid understanding of which measures the program should track to align it with the desired ultimate outcomes and to present as evidence of the program’s contributions to progress toward these outcomes. Phase two will also include the availability of EMS data from the NEMSIS project currently underway in partnership with the NHTSA, expected to be a very useful window into the effectiveness of EMS assets, techniques and other intermediate outcomes that can assist in shaping future AFG program priorities.
Table 10-1 lists the key evaluation questions for the AFG program and suggests a practical phased approach to implementation.
115 OMB, 2004.
175 Table 10-1. Phased Approach to Implementing an Evaluation Framework
Impact Evaluation Question Phase 1 Approach Phase 2 Approach
1. Whether a shift to Identify and recommend Use a quasi-experimental emphasizing prevention and nationally-recognized prevention design to calculate the protection efforts by its grant programs; modify the impact the program has had recipients is having the application, closeout on the incidence rate of fires expected influence on public questionnaires so they can be nationwide; use a random death and injury rates. compared. controlled trial to objectively identify the most effective prevention programs.
2. Whether its shift in emphasis Establish a system to track grant Use a quasi-experimental to assisting its grantees to applications, awards, and design to calculate the meet the measures subsequent reported performance impact the program has on established under the to the specific critical capabilities the stated community National Preparedness Goal and activities the grants are priorities. is having a positive intended to influence; align AFG influence on public deaths with the further development of and injuries. DHS priorities.
3. Whether the competitive The closeout questionnaire Conduct a comparative grant program is cost- should be revised to focus on the analysis of the impact on effective. outcomes as described in Chapter outcome measures 8. compared with the grant funds distributed to determine uses of funds and community characteristics correlated with cost- effectiveness.
4. Whether regional projects Modify the application process to Use a quasi-experimental are successful in terms of the collect the data described in design to analyze the impact preparedness goals. Chapter 9. on reported risks and needs from those applicants that were formerly granted a regional award.
5. Whether the AFG program Modify the application process to Conduct a random is leveraging more public collect budget data from all controlled trial to measure investments and a faster rate relevant years; an analysis of the degree of supplantation of positive change in the communities awarded equipment of local budget funds by the practices of fire departments or vehicle should be conducted AFG program. within the U.S. to verify budgeting replacement costs at the end of an item’s amortized life.
176 Impact Evaluation Question Phase 1 Approach Phase 2 Approach
6. Whether real improvements Modify the application and Participate in analysis of the are occurring in the EMS closeout questionnaire to gather NEMSIS data with HHS and function in the U.S. data about EMS incidents. DOT to determine the resources and intermediate Work within DHS, HHS, and outcomes associated with DOT/NHTSA during the current successful EMS events. Use pilot stage of NEMSIS for the this information as input purposes of reporting to the into the AFG prioritization national repository of EMS data. process.
7. Whether the statistics about Modify the application and Once applications are areas and populations at risk closeout questionnaire to submitted that report CRA are being made more emphasize the CRA and data, use an experimental positive as a result of the associated high-priority design to understand the AFG program community risks. relationship between high- priority risks and community characteristics to better target AFG funds.
8. To what extent is the risk to Modify applications and closeout Conduct a random the public of death or injury questionnaire to gather controlled trial to compare from fire, other hazards and standardized data that can be the key measures for grant medical emergencies compared with national surveys. recipients with non- reduced by the AFG recipients. program?
FINDINGS AND RECOMMENDATIONS
The Panel finds that:
• While the effort to devise a national preparedness plan that will in fact make America safer, the proposed integration between that plan and the AFG program makes closer collaboration between the two DHS programs vital for success.
• Even though the AFG program contributes a small portion of the state and local emergency response and prevention budgets, it has the potential make a large impact on the communities that are awarded grants. That impact has not yet been modeled sufficiently.
• Little data is gathered currently from which to draw conclusions about which uses of AFG funds have the largest impact on outcomes.
• There is no prescribed way for AFG grantees to analyze, or report, whether they have realized their expected outcomes for the use of the grant award money.
177 • A variety of intermediate factors logically influence intended outcomes, but these measures are not being gathered or analyzed.
• It is unknown whether or to what degree AFG grants leverage or supplant other public funds.
• The degree of impact that AFG grants have on improving community capabilities is clouded by the lack of intermediate measures currently gathered and analyzed.
Therefore, the Panel recommends that the AFG program:
• Be represented, along with the U.S. Fire Service in general, as DHS continues to refine and update the elements of the National Preparedness Plan most relevant to the AFG program to ensure that the plan covers the broader needs of homeland security;
• Develop a model that establishes a baseline for measuring and predicting improvement in outcome measures attributable to the awarding of each category of grant and a target for continuous improvement, once results data are requested from grantees;
• Modify the performance reporting requirements for grantees to be one year after grant money is fully distributed rather than the current timeframe: one year after the award;
• Add an analysis of the actual outcomes reported in the previous years’ performance reports as a significant input for the annual prioritization and criteria setting session;
• Request applicants to describe whether the proposed prevention or mitigation program is based on a model recommended by a nationally recognized organization;
• Revise the closeout questionnaire to focus on the outcomes that the grant was originally intended to affect to enable analysis of grantee success as well as cost-benefit comparisons across categories of applicants;
• Capture additional data elements during the grant application process to establish baselines of data related to capabilities and intermediate outcomes that are necessary to establish correlations, and eventually, measure the impact of the AFG program; and
• Analyze communities awarded a large piece of equipment or vehicle in the earlier years of the program to see if they are in fact allocating money in their budget to replace the item at the end of its amortization schedule.
178 APPENDIX A
USES OF GRANT FUNDS IN 2005
Table A-1 through A-7 below provide a detailed break-out of the uses of the awards that were granted in each category, excluding prevention, by quantity of award in FY 2005.
Table A-1. Total AFG FY05 Awards for Vehicles
VEHICLES Dept Type All Paid On All Paid/ Call/ ITEM Combination Volunteer Career Stipend Total Pumper 20 217 5 9 251 Tanker 19 213 0 13 245 Tender 6 12 0 3 21 Platform 0 2 3 0 5 Ambulance 11 12 8 5 36 Brush Truck 7 49 2 2 60 CAFS Pumper 2 7 0 1 10 Foam Pumper 1 0 1 0 2 Ladder Towers 0 0 2 0 2 Mini-Pumper 1 8 0 0 9 Pumper/Tanker 14 144 2 10 170 Rescue/Pumper 2 24 0 1 27 Quick Attack Unit 7 17 1 0 25 Aerial Ladder Truck 1 0 6 0 7 Rescue (Light, Medium, Heavy) 1 0 5 0 6 Type I, Type II, Type III Engine 2 9 1 0 12 Patrol Unit (Pick up w/ Skid Unit) 1 1 0 0 2 Fire Boats (more than 13 feet long) 2 1 1 0 4 Quint (Aerial device less than 76 ft) 1 3 2 0 6 Quint (Aerial device 76 ft or greater) 6 1 3 1 11 Total 104 720 42 45 911
A-179 APPENDIX A
Table A-2. Total AFG FY05 Awards for Vehicles
EQUIPMENT Dept Type All Paid On All Paid/ Call/ ITEM Combination Volunteer Career Stipend Total AEDs 9 26 4 342 Pagers 66 261 18 18363 Cameras 8 24 2 438 Ladders 14 82 6 5107 Nozzles 91 463 24 16594 Splints 6 2 0 08 Suction 4 9 2 015 Headsets 27 35 13 883 RIT Pack 95 184 27 18 324 Wildland 14 71 2 693 Computers 24 77 26 4131 Mini-cams 0 0 1 01 Repeaters 43 62 26 6137 Skid Unit 5 35 0 2 42 Backboards 2 11 0 013 Hand Tools 4 78 1 17 100 Power Saws 55 168 20 5 248 Suppression 1 1 0 02 Base Station 49 72 34 15 170 Stethoscopes 2 2 0 04 Thermometers 2 1 0 03 Mobile Radios 110 253 60 28 451 Portable Pump 8 66 1 1 76 Auto-injectors 1 0 01 Two-Way Pagers 4 11 1 0 16 Fixed Generator 11 33 4 6 54 Portable Radios 170 420 86 49 725 Pulse Oximeters 2 10 0 0 12 Rehab Equipment 1 2 0 5 8 Cervical Collars 0 1 1 0 2 Lights, Portable 10 73 5 5 93 Various Supplies 10 10 3 0 23 Washer/Extractor 81 115 24 17237 Reference Library 5 1 0 0 6 Spark Proof Tools 2 2 0 0 4 Other EMS (explain) 17 32 9 2 60 ALS Airway Equipment 2 1 1 1 5 All-Terrain Vehicles 3 10 3 1 17 BLS Airway Equipment 3 14 0 0 17
A-180 APPENDIX A
EQUIPMENT Dept Type All Paid On All Paid/ Call/ ITEM Combination Volunteer Career Stipend Total Biological Detection 1 2 3 0 6 Blood Pressure Cuffs 2 6 0 0 8 Portable Deluge Sets 10 70 3 7 90 Defibrillator/Monitor 15 19 12 248 Other Haz-Mat (explain) 4 9 2 0 15 Thermal Imaging Devices 123 394 59 29 605 Adapters, Wyes & Siamese 55 271 44 15 385 Vest Extrication Devices 3 1 0 1 5 Portable/Mobile Generator 11 65 10 2 88 Hose-(31/2 inches or less) 111 502 38 17 668 Mobile Data Terminal (MDT) 16 2 25 2 45 Other EMS/Rescue (explain) 23 61 42 2 128 Technical Rescue Equipment 72 77 27 1 177 Other Specialized (explain) 96 145 55 15 311 Hydrant and Spanner Wrenches 10 60 4 1 75 Computer Aided Dispatch (CAD) 5 1 7 2 15 Monitoring & Sampling Devices 8 13 3 1 25 Other CBRNE-related Equipment 8 1 5 0 14 Other Investigation (explain) 2 6 0 1 9 Other Communications (explain) 196 146 158 18 518 Monitoring and Sampling Devices 36 61 24 2 123 Other Basic Equipment (explain) 204 870 95 53 1222 Foam Educators and foam concentrate 21 92 2 0 115 Boats (13 feet in length and under) 3 4 4 0 11 Stretchers, Backboards, Splint, etc. 13 19 11 2 45 Automated External Defibrillators (AEDs) 15 56 1 476 Compressors/Cascade/Fill Station (Fixed) 146 407 73 56682 Compressors/Cascade/Fill Station (Mobile) 40 111 17 10178 Ropes, Harnesses, Carabiners, Pulleys, etc. 67 150 34 3 254 Hose-Large Diameter (LDH 4 inches or larger) 53 179 27 10 269 Powered/Mechanical Extrication Tools/Equipment 157 482 58 19716 Decon, Clean-Up, Containment and Packaging Equipment 27 7 4 0 38 Total 2514 7007 1251 517 11289
A-181 APPENDIX A
Table A-3. Total AFG FY05 Awards for PPE
PPE Dept Type All Paid On All Paid/ Call/ ITEM Combination Volunteer Career Stipend Total Boots 126 457 37 49669 Hoods 63 277 17 33390 Gloves 104 413 30 48595 Goggles 33 105 5 11154 Helmets 138 472 35 53698 Canteens 4 7 0 112 Shelters 20 33 10 265 SCBA/CBRN 23 34 22 382 Face Pieces 135 330 58 59 582 Flashlights 27 142 4 12185 Respirators 9 21 6 137 Escape Masks 1 2 1 1 5 PASS Devices 38 150 8 16 212 Pants, Coats 63 138 21 22 244 Splash Suits 2 2 0 4 Tyveck Suits 4 5 0 9 Coats & Pants 123 458 51 56 688 Air-Line Units 11 28 16 8 63 Infection Control 4 11 2 1 18 Wet and Dry Suits 6 13 4 23 Encapsulated Suits 5 4 1 10 Hearing Protection 1 4 0 1 6 Jumpsuits/Coveralls 18 86 4 9117 Other PPE (explain) 259 458 105 47 869 Accountability Systems 42 72 24 2 140 Complete set of turnout 227 998 47 61 1333 Other CBRNE-related PPE 9 7 14 30 Proximity and Entry Suits 1 0 0 1 Spare Cylinders--30 minutes 46 176 21 20 263 Spare Cylinders--45 minutes 22 59 11 8 100 Spare Cylinders--60 minutes 19 25 5 2 51 SCBA--30 minutes with face piece- No extra bottle 59 197 26 27 309 SCBA--45 minutes with face piece- No extra bottle 25 54 6 5 90 SCBA--60 minutes with face piece- No extra bottle 16 14 4 3 37
A-182 APPENDIX A
PPE Dept Type All Paid On All Paid/ Call/ ITEM Combination Volunteer Career Stipend Total SCBA--30 minutes with face piece- With extra bottle 159 761 49 89 1058 SCBA--45 minutes with face piece- With extra bottle 140 296 27 27 490 SCBA--60 minutes with face piece- With extra bottle 30 83 20 7 140 Chemical/Biological Suits (Must conform to NFPA 1994, 2001 edition) 5 1 8 014 Total 2017 6393 699 684 9793
Table A-4. Total AFG FY05 Awards for Training
TRAINING Dept. Type Paid On All All Paid/ Call/ ITEM Combination Volunteer Career Stipend Total Specialist 0 0 1 01 RIT Training 12 26 4 2 44 Environmental 1 0 0 01 Other Training 40 39 27 7 113 Safety Officer 0 1 2 0 3 Vehicle Rescue 2 10 1 1 14 Operations (NFPA 472) 6 21 1 1 29 Exercises/Preparedness 0 3 1 04 NIIMS (Unified Command) 2 6 3 0 11 Operations-level Training 0 5 1 0 6 Technician-level Training 1 0 1 0 2 Fire Inspector (NFPA 1031) 1 0 2 0 3 Paramedic Training (EMT-P) 1 0 5 0 6 Basic Wildland Firefighting 3 7 1 0 11 Driver/Operator (NFPA 1002) 12 21 3 3 39 Officer Training (NFPA 1021) 5 4 2 0 11 Infection Control (NFPA 1581) 0 1 0 0 1 Other CBRNE-related Training 1 0 4 0 5 Fire Investigator (NFPA 1033) 2 0 0 0 2 Telecommunications/Dispatcher 1 0 0 1 2 Instructor Training (NFPA 1041) 7 8 0 116 Incident Management Course 2 4 0 1 7
A-183 APPENDIX A
TRAINING Dept. Type Paid On All All Paid/ Call/ ITEM Combination Volunteer Career Stipend Total (IMC) Medical First Responder Training 2 7 1 111 Wildland Firefighter Certification 0 1 0 01 EMS for Incidents Involving CBRNE 100 01 Hazmat - Technician/Specialist level 3 6 5 014 Confined Space Rescue - Awareness level 2 4 1 1 8 Firefighter I, Firefighter II (NFPA 1001) 36 159 7 21 223 Emergency Medical Technician - Basic (EMT-B) 8 9 1 0 18 Emergency Medical Technician - Intermediate (EMT-I) 0 0 1 0 1 Airport Rescue Firefighting (ARFF) (NFPA 1003) 1 3 0 0 4 Emergency Medical Technician - Intermediate (EMT-I) 3 0 0 0 3 Weapons of Mass Destruction - Awareness level (CBRNE) 1 0 0 0 1 Technical Rescue/Urban Search and Rescue - Awareness level (NFPA 1670/1006) 0 1 0 0 1 Technical Rescue/Urban Search and Rescue - Operations Level (NFPA 1670/1006) 2 4 2 0 8 Technical Rescue/Urban Search and Rescue - Technician Level (NFPA 1670/1006) 7 1 8 1 17 Total 165 351 85 41 642
A-184 APPENDIX A
Table A-5. Total AFG FY05 Awards for Wellness & Fitness
WELLNESS & FITNESS Dept. Type All Paid On All Paid/ Call/ ITEM Combination Volunteer Career Stipend Total Other 7 1 9 1 18 Initial Physical Exam 101 75 80 11 267 Employee assistance program 0 0 1 0 1 Job Related Immunization Program 101 75 80 11 267 Injury/illness rehabilitation program 0 1 4 0 5 Periodic Physical Exam/Health Screening 101 75 80 11 267 Formal fitness and injury prevention program 60 23 50 4 137 Total 370 250 304 38 962
Table A-6. Total AFG FY05 Awards for EMS Equipment
EMS Equip. Dept Type ITEM Combination All All Paid/ Paid On Total Volunteer Career Call/ Stipend Other 33 118 17 8 176 Computers 8 4 3 0 15 Defibrillators 26 19 7 6 58 BLS/ALS equipment 16 24 2 7 49 Mobile and Portable Communications 26 50 4 4 84 Total 109 215 33 25 382
Table A-7. Total AFG FY05 Awards for EMS Training
EMS Training Dept Type ITEM Combination All All Paid/ Paid On Total Volunteer Career Call/ Stipend 12 Lead Interpretation and 1 0 0 0 1 Thrombolytics 3 lead ECG interpretation and how to 0 0 0 1 1 use 12 lead ECG Advanced Hazmat Life Support 1 0 0 0 1 AED Training 0 1 0 0 1 AHA Training Media 1 0 0 0 1
A-185 APPENDIX A
EMS Training Dept Type ITEM Combination All All Paid/ Paid On Total Volunteer Career Call/ Stipend ALS 1 0 0 0 1 ALS training 1 0 0 0 1 Books and materials 0 1 0 0 1 Books and Materials - Emergency 0 1 0 0 1 Vehicle Operations Course Casualty Simulators 1 0 0 1 Certification - Respirator Fit Test 0 1 0 0 1 Certification and re-certification 0 0 1 0 1 expenses Certification courses, continuing 0 1 0 0 1 education and skills maintenance classes
Class room training with lecture, 10 01 discussion and hands on practice. Classroom 0 1 0 0 1 Classroom instruction 1 0 0 0 1 Classroom training with lecture, 1 0 0 0 1 discussion and hands on practice. Classroom videos 1 0 0 0 1 Clinical Assessment Skills 0 1 0 0 1 Continuing Education 0 1 1 1 3 cost of training and testing fees for 0 0 1 0 1 EMT-FR CPR Instructor Certification 1 0 0 0 1 CPR Training Manikins 0 1 0 0 1 CPR, Intubation, IV sticks, Monitoring 0 1 0 0 1 heart rhythms, blood pressure and chest wounds. CPR/AED 0 2 0 0 2 Emergency Medical Services 1 1 0 0 2 Emergency Medical Technician-Basic 0 0 1 0 1
EMS Training, books and supplies 0 0 0 1 1 EMT Basic 1 0 0 0 1 EMT Certification & Continuing 0 1 0 0 1 Education EMT Classes 0 1 1 0 2 EMT Training 0 1 0 0 1 EMT-B Class 1 0 1 0 2 EMTB/P Certification conducted at 0 0 1 0 1 facility
A-186 APPENDIX A
EMS Training Dept Type ITEM Combination All All Paid/ Paid On Total Volunteer Career Call/ Stipend EMT-I B 0 1 0 0 1 Exam and Course Fees 0 0 0 1 1 Extrication equipment training 0 1 0 0 1 First Responder Courses for all Fleming 0 0 1 0 1 County Fire Department personnel
First Responder Recertification Course 0 1 0 0 1 first-aid / C.P.R. 0 1 0 0 1 General Training as lsited in narrative- 0 0 1 0 1 Fees Incident Command 1 0 0 0 1 Initial certification 1 0 0 0 1 Instructional cost for training, Books and 0 0 1 0 1 materials, Training equipment and supplies, Exam and fees Instructional Cost/Books and Materials 0 0 1 0 1
Instructional costs for EMS training 0 2 1 1 4 instructor lead training 1 0 0 0 1 Lifelike Patient handling & extrication 0 1 0 0 1
Medical continuing education 0 0 1 0 1 N.I.M.S. (NATIONAL INCIDENT 0 1 0 0 1 MANAGEMENT SYSTEM) Paramedic Training 0 1 0 0 1 Recertification 0 0 0 1 1 Required CPR Training 0 1 0 0 1 State Certified training 1 0 0 0 1 The Uses of the LifePak 12 as an AED 1 0 0 0 1
Total cost of an EMT-B class and CPR 0 0 0 1 1 certification for 10 to 12 people Train 6 EMT-Bs to be EMT-P 1 0 0 0 1 Training Equipment and Supplies 5 3 0 1 9 Training props 1 0 0 0 1 Water/Flood Rescue 0 1 0 0 1 wilderness first aid 0 1 0 0 1 Total 26 31 13 8 78
A-187
A-188 APPENDIX B
DATA COLLECTED BY THE NATIONAL FIRE INCIDENT REPORTING SYSTEM (NFIRS)
Additional information about uses and limitations of NFIRS data is provided in Appendix D NFIRS Data Collected NFIRS Module Data Collected Module 1. BASIC • Fire Department Identifier 7. HAZMAT • Hazardous materials MODULE • Location MODULE (to identification (Collect • Incident Type document • Container information information • Dates/Times/Shifts REPORTABLE • Release amounts and common to • Actions Taken hazmat location all incidents) • Dollar Losses and Values incidents) • Actions taken Casualties • Mitigation factors • Hazmat Releases • Property Use • Persons and Entities Involved 2. FIRE • Property Details 8. WILDLAND • Property details MODULE • On-Site Materials FIRE MODULE • Fire cause • Ignition: Area of Fire Origin, (to document • Ignition information Source of Ignition, Material REPORTABLE • Fire suppression and mgmt Ignited, Factors Contributing, wildland fires) • Mobile property type Human Issues, Equipment • Equip involved in ignition Involved • Weather data • Human Factors Involved • Fuel model at origin • Mobile Property Description • Total acres burned • Fire Origin and Spread • Property management Description • Person responsible • Fire Suppression Factors • Fire behavior 3. • Structure type 9. • Apparatus identification STRUCTUR • Building status, height, main APPARATUS and type E FIRE floor size MODULE • Dispatch, arrival, clear MODULE • Fire origin, fire spread, (optional dates and times number of stories damaged module to help • Actions taken by each by flame manage and apparatus • Material contributing to flame track apparatus • Number of personnel used spread and resources on each apparatus • Presence of detectors, used on detector type, power supply, incidents) detector operation, detector effectiveness, detector failure reason • Presence of automatic extinguishments system (AES), type of AES, AES
B-1 APPENDIX B
NFIRS Data Collected NFIRS Module Data Collected Module operation, AES effectiveness and AES failure reason 4. • Person’s identification 10. • Apparatus identification, CIVILIAN • Demographic information PERSONNEL type FIRE • Injury causes, including MODULE • Dispatch, arrival, clear CASUALTY human and contributing (Note: The dates and times MODULE factors personnel • Use (only for fire • Activity when injured module or the • Actions taken by each casualties) • Location when injured apparatus/resour apparatus • Symptoms, part of body ces module may • Personnel ID, rank, actions injured be used, but not taken both) • Disposition 5. THE FIRE • Person’s identification and 11. ARSON • Agency investigating SERVICE age MODULE (used • Case status CASUALTY • Injury time whenever the • Suspected motivation MODULE • Assignment and activity at Cause of factors (only used time of injury Ignition is coded • Entry methods, devices, when fire • Severity and injury and as “intentional,” other information service disposition or as “under • Property ownership personnel • Location of victim when investigation.”) • Laboratory used suffer an injured The Juvenile Firesetter section injury, fall, • Symptoms and portion of includes information: or exposure body injured • Age, gender, race, and involved • Cause of injury, factors Ethnicity of each juvenile with any contributing, object involved involved. incident) • Where injury occurred • Family Type • Equipment profiles • Suspected motivation and risk factors • Disposition 6. EMS • Incident location and type • SUPPLEMENTAL FORM (used as a local MODULE • In-service dates and times option for recording additional persons or (optional- • Provider assessment entities involved in the incident for those only used • Victim demographics departments that use paper-based incident when that • Injury/illness description reporting) option has • Procedures used been chosen • Safety equipment involved • NARRATIVE REPORT (serves as an official, by state or • Care level legal record of an incident and must accurately local describe the incident and the actions taken to • Patient status authorities) mitigate it. While many of these facts may be collected in uniform, coded fields – some information can best be presented in a detailed narrative.)
B-2 APPENDIX C
CDC INVENTORY OF NATIONAL INJURY DATA SYSTEMS
The CDC has compiled a list of 43 different federal data systems operated by 16 difference agencies and 3 private registry systems that provide nationwide injury-related data.
Each data system is listed along with the agency or organization and associated web sites. Additional information about survey/surveillance methods and findings is available by navigating to the web address provided. Data systems are organized according by topic. Some data systems are listed more than once relevant to different violence- and injury-related topics. Data systems that have state-based data are noted with an asterisk (*).
Behavioral Risk Factors/Injury Incidence
• Behavioral Risk Factor Survey System (BRFSS)*, CDC-NCCDPHP, annual survey URL: http://www.cdc.gov/brfss/ • Youth Risk Behavior Survey (YRBS)*, CDC-NCCDPHP, biannual school-based survey for state/local, occasional national survey URL: http://www.cdc.gov/nccdphp/dash/yrbs/index.htm • Injury Control and Risk of Injury Survey (ICARIS I, II, II½ ), CDC-NCIPC, periodic survey Conducted by NCIPC. No web site available. • National Health Interview Survey (NHIS), CDC-NCHS, annual household survey URL: http://www.cdc.gov/nchs/nhis.htm
Injury Morbidity Data
• National Electronic Injury Surveillance System (NEISS), CPSC, NEISS All Injury Program, CPSC/CDC-NCIPC, and NEISS-work-related injury, CPSC/CDC-NIOSH, ongoing URL: http://www.cpsc.gov/epidemiology/ URL: http://www.cpsc.gov/library/neiss.html URL: http://www.cdc.gov/ncipc/wisqars (nonfatal injury data) URL:http://www2a.cdc.gov/risqs/default.asp (work-related injury) • National Hospital Ambulatory Medical Care Survey (NHAMCS), CDC-NCHS, annual survey URL: http://www.cdc.gov/nchs/about/major/ahcd/ahcd1.htm • National Ambulatory Medical Care Survey (NAMCS), CDC-NCHS, annual survey URL: http://www.cdc.gov/nchs/about/major/ahcd/ahcd1.htm • National Hospital Discharge Survey (NHDS), CDC-NCHS, annual survey URL: http://www.cdc.gov/nchs/about/major/hdasd/nhds.htm • Healthcare Cost & Utilization Project (HCUP), AHRQ URL: http://www.ahrq.gov/data/hcup/ • Medical Expenditure Panel Survey (MEPS), AHRQ/CDC-NCHS URL: http://www.meps.ahrq.gov/WhatIsMEPS/Overview.HTM
C - 1 APPENDIX C
• Indian Health Service-Ambulatory Care & Inpatient Care Systems, IHS, ongoing URL: http://www.ihs.gov/nonmedicalprograms/ihs%5Fstats/ • Outcome and Assessment Information Set (OASIS), CMS, Home Health Care System, ongoing URL: http://www.cms.hhs.gov/oasis • Minimum Data Set For Nursing Home Resident Assessment and Care Screening (MDS- Nursing Homes), CMS, URL: http://www.cms.hhs.gov/states/mdsreports
Injury Deaths-Death Certificates
• National Vital Statistics System (NVSS)*, CDC-NCHS, ongoing URL: http://www.cdc.gov/nchs/nvss.htm • National Mortality Follow-back Survey- 1993 (NMFS93), CDC-NCHS, periodic URL: http://www.cdc.gov/nchs/about/major/nmfs/nmfs.htm
Automotive/Transport Injury Data
• Fatality Analysis Reporting System (FARS)*, NHTSA, ongoing, MV crash deaths URL: http://www-fars.nhtsa.dot.gov/ • National Automotive Sampling System-General Estimates System (NASS-GES), NHTSA, annual URL: http://www-nrd.nhtsa.dot.gov/departments/nrd-30/ncsa/ges.html • National Automotive Sampling System-Crashworthiness Data System (NASS-CDS), NHTSA, annual URL: http://www-nrd.nhtsa.dot.gov/departments/nrd-30/ncsa/cds.html • NHTSA Special Crash Investigation (SCI), NHTSA, (includes air-bag related injuries), on-going. URL: http://www-nrd.nhtsa.dot.gov/departments/nrd-30/ncsa/sci.html Air Bag Fatality Summary Report & Tables. Cases shown as having a case status of "Available" can be found at: http://www-nass.nhtsa.dot.gov/BIN/logon.exe/airmislogon • DOT Federal Railroad Administration, Railway Safety Statistics*, annual (includes statistics on fatal and nonfatal injuries associated with train collisions) URL: http://safetydata.fra.dot.gov/OfficeofSafety/Default.asp
Automotive Behavioral Injury Data
• National Occupant Protection Use Survey (NOPUS), NHTSA, periodic URL: http://www-nrd.nhtsa.dot.gov/departments/nrd-01/summaries/4313ga.html • Motor Vehicle Occupant Safety Survey, NHTSA, biennial URL: http://www.nhtsa.dot.gov/people/injury/research/ • National Survey of (Drinking and Driving)/(Distracted and Drowsy Driving)/(Speeding and Unsafe Driving) Attitudes and Behaviors, NHTSA, periodic URL: http://www.nhtsa.dot.gov/people/injury/research/
Occupational Injury Data
C - 2 APPENDIX C
• National Traumatic Occupational Fatality Surveillance System (NTOF)*, CDC-NIOSH, URL: http://www.cdc.gov/niosh/injury
• National Electronic Injury Surveillance System – Work RISQ – Nonfatal work-related injuries, CDC-NIOSH, URL: http://www2a.cdc.gov/risqs/default.asp • Census of Fatal Occupational Injuries (CFOI)*, BLS, ongoing URL: http://www.bls.gov/iif/oshfat1.htm • Survey of Occupational Injuries and Illnesses (SOII)*, BLS, annual survey URL: http://www.bls.gov/iif/home.htm • Census of Agriculture (COA)*, DOC, periodic URL: http://www.nass.usda.gov/Census_of_Agriculture/index.asp
Violent Death Data
• National Incident Based Reporting System (NIBRS)*, FBI, ongoing URL: http://www.fbi.gov/hq/cjisd/ucr.htm • Law Enforcement Officers Killed and Assaulted (LEOKA)*, FBI, ongoing URL: http://www.fbi.gov/ucr/ucr.htm • National Violent Death Reporting System (CDC-NVDRS)*, CDC-NCIPC, 17 states URL: http://www.cdc.gov/ncipc/profiles/nvdrs/facts.htm • Uniform Crime Reports-Supplemental Homicide Reports (UCR-SHR)*, FBI, ongoing URL: http://www.fbi.gov/ucr/ucr.htm
Crime and Victimization Data
• National Crime Victimization Survey (NCFS), BJS, annual survey URL: http://www.ojp.usdoj.gov/bjs/cvict.htm • National Child Abuse and Neglect Data System (NCANDS)*, Child Welfare Information Gateway, ACF, annual, URL: http://www.childwelfare.gov/ • National Incidence Study of Child Abuse and Neglect (NIS), Child Welfare Information Gateway, ACF, periodic, URL: http://childwelfare.gov/systemwide/statistics/nis.cfm • National Violence Against Women Survey, conducted only once, in 1995-1996, by NIJ/CDC-NCIPC. Although there is no web site, publications from the survey are all available on the web at: URL: http://www.ncjrs.org/pdffiles1/nij/181867.pdf URL: http://www.ncjrs.org/pdffiles1/nij/183781.pdf URL: http://www.ncjrs.org/pdffiles/169592.pdf URL: http://www.ncjrs.org/pdffiles/172837.pdf
C - 3 APPENDIX C
Drug Abuse Data
• Drug Abuse Warning Network (DAWN), SAMHSA, ongoing URL: http://www.dawninfo.samhsa.gov • National Survey on Drug Use and Health, formerly called the National Household Survey on Drug Abuse (NHSDA), SAMHSA, annual URL: http://www.oas.samhsa.gov/nhsda.htm • Monitoring the Future Survey (MTFS), NIDA, school-based, annual URL: http://www.nida.nih.gov/DrugPages/mtf.html • Youth Risk Behavior Survey (YRBS)*, CDC-NCCDPHP, school-based, biennial URL: http://www.cdc.gov/nccdphp/dash/yrbs/index.htm
Other Injury Data
• National Fire Incident Reporting System (NFIRS), USFA, ongoing URL: http://www.usfa.fema.gov/inside-usfa/nfirs.cfm • Medicare Provider Analysis and Review (MEDPAR) Limited Data Sets, CMS, ongoing URL: http://www.cms.hhs.gov/LimitedDataSets/ • Medicare Current Beneficiary Survey (MCBS), CMS, ongoing URL: http://www.cms.hhs.gov/apps/mcbs/
PRIVATE INJURY DATA SYSTEMS
Trauma care/poisoning data
• National Trauma Data Bank (NTDB) – American College of Surgeons, ongoing URL: http://www.facs.org/trauma/ntdb.html • Toxic Exposure Surveillance System (TESS) – American Assoc. Poisoning Control Centers, annual URL: http://www.aapcc.org/poison1.htm • United States Eye Injury Registry – American Society of Ocular Trauma URL: http://www.useironline.org
C - 4 APPENDIX D
DATA SOURCES FOR PERFORMANCE MEASUREMENT
This appendix provides additional analysis of the major incident-based reporting systems and the major survey data sets that are discussed in Chapter 8.
MAJOR INCIDENT-BASED REPORTING DATA SOURCES
The following sections provide explanation of the proposed sources of national fire and EMS data described in the report. Among incident-based reporting data sources, this section covers the following:
• NFIRS • CPSE/CFAI Vision Software • Consumer Product Safety Commission (CPSC) • National Electronic Injury Surveillance System (NEISS) • CDC National Vital Statistics • The National Emergency Medical Services Information System (NEMSIS)
National Fire Incident Reporting System (NFIRS)
Incident data can be used by fire departments and other emergency service providers to document their experience, support management decisions, and identify, prepare and justify budget requests. Each local agency then can send their incident data to the state electronically, where the information is combined with data from other fire departments into a statewide database where trends can be recognized to aid in the development of safety and prevention programs including statewide standards. State data are then sent to the National Fire Data Center (NFDC) at the United States Fire Administration for analysis that includes comparisons across states and metropolitan areas to develop national public education programs, make recommendations for national codes and standards, and guide allocation of federal funds.
“This system is based on the work of the National Fire Information Council (NFIC) and the National Fire Protection Association (NFPA) Technical Committee on Fire Reporting. The NFPA Technical Committee on Fire Reporting is responsible for developing and maintaining NFPA 901, Standard Classifications for Incident Reporting and Fire Protection Data. This standard establishes basic definitions and terminology for use in incident reporting and serves as a means of classifying data so that the information can be aggregated (DHS, 200X).
In addition to a basic module that collects data for all incident types, more detailed information can be collected with other modules to further profile fires, structure fires, civilian fire casualties, fire fighter casualties, hazardous materials, wild land fires, arson, apparatus, personnel, and EMS incidents as necessary. See Appendix B for the data captured by NFIRS.
If fully and consistently utilized, NFIRS data can be used to:
• Describe and pinpoint community fire and emergency locations, causes and other factors
D - 1 APPENDIX D
• Support budget requests with statistical data
• Support fire code refinement
• Assess fire code enforcement or fire education programs
• Plan for future fire protection needs
• Improve allocation of resources (by location and by time)
• Support fire engineering models and analysis
Limitations of NFIRS include:
• Use is optional in most jurisdictions. AFG requires applicants to subscribe and to confirm they are using it, but no integration exists to verify this. There is also no requirement for fire departments to continue use after the grant period of performance ends.
• According to Mick Ballesteros, Epidemiologist and Acting Team Leader of the Home and Recreation Injury Prevention Team, CDC's Injury Center, a serious weakness in NFIRS is the lack of quality control and gaps in reporting (personal interview, September 21, 2006). Further research should identify whether those fire departments not participating bias the results of data analysis performed through NFIRS (e.g., a lower number of small and rural departments participate).
• While the benefits of NFIRS to researchers and policy makers is clear, the cost to each individual fire department is significant and sometimes without a clear payback to the department. These costs are mitigated in three ways:
o First, states that mandate the use of NFIRS typically provide funds for the localities to implement it. For example, the State of Washington gives funds for the purposes of getting NFIRS submission process set up in the amount of $62,000 per award (although not clear whether this is coming from State general revenues or from homeland security federal funds or a combination). With an average AFG award of $82,000 [confirm], it is clear that a $62,000 investments required to participate would be a deterrent to many departments (Office of the State Fire Marshal, Washington State Patrol, 2006).
o Second, the U.S. Fire Academy offers free classes on how to fill out NFIRS reports (required before using NFIRS) as well as data analysis classes to help officials benefit from NFIRS data at the local level (available in Emmitsburg, MD, over the internet, and through limited on-site formats). They also provide a Geospatial Information System (GIS) class in Wilson, South Carolina, to demonstrate how fire departments can increase their safety and effectiveness by accessing and integrating GIS data. The Fire Academy is further enhancing their
D - 2 APPENDIX D
distance learning capabilities. An underlying assumption with these programs is that unless the fire departments use the data locally, they will not see the value in entering high-quality data for State and Federal purposes.
o Third, the U.S. Fire Administration, on a matching basis, provides one-time grant funds to requesting fire departments to cover hardware and software costs (no reimbursement for salaries).
An October, 2006, estimate by the U.S. Fire Administration was that 20,000 fire departments are participating out of a fire department population of approximately 30,000. The adoption rate is considered sufficient for the intended sample analysis by the U.S. Fire Administration. The number of participating departments for 2003 was 17,761, which was more than a 13% increase over 2002 and “there have been similar annual increases each year since 2000” (Furr, 2005). In fact, according to focus group participants from the U.S. Fire Administration (personal interview, October 19, 2006) “we’re not necessarily looking for universal coverage.” It was not our idea, or our intention, to require NFIRS for the grant program. We have a large enough sample size. We’re more concerned with data quality. And we’re concerned about making our sample more representative.”
Participants in NFIRS comprise the National Fire Information Council (NFIC), which includes 42 states, the District of Columbia and 31 metropolitan jurisdictions with nearly 14,000 fire departments participating throughout the nation. NFIC is described as a partnership of Federal, State and local participants which is itself partnered with the U.S. Fire Administration/DHS is through a cooperative agreement to provide federal funding.
NFIC’s Strategies to “fight fire with facts” include:
• System development and expansion
• Integration of new computer information technologies
• Technical assistance to member states/metros
• Regional and national training workshops
• Data analysis
• Use of data for public fire safety awareness education
Among others, its objectives include “To enhance the quality of life for all people by employing NFIRS data to assist in developing effective fire prevention and protection strategies”
Many fire departments are using the 1999 “4.1” version of NFIRS because it would cost too much to upgrade. A Blue Ribbon Commission’s report from 1999, in testimony to a House
D - 3 APPENDIX D
Committee recommends that “an additional $2 million per year be put toward state grants for the marketing, training, and creation of incentives to ensure 100% NFIRS 5 participation.” 116
Center for Public Safety Excellence (CPSE) VISION Software
CPSE offers software called VISION that allows fire and emergency service agencies to collect data online, import existing data from third party applications, and serve as a central database of benchmarking information for fire and emergency service agencies nationwide. The software allows agencies to update and exchange community data with ISO and with all participating organizations. It will provide a means to develop a comprehensive community risk hazard analysis, access comparative data, and will incorporate this information into a GIS-based integrated risk/deployment model. The fee structure for the data analysis software, VISION is based on population, with annual fees ranging from $195 to $15,995 and setup fees ranging from $250 to $30,000. (CPSE, 2006)
Centers for Disease Control (CDC)/ Consumer Product Safety Commission (CPSC) National Electronic Injury Surveillance System (NEISS)
The U.S. Consumer Product Safety Commission (CPSC) has operated a statistically valid injury surveillance system known as the National Electronic Injury Surveillance System (NEISS).
The primary purpose of NEISS has been to provide timely data on consumer product-related injuries occurring in the U.S. In the year 2000, CPSC initiated an expansion of the system to collect data on all injuries. With the expansion, NEISS becomes an important public health research tool, not just for CPSC, but for users throughout the U.S. and around the world. NEISS injury data are gathered from the emergency departments of 100 hospitals selected as a probability sample of all 5,300+ U.S. hospitals with emergency departments. The system's foundation rests on emergency department surveillance data, but the system also has the flexibility to gather additional data at either the surveillance or the investigation level. (CPSC, 2006)
Surveillance data entered through NEISS enables CPSC analysts to estimate the number of injuries associated with (not necessarily caused by) specific consumer products. Based on the associations, CPSC often initiates additional follow-up research to understand whether there is a causal relationship that must be ameliorated (e.g., a product recall, a public awareness campaign, or a product safety standard).
At the sampled hospitals participating in the NEISS program, information as to how the injury occurred is entered by staff members into the patient's medical record. At the end of each day, a NEISS hospital coordinator reviews all daily records, flagging those that meet the criteria for
116 Testimony of Stephen P. Austin, Director of Government Relations, International Association of Arson Investigators Chairman of the Blue Ribbon Panel, for the United States House of Representatives, Basic Research Sub-Committee, House Science Committee, March 23, 1999 (Available at (http://www.house.gov/science/austin_032399.htm).
D - 4 APPENDIX D
inclusion in NEISS and transcribes the data in coded form for upload to the central system. (CPSC, 2006)
However, one limitation is that reporting categories/codes do not use an adequately specific level of granularity to identify fire-related causes of injury, e.g., “burn” may be related to a fire in home or to an industrial accident involving hot machinery.
CDC National Vital Statistics
The National Vital Statistics System is the longest-running example of inter-governmental data sharing in Public Health based on a system of relationships, standards, and procedures to collect and disseminate the Nation's official vital statistics. It consolidates the officially recorded data about births, deaths, marriages, divorces, and fetal deaths from each responsible government entity. While this census approach to data collection yields high-quality data, it results in a lag of three years or more (i.e., 2004 data will be available late in 2006). Data are available in summary reports and in searchable format through the CDC WISQARS website free of charge.
The CDC data include a breakdown of residential fire deaths by demographic categories. It shows, for example, that Blacks or African Americans are three times more likely than Whites to die in residential fires. It also shows that males are 78% more likely to die in residential fires than females. Further, those with some college education are only 20% as likely to die in a residential fire as those with less than a high school education. There is no causal relationship drawn by this study for these differences.
CDC also has an injury prevention center that has joined with other federal agencies and non- federal organizations since 2002 in a Fire Safety Council, which has a website and is committed to ending residential fire deaths by 2020. According to Mick Ballesteros, PhD., epidemiologist and Acting Team Leader of the Home and Recreation Injury Prevention Team, CDC's Injury Center, “I would like to see a standard [public education/safety] curriculum developed that AFG would fund in all places.” (personal interview, September 21, 2006).
The National Emergency Medical Services Information System (NEMSIS)
The National Emergency Medical Services Information System (NEMSIS) is the future national repository intended to collect and consolidate EMS data from every state (NEMSIS, 2006). NEMSIS is cosponsored by NHTSA and the Health Resources and Services Administration (HRSA) and is operated by the National Association of State EMS Officials.
This is a future solution to standardize state and local data sets using an XML standard, across the continuum of care. While NEMSIS is in its developmental stages, it has a memorandum of understanding in place with 48 states and two territories as well as partnerships with thirteen professional organizations and agencies including the CDC, FEMA/DHS and the NHSTA.
D - 5 APPENDIX D
MAJOR SURVEY DATA SETS
The following sections provide explanation of the proposed sources of national fire and EMS data described in the report. Among survey reporting data sets, this section covers the following:
• NFPA/USFA Needs Assessment Census • AFG “eGrants” Grant Management System (GMS) • USFA Firefighter Fatality Database • NFPA US Firefighter Fatalities • Injury Control and Risk Survey (ICARIS) • CDC National Fire Risk Factor Survey (NFRFS)
NFPA/US Fire Administration’s Needs Assessment
In 2001 and again in 2005, DHS/FEMA conducted a study in conjunction with NFPA to:
• define the current role and activities associated with the fire services; • determine the adequacy of current levels of funding; and • provide a needs assessment to identify shortfalls.
Out of the 26,354 departments mailed surveys, 46% responded. The findings are grouped by revenue and budgets; personnel and capabilities; fire prevention and code enforcement; facilities, apparatus and equipment; communications and communications equipment; ability to handle unusually challenging incidents; and new and emerging technology.
As described in Chapter 4, DHS commissioned the NFPA to undertake an additional effort to match the AFG grants with the findings from the 2005 needs assessment. This was made possible by matching survey respondents with AFG applicants through NFIRS IDs and/or address information. NFPA was able to match 97% of award funds with a need expressed in a survey.
AFG “eGrants” Grant Management System (GMS)
In addition to supporting the grant lifecycle (See Chapter 2), the GMS has additional features such as audits, workflow, internal messaging, accounts manager, etc. The system has two interfaces: grant applicants and internal program administration. The application is divided into its logical pieces: (1) Overview, (2) Contact Information, (3) Applicant Information (FD, Nonaffiliated EMS, Regional Request—which requires you to choose which jurisdictions are affiliated), (4) Department characteristics (I&II) (% of career members, etc), (5) Department Call Volume, (6) Request Information, (7) Request Details, (8) Budget, (9) Narrative Statement, (10) Assurances and Certifications, (11) Review Application, (12) Submit Application.
Applicants are required to submit a new application every year. They are not able to modify previous apps because the questions may have changed or people may not fill out the questions
D - 6 APPENDIX D
with as much detail as they should for values that have changed. Accurate and detailed information is important because some decisions are based on values like the number of riding positions (SCBA would not be awarded in excess of their riding positions).
Scoring is done by an algorithm outside of the GMS system. Then the peer review panel gets printed applications to evaluate. After manually-scored values are imported back into the GMS, the results of the panel review are visible in the GMS. It shows each panelist’s name, plus a narrative score, cost benefit score, financial need score plus any comments from the reviewers.
Both a benefit and a challenge of the current GMS is its reliance on narrative statement sections. The narratives allow applicants to express their situations and needs in ways that would be difficult to do with quantitative values, but they also make it difficult to objectively analyze surveys and difficult to separate one’s grant-writing ability from their true need. The system does have the ability to search the text for key words on an ad-hoc basis. One approach for improving the ability to analyze applications would be to build more robust key word queries of the database. Another approach is to look at commonly-requested narrative text queries to find candidates for individual data fields (e.g., “Was this grant for strategic uses: yes/no”). Another approach is to come up with an information framework and figure out what are the high priority questions to ask applicants, such as the questions related to the National Preparedness Goal if that strategic use of the AFG program is selected. Similarly, beginning 2005 the AFG program has begun asking applicants for 3-year histories of injuries and deaths as a way to gather more objective data.
Firefighter Fatality Data Sources: USFA and NFPA
There are two sources of firefighter fatality data that are relatively comparable. The NFPA produces a periodic report with analysis of the causes of firefighter deaths, as does the US Fire Administration. The sources of fatality data include news reports, communication with fire departments, states, and other sources.
Generally there have been about one hundred firefighter fatalities recorded per year, but these two data sources typically differ by about two deaths per year (although they differ by 12 for 2005 data). One main difference is that the USFA count includes heart attacks and strokes within 24 hours of an incident while the NFPA does not. The International Fallen Firefighters union has its own database as well, but that does not include volunteers.
The USFA produces a finalized fatality report for the public domain by April/May for the previous year, followed by year-to-date revisions each subsequent month.
Injury Control and Risk Survey (ICARIS)
The CDC performed this data collection most recently in 2003. It is generally done about once per decade, with the previous version in 1994. Among other topics, it includes questions about smoke alarms, and fire escape plans. The 2003 version has been accepted for publication, but is currently pending release.
D - 7 APPENDIX D
CDC National Fire Risk Factor Survey (NFRFS)
NFRFS gathers data on fire-related injury risk factors, such as smoking, and protective factors, such as regular maintenance of smoke alarms. The idea is that “understanding the level of fire- injury risk factors in a population is also a valuable way to help state and local fire-injury control programs set priorities and guide assessment of interventions and program performance.” (http://www.firesafety.gov/programs/).
It includes secondary analysis of data from the following primary sources:
• CPSC's National Electronic Injury Surveillance System (NEISS)
• Injury Control and Risk Survey (ICARIS), the National Health Interview Survey (NHIS)
• Vital Statistics data, the National Electronic Injury Surveillance System - All Injury Program (NEISS-AIP)
• Behavioral Risk Factor Surveillance System (BRFSS)
For example, it provides a breakdown of residences with functioning smoke alarms on every floor by demographic categories. It shows that Hispanic or Latino residents are 7% less likely to have a functioning smoke alarm on every level than non-Hispanic/Latinos. It also shows that those with less than a high-school education are 7% less likely to have a smoke alarm on every floor (CDC, 1999).
D - 8 APPENDIX E
INSURANCE SERVICE OFFICE (ISO) PROGRAMS AND DATA
ISO supplies data, analytics, and decision-support services to fields including insurance, finance, real estate, health services, government, and human resources to help customers measure, manage, and reduce risk. ISO collects information useful to insurance underwriting, including assessments of public fire protection, flood risk, and the adoption and enforcement of building codes in individual communities. ISO performs the assessments as a service to the insurance industry and does not charge a fee to the communities.
Two of ISO’s programs are the Public Protection Classification (PPC) program and the Building Code Effectiveness Grading Schedule (BCEGS) program, each described below.
The Public Protection Classification (PPC) program
ISO assesses municipal fire-protection efforts in communities throughout the United States using its PPC program. ISO's staff collects information about the quality of public fire protection in more than 44,000 fire districts across the United States. In each of those fire districts, ISO analyzes the relevant data and assigns a Public Protection Classification — a number from 1 to 10. Class 1 represents exemplary fire protection, and Class 10 indicates that the area's fire- suppression program does not meet ISO's minimum criteria. Using a manual called the Fire Suppression Rating Schedule (FSRS), ISO objectively assesses three major areas:
• Fire alarm and communications systems—A review of the fire alarm system accounts for 10% of the total classification. The review focuses on the community's facilities and support for handling and dispatching fire alarms.
• The fire department—A review of the fire department accounts for 50% of the total classification. ISO focuses on a fire department's first-alarm response and initial attack to minimize potential loss. Here, ISO reviews such items as engine companies, ladder or service companies, distribution of fire stations and fire companies, equipment carried on apparatus, pumping capacity, reserve apparatus, department personnel, and training.
• The water supply—A review of the water-supply system accounts for 40% of the total classification. ISO reviews the water supply a community uses to determine the adequacy for fire-suppression purposes. They also consider hydrant size, type, and installation, as well as the inspection frequency and condition of fire hydrants (ISO,2006).
A community's investment in fire mitigation, defined by ISO to include “fire-protection features only as they relate to suppression of fires in structures” (ISO, 2006), is used as reliable predictor of future fire losses. So insurance companies use PPC information to help establish fair premiums for fire insurance — generally offering lower premiums in communities with better protection. Many communities use the PPC as a benchmark for measuring the effectiveness of their fire-protection services. The PPC program is also a tool that helps communities plan for, budget, and justify improvements. See Figure E-1 for a map of the communities across the US
E - 1 APPENDIX E that have achieved a class 1 rating and Figure E-2 for a distribution of the number of communities that have achieved each score.
Figure E-1. Communities with a PPC Class 1 (exemplary) Ratings (ISO 2006)
Figure E-2. Nation-wide distribution of PPC Class (ISO, 2006)
There is a correlation between ISO rating and commercial and residential property loss from fire. On average, per $1,000 of insured property, communities in the worst classification had commercial-property fire losses more than three times as high as communities in the best classification. And communities in the worst classification had homeowners fire losses more than twice as high as communities in the best classification (ISO, 2006). See Figures E-3 and E- 4. below for the average claims by ISO classification.
E - 2 APPENDIX E
Figure E-3. Cost Of Fire Claims Per Figure E-4. Cost Of Fire Claims $1,000 Of Insured Commercial Property Per $1,000 Of Insured Homeowner (ISO, 2006) Property (ISO, 2006)
The Building Code Effectiveness Grading Schedule (BCEGS) program
ISO assesses the building codes in effect in individual communities and how those communities enforce their building codes. The building code assessments (BCEGS) place special emphasis on mitigation of losses from natural hazards. The concept is that municipalities with well- enforced, up-to-date codes should demonstrate better loss experience, and insurance rates can reflect that. The BCEGS program assigns each municipality a BCEGS grade of 1 (exemplary commitment to building-code enforcement) to 10.
ISO gives insurers BCEGS classification information for determination of premiums, similar to the way it does with PPC program discussed above. But while the PPC score applies to all structures in the community (since all are affected by such factors as water supplies and fire department staffing), the community’s BCEGS scores in any given year are permanently assigned to the buildings awarded certificates of occupancy in that year. In other words, if a community had a BCEGS score of 3 in 2002, all buildings awarded certificates of occupancy in 2002 are thereafter assigned a score of 3, even if the community improves to a level of 2 the following year. ISO reassesses each community at least every five years, but classifications for existing buildings will not change as a result of a community's reclassification.
A community's classification is based on (ISO, 2006):
Administration of codes, including:
• Building-code edition in use • Modification of the codes • Zoning provisions to mitigate natural hazards • Training of code enforcers • Certification of code enforcers • Incentives for outside education/certification • Building officials' qualifications • Contractor/builder licensing and bonding • Public-awareness programs
E - 3 APPENDIX E
• Participation in code-development activities and the appeal process
Review of building plans, including:
• Staffing levels • Qualifications • Level of detail of plan review • Performance assessments • Review of plans for one- and two-family dwellings, multifamily dwellings, and commercial buildings
Field inspections, including:
• Staffing levels • Qualifications • Level of detail of inspections • Performance assessments • Final inspections • Issuance of certificates of occupancy
In addition, ISO collects underwriting information, including natural hazards common to the area, number of inspection permits issued, number of inspections completed, the building department's funding mechanism and date of establishment, size of the jurisdiction and population, and the fair-market value of all buildings.
See a nationwide distribution of these grades in Figure E-5 below:
E - 4 APPENDIX E
Figure E-5. Distribution of Communities by BCEGS Class (number represents quantity within each classification) (ISO, 2006)
Note: The personal lines classification addresses building code adoption and enforcement for 1- and 2-family dwellings. The commercial lines classification is for all other buildings.
E - 5
E - 6 APPENDIX F
INDIVIDUALS INTERVIEWED OR CONSULTED
Alex Furr, Division Director, Director of USFA's National Fire Data Center (NFDC)
Amit Yadav, Program Manager, REI Systems
Bill Latta, President, Res Q Tec
Bob Krepps, President, IDEX Corp
Bradley S. Pabody, United States Fire Administration (USFA); Fire Program Specialist, Preparedness Directorate, U.S. Fire Administration, U.S. Department of Homeland Security (DHS)
Bill Franz, Vice President, Battalion Three Inc.
Carey Cozad, Chief Executive Officer, Wheatland Fire Equipment Company
Carl E. Peterson, Assistant Division Director, Public Fire Protection Division, National Fire Protection Association (NFPA)
Dalit Baranoff, Professor of History, University of Maryland, MD
David Janssen,* Chief Administrative Officer, Los Angeles County, CA
Dennis Stark, Fire Marshal, City of Alexandria, MN
Doug Bonney, Regional Sales Manager, Kochek Co. Inc,
Doug Pitkin, Budget Examiner, Office of Management and Budget (OMB)
Drew Dawson, Division Director, EMS, National Highway Traffic Safety Administration (NHTSA)
Frank Fairbanks,* City Administrator, Phoenix, AZ
Gary O. Tockle, Assistant Vice President, Public Fire Protection, National Fire Protection Association (NFPA) International
Garey Cozad, CEO/Owner, Wheatland Fire Equipment Company
Gerry Hoetmer, Executive Director, Public Enterprise Risk Institute, (PERI)
George Horrace, VP, Operations, United Plastic Fabricating, Inc.,
Giff Swayne, Fire and Emergency Manufacturers and Services Association (FEMSA); President, Holmatro Rescue Equipment
F - 1 APPENDIX F
Glenn Gaines, Program Consultant, Office of Grants and Training, Fire Grant Program Office, U.S. Department of Homeland Security (DHS)
Craig Gerhart, County Administrator, Prince William County, VA
Gregg Pane, Director, Department of Health, DC
George (G.T.) Goros, Vice President Mfg. Operations, United Plastic Fabricating, Inc.
Harry R. Carter, Ph.D. CFO, Municipal Fire Protection Consultant
Heber C. Willis, III Branch Chief, Office of Grants and Training, Fire Grant Program Office, U.S. Department of Homeland Security (DHS)
Henry Wulf, Director, Governments Division, U.S. Census Bureau
Jack Snell, National Institute of Standards and Technology (NIST)
Jason Averill, Research Engineer, Fire Research Division & Building and Fire Research Laboratory, National Institute of Standards and Technology (NIST)
Jeffrey D. Johnson, Chief, Tualatin Valley Fire & Rescue, OR
Jim Heeschen, Statistician, U.S. Fire Administration, National Fire Data Center, U.S. Department of Homeland Security (DHS)
John W. Donahue, Chief, Maryland Institute for Emergency Medical Services Systems (MD)
John Aguilar Fisher, President, Cabezon Group
John Granby, Board Member, Fire and Emergency Manufacturers and Services Association (FEMSA); Vice President, Domestic Preparedness and Government Markets Protective Systems Group, Lion Apparel
John R. Hall, Jr., Assistant Vice President, Fire Analysis and Research, National Fire Protection Association (NFPA)
John Loonsk, M.D., Director, Office of Interoperability and Standards, United States Department of Health and Human Services (HHS)
John E. Sztykiel, President and Chief Executive Director, Spartan Motors Inc.
Jonathon McLaughlin, Vice president, Metro Fire
Kathy Patterson, Office of Grants and Training, U.S. Department of Homeland Security (DHS)
Ken Creese, Sales Manager, Sutphen Corporation
F - 2 APPENDIX F
Mark A. Whitney, Fire Programs Specialist, National Fire Data Center, United States Fire Administration, U.S. Department of Homeland Security (DHS)
Mary Beth Michos, Chief, Prince William City Dept. of Fire & Rescue, VA
Michael “Mick” Ballesteros, PhD Epidemiologist and Acting Team Leader Home and Recreation Injury Prevention Team, Centers for Disease Control
Michael Conduff,* Former City Manager, Denton, TX
Michael J. Karter, Jr., Senior Statistician, Fire Analysis and Research, National Fire Protection Association (NFPA) International
Mike Natchipolsky, Director of Sales, Firehouse.com
Mike Waters, Vice President of Risk Decision Services Department, Insurance Services Organization
Miriam Gerver, Cabezon Group
Pam Syfert,* City Manager, Charlotte NC
Patricia Malek, Branch Chief Policy Analysis Branch
Pinky Sheets, EGrants Support Manager, Assistance to Firefighters Grant Program, U.S. Department of Homeland Security (DHS); REI Systems
Ray Nichols, President, Nichols Oxygen Service Inc.
Robert Koedam, Chief, Triune-Halleck Volunteer Fire Department, Morgantown, WV
Robert P. Konupek, President, Apollo Fire Equipment Company
Robert W. Kreps, President, Fire and Rescue, IDEX Corporation
Robert Giffen, Program Director, Institute of Medicine
Rocky Lopes, Project Manager for Homeland Security, Community Services Division, National Association of Counties (NACo)
Ron Carlee,*∗ County Manager, Arlington County, VA
Ron Elmo, President, Guardian Fire Equipment
Ronny J. Coleman, Former Chief, Fullerton and San Clemente, CA; International Fire Chiefs Association (IAFC)
∗ Academy Fellow
F - 3 APPENDIX F
Scott Pattison, Executive Director, National Association of State Budget Officers
Scott Wysar, Vice President, Emergency One
Sheryl Sculley,* City Manager, San Antonio, TX
Stanford Stewart, National Fire Data Center, United States Fire Administration, U.S. Department of Homeland Security (DHS)
Steve Lawrence, Board Member, Fire and Emergency Manufacturers and Services Association (FEMSA); Rosecliff Partners, LLC
Terry Fisher, Vice President, McKing Consulting Corporation
Thomas J. Wieczorek, Executive Director, Center for Public Safety and Excellence (CPSE), VA
Tim Dean, President, Pro Poly
Timothy J. Bendle, U.S. Tanker—Fire Apparatus, INC.
Tony Paish, Deputy Director, World Fire Statistics Center
Tracey Trautman, Director, Central Division, Office for Domestic Preparedness, Preparedness Programs Division (PPD), U.S. Department of Homeland Security (DHS)
William G. Kowgios, MS, CFP, Vice President, Management Assistance Corporation
* Academy Fellow
F - 4 APPENDIX G
PANEL AND STAFF
PANEL
James Kunde, Panel Chair∗—Associate Professor and Program Coordinator, University of Texas, Arlington. Former Executive Director, Coalition to Improve Management in State and Local Government; Executive Director, Public Services Institute; Senior Staff Assistant, Center for Dispute Resolution, Southern Methodist University; Director of Programs, Charles Kettering Foundation; City Manager, City of Dayton, Ohio; City Development Director, Kansas City, Missouri.
Paul Brooks—Assistant Fire Chief, Greensboro, North Carolina Fire Department; Manager, Office of Strategic Planning and Information Systems; treasurer and member of NC State Firemen's Association; Vice-Chair, Commission on Fire Accreditation International; Member of the National Fire Protection Association 1710 Committee; faculty member, Fire and Rescue Management Institute at University of North Carolina, Charlotte; Consultant with Municipalities, non-profits, and other agencies for organizational development, strategic planning, and growth planning for distribution and concentration of fire resources. Former Program Manager, Greensboro Fire Department.
Harry Hatry*—Director, Public Management Program and former Director, State-Local Government Research Program, The Urban Institute. Former Associate, State-Local Finances Project, George Washington University; Office of Assistant Secretary of Defense for Systems Analysis.
Bruce McDowell*—President, Intergovernmental Management Associates. Former positions with U.S. Advisory Commission on Intergovernmental Relations: Director of Government Policy Research; Executive Assistant to the Executive Director. Former Director, Governmental Studies, National Council on Public Works Improvement. Former positions with the Metropolitan Washington Council of Governments: Director, Regional Management Information Service; Assistant Director, Regional Planning; Director, Program Coordination.
Darrel Stephens*—Police Chief, Charlotte/Mecklenburg, North Carolina. Former Police Chief and City Administrator, City of St. Petersburg, Florida; Executive Director, Police Executive Research Forum; Police Chief, City of Newport News, Virginia; Police Chief, City of Largo, Florida; Assistant Police Chief, City of Lawrence, Kansas; Police Officer, Kansas City, Missouri.
Glenn Corbett—P.E., Associate Professor of Fire Science at John Jay College of Criminal Justice in New York City, Technical Editor of Fire Engineering magazine, and an Assistant Chief of the Waldwick, New Jersey Fire Department. He recently served on the Federal Advisory Committee of the National Construction Safety Team and currently serves on the Fire Code Advisory Council for the State of New Jersey. He is the coauthor of Brannigan's Building
∗ Academy Fellow
G - 1 APPENDIX G
Construction for the Fire Service, 4th Edition. He has served as the Administrator of Engineering Services for the San Antonio, Texas Fire Department and as a Fire Protection Consultant for the Austin, Texas Fire Department.
STAFF
National Academy of Public Administration
J. William Gadsby∗—Vice President for Academy Studies, National Academy of Public Administration. Former Director, Management Studies Program, National Academy of Public Administration. Former positions with U.S. General Accounting Office: Senior Executive Service; Director, Government Business Operations Issues; Director, Federal Management Issues; Director, Intergovernmental and Management Issues. Former Assistant Director, Financial Management Branch, U.S. Office of Management and Budget.
Terry F. Buss, Ph.D.—Program Area Director. Former Dean, School of Policy and Management, Florida International University; Policy Director, Community Development and Planning, U.S. Department of Housing and Urban Development; Senior Policy Analyst, Congressional Research Service; Senior Strategy Advisor, World Bank; Senior Policy Advisor, Council of Governors’ Policy Advisors; consultant for U.S. Information Service and U.S. Agency for International Development in 10 countries; director of policy research centers at Ohio State, Youngstown State, and University of Akron; director of public management departments at University of Akron and Suffolk University. Two-time Fulbright Scholar— Hungary.
F. Steven Redburn*—Project Director for Academy Studies, National Academy of Public Administration. Former Chief, Housing Branch, U.S. Office of Management and Budget; Economist, Special Studies, U.S. Office of Management and Budget; Program Analyst, Office of Policy Development and Research, U.S. Department of Housing and Urban Development; Director, Center for Urban Studies, Youngstown State University.
Jim Frech—Director, Center for Intergovernmental Relations. Former non-profit positions included Senior Staff, Government Performance Project responsible for the design and implementation of a web-based surveying tool for a nationwide evaluation of state government management capabilities; Director, Governance Group of The Finance Project, which specialized in performance based management. Various private sector positions with Andersen Consulting, the Lewin Group, and Sapient specializing in bringing modern management practices to improve the operations of government programs. Government positions included policy analyst for the State legislature of Illinois, management analyst at OMB, and staff for Senator Patrick J. Leahy. Also, a former member of the Board and past-president of the National Grants Management Association and Project Advisor to the Administrative Conference of the United States.
∗ Academy Fellow
G - 2 APPENDIX G
Ednilson Quintanilla—Research Associate, National Academy of Public Administration. Former Positions: Supervisor’s Analyst, Santa Cruz County Board of Supervisors, County of Santa Cruz, CA; Alternate County Planning Commissioner, County of Santa Cruz, CA; Alternate County Transportation Commissioner, County of Santa Cruz, CA; Chair of the Board of Directors, Central Coast Energy Services, CA; Chair of the Board of Directors, Women’s Crisis Support, Santa Cruz, CA.
Martha S. Ditmeyer, Senior Administrative Specialist—Staff for a wide range of Academy studies. Former staff position at the Massachusetts Institute of Technology and the Communication Satellite Corporation.
Pivotal Insight, LLC
Rob Cimperman—Senior Consultant, Pivotal Insight LLC, Certified Project Management Professional (PMP); Former positions: Director of Process Competency Group at ExcelaCom Consulting; Manager of Process and Technology Integration at XO Communications; Technology Analyst at American Management Systems (AMS); Founder of UATsolutions to build and market testing software.
G - 3