Evaluation Guide for Pedestrian Safety ACKNOWLEDGMENTS
Authors
Safe States Alliance • Jamila Porter, DrPH, MPH, Director of Programs and Evaluation • Ina Robinson, MPH, Evaluation and Technical Assistance Coordinator
Research and Evaluation Group • Shenee Bryan, MPH, MPA, Director of Programs and Evaluation • LaTasha Barnwell, MPH, Research Associate
Transtria • Laura Brennan, PhD, MPH, Chief Executive Ofcer • Brandye Mazdra, MPH, Director of Projects and Operations
Advisors to the Authors
• Ross Brownson, PhD, Bernard Becker Professor of Public Health, Washington University in St. Louis • Sarah Strunk, MHA, Strategic Advisor, Active Living By Design • Ian Thomas, PhD, State and Local Program Director, America Walks • Denise Yeager, MPH, Healthcare Program Consultant and Operation Analyst, Georgia Division of Public Health, Injury Prevention Section
The Evaluation Guide for Pedestrian Safety was developed by the Safe States Alliance, with support from the National Highway Trafc Safety Administration (NHTSA). Content and structure for the guide were provided by Research and Evaluation Group and Transtria. Web development, design, and hosting of the guide were provided by Banyan Communications.
Evaluation Guide for Pedestrian Safety Accreditation | 2 TABLE OF CONTENTS
About the Evaluation Guide ...... 4 Section 4: Methodology and Data Collection ...... 59 Create Your Evaluation Plan ...... 10 Step 1: Identify data sources or design data Resource Center ...... 11 collection methods...... 59 Step 2: Create a matrix to connect your evaluation Section 1: Evaluation Purpose and Partners ...... 12 questions, indicators, and data sources ...... 64 Step 1: Convene partners to plan the evaluation ...... 12 Step 2: Identify your population of interest and your Section 5: Data Analysis ...... 66 vision for change ...... 17 Step 1: Convene data analysis partners ...... 66 Step 3: Consider the intervention stage and corresponding Step 2: Identify variables and themes ...... 67 types of evaluation ...... 18 Step 3: Organize data and ensure data quality ...... 70 Step 4: Determine evaluation audiences and the relevance Step 4: Agree on data analysis methods ...... 72 of the evaluation to these audiences ...... 21 Step 5: Interpret results ...... 75 Step 5: Create a clear, concise purpose statement ...... 23 Step 6: Create a data analysis plan ...... 78 Step 6: Design your collaborative, including a governance Section 6: Reporting and Dissemination ...... 79 structure and process, partner responsibilities, Step 1: Identify audiences and develop a dissemination plan ...... 79 and budget and time constraints ...... 25 Step 2: Select key messages for specifc purposes ...... 80 Section 2: Describing the Intervention ...... 31 Step 3: Select communication methods and create evaluation Step 1: Describe your Pedestrian Safety Intervention ...... 31 products ...... 80 Step 2: Create a Logic Model ...... 36 Step 4: Monitor Dissemination Eforts ...... 86
Section 3: Evaluation Design ...... 39 Glossary ...... 87 Step 1: Develop evaluation questions aligned with the Evaluation Templates Appendix ...... 96 evaluation purpose statement ...... 39 Tools and Resources Appendix ...... 116 Step 2: Identify the population or subpopulations, intervention goals and objectives, and types of evaluation for each evaluation question ...... 42 Step 3: Specify the evaluation design to maximize causal inferenceand internal validity for each evaluation question ...... 44 Step 4: Determine sampling strategies to maximize external validity in the selected population or subpopulations for each evaluation question ...... 53
Evaluation Guide for Pedestrian Safety Table of Contents | 3 ABOUT THE EVALUATION GUIDE
The Evaluation Guide for Pedestrian Safety is a robust and comprehensive resource designed to help you and your partners efectively evaluate your pedestrian safety interventions.
How You Can Use This Guide You can use this guide to: • Plan your evaluation of a pedestrian safety intervention using a helpful, step-by-step process. • Obtain useful tips to help you: • Collaborate successfully with partners. • Infuse principles associated with social, health, and economic equity into your evaluation. • Implement and manage your evaluation process efectively. • Capitalize on assets and overcome anticipated barriers during the evaluation process. • Access templates, tools, resources, and examples from the feld to inform and enhance your evaluation. Why Evaluation Is Essential There are a variety of reasons that it is essential to evaluate pedestrian safety interventions. Evaluation can: 1. Improve interventions by identifying what works to increase pedestrian safety. Evaluationfndingscan: • Refect strengths and weaknesses of your partnership’s structure and processes. • Strengthen the design and implementation of your pedestrian safety intervention. • Highlight the reach of your pedestrian safety intervention to vulnerable populations. • Demonstrate impacts of your pedestrian safety intervention on population health behaviors, outcomes, and quality of life. • Assess sustainability of your pedestrian safety intervention over time. 2. Build support for collaborative partnerships that can mobilize resources for pedestrian safety initiatives. Evaluation partnerships can: • Raise awareness and visibility of pedestrian safety interventions. • Leverage political and community support, funding, and resources for pedestrian safety interventions.1
Evaluation Guide for Pedestrian Safety About the Evaluation Guide | 4 3. Establish an evidence base for scale up of efective pedestrian safety policies and systems. Systematicreviewsofevaluationfndingscan: • Facilitate state, regional, and local decision-making about priority pedestrian safety interventions. • Provide state, regional, and local model practices or implementation guidelines. 4. Make the connections between pedestrian safety and population health explicit. 2,3 “Field-building”basedonevaluationfndings(andresearch)can: • Provide data support to show pedestrian safety interventions increase active travel. • Show how increases in active travel increase the proportion of the population meeting physical activity recommendations and decreases in auto use reduce sedentary behaviors. • Demonstrate that increases in physical activity and reductions in sedentary behaviors reduce morbidity and mortality (e.g., obesity, cardiovascular disease, cancer). • Draw connections between active travel/reduced auto use and cleaner air, community development, economic growth, sustainability, and other related outcomes.
“Evidence-based public health utilizes the current best available evidence to make decisions in the public health service, and also to develop action plans, public health programs, and policies for addressing public health issues.” 4
FOOTNOTES 1. Redmon, T., D. Gelinne, L. Walton, J. Miller. Jan/ Feb 2012. Spotlight on pedestrian safety. Publication Number: FHWA-HRT-12-002 75(4). Retrieved from: https://www.fhwa.dot.gov/publications/publicroads/12janfeb/03.cfm 2. U.S. Centers for Disease Control and Prevention. 2010 CDC Recommendations for Improving Health through Transportation Policy. Retrieved from: https://www.cdc.gov/transportation/docs/fnal-cdc-transportation-recommendations-4-28-2010.pdf 3. U.S. Department of Transportation. Walking and Biking are Good for Public Health. Retrieved from: https://www.transportation.gov/safer-people-safer-streets 4. Brownson, R. C., E. A. Baker, A.D. Deshpande, K. N., Gillespie. Evidence-Based Public Health. 3rd Ed. New York: Oxford University Press; 2018. Retrieved from: https://global.oup.comacademic/product/evidence-based-public-health-9780190620936?cc=us&lang=en&
Evaluation Guide for Pedestrian Safety About the Evaluation Guide | 5 What Interventions You Can Evaluate The Evaluation Guide for Pedestrian Safety is designed to evaluate pedestrian safety interventions, specifcally pedestrian safety action plans (PSAPs) and pedestrian safety education campaigns and promotions.
• Pedestrian Safety Action Plans (PSAPs) create a framework for state and local ofcials and a variety of partners to address pedestrian safety problems in specifc geographic areas to identify, implement, and evaluate optimal solutions to improve pedestrian safety.
• Pedestrian safety education campaigns and promotions are coordinated eforts designed to improve pedestrian safety for a defned population by targeting knowledge, attitudes, awareness, beliefs, behaviors, and/ or social norms related to pedestrian safety. These eforts can vary in complexity depending upon a variety of factors, such as duration, resources, and message.
Pedestrian safety interventions are frequently comprehensive. • PSAPs are implemented as part of larger city, county, or regional eforts that include a variety of diferent pedestrian safety interventions, such as engineering initiatives, educational eforts, or enforcement practices.
Pedestrian safety interventions can also be standalone. • Policy, practice, or environmental changes can be independently designed and implemented, particularly when resources and support are not available. • Educational and promotional eforts, such as social marketing campaigns, media advocacy,or training programs, can be implemented to increase awareness and generate action related to comprehensive pedestrian safety interventions.
PSAPs and education campaigns and promotions often incorporate multiple intervention strategies to improve pedestrian safety. • These intervention strategies can be categorized across nine domains, including the traditional “E’s” (engineering, education, enforcement, encouragement, equity, evaluation), as well as others that include advocacy, collaboration, and land use planning and policy. • The table below describes each domain and provides examples of common PSAP and education campaign and promotion interventions for each domain.
Evaluation Guide for Pedestrian Safety About the Evaluation Guide | 6 Nine Pedestrian Safety Domains, Descriptions, & Intervention Examples Domain Description Intervention Example
Advocacy To generate political will and community support Decision-maker engagement and support (elected ofcials, appointed ofcials, community leaders) through presenting to city/ county/ state councils/ committees, adopting principles for practice among appointed ofcials, etc.
Collaboration To partner and organize to create change Provide training and other types of technical assistance to increase the knowledge, skills, and abilities of individual community leaders or champions to engage in certain partnership and community activities
Engineering To design sites and streets, intersections, sidewalks/ Improvements to sidewalks, bike lanes or streets for trafc calming on crosswalks/signage, or timed trafc lights routes to schools, child care agencies, parks and recreation facilities, and worksites
Education To increase educational outreach to communities School districts implement Safe Routes to School program/ Walking School Bus program
Enforcement To develop active transportation and pedestrian Government permits authority to local cities and counties to pass a local safety policies and enhance local law enforcement of resolution or ordinance to support active transportation or pedestrian these policies (speed limits, yielding to pedestrians safety in crosswalks, and proper walking and bicycling behaviors) and community or site enforcement of active transportation and pedestrian safety
Encouragement To host events inspiring communities to try something Social marketing campaigns (systematic application of marketing, along new, which often results in the development of with other concepts and techniques, to achieve specifc behavioral goals ongoing programs to encourage walking and bicycling for a social good (e.g., Take A Walk, Feet First, Travel Smart)
Equity To support safe, active, and healthy opportunities Policy that supports development, funding, or maintenance of active for children and adults in low-income communities, transportation and pedestrian safety facilities and design features in communities of color, and beyond afordable housing/ housing authority locations
Evaluation To assess the need and scope for efective pedestrian Parent surveys help pinpoint reasons why parents drive their children to interventions. school instead of allowing them to walk or bicycle, and the changes that could encourage active school transportation options.
Land Use To design and develop communities (permission, Small area plan incorporating active transportation or pedestrian safety Planning & protection, and use of land) provisions (e.g. neighborhood, corridor) Policy
For additional descriptions and examples of pedestrian safety domains, see Pedestrian Safety Strategies and Domains.
Evaluation Guide for Pedestrian Safety About the Evaluation Guide | 7 Who’ll Enjoy Using This Guide A range of diferent professional and lay audiences can use this guide.5,6,7,8 The table below highlights some of the ways these audiences can use this guide.
You’ll enjoy using this Because you can contribute to an evaluation of a pedestrian safety intervention by: evaluation guide if you’re a:
Public health practitioner • Identifying or collecting primary or secondary data related to pedestrian safety behaviors, policies, practices, environments, and educational or promotional eforts • Analyzing the magnitude, trends, and patterns of pedestrian fatalities, injuries, and associated risk and protective factors • Examining trends among subpopulations to convey connections to the social determinants of health and indicators of health inequities
Transportation, highway safety, • Prioritizing pedestrian safety policies and improvements for evaluation or planning professional • Identifying existing and needed data to use for evaluation (e.g., pedestrian counts, trafc fow data, road safety audits/ assessments, crash reports). • Designing policies, practices, or environments based on evaluation fndings (e.g., urban growth boundaries, roundabouts, countdown timers)
Law enforcement professional • Identifying pedestrian safety practices ready for evaluation • Identifying existing data to use for evaluation (e.g., trafc code violations, crime rates) • Designing equitable and community-centered policies, practices, and environments based on evaluation fndings (e.g., warnings vs. citations)*
Electedorappointedofcial • Voting in formal policies, such as Vision Zero policies, that provide unambiguous direction to city bureaucrats and engineers (e.g., eliminate trafc deaths by 2030). • Serving as a champion to rally support for evidence-based pedestrian safety policies and improvement projects • Allocating funding to pedestrian safety policies and projects through budget line-items, earmarks, tax incentives, or related approaches
Administrator,teacher,orstaf • Demonstrating the value of Safe Routes to School initiatives to students, families, and communities member in a child care center, • Serving as a champion for safe environments to support active children school, or other educational setting • Linking pedestrian safety initiatives to improved physical activity and educational outcomes in children
Administrator,provider,orstaf • Connecting pedestrian safety initiatives to reductions in use of emergency services and associated health care member in a health care setting costs in communities • Engaging health care providers in educational approaches to increase safe driving practices and implications for individual and community health
Evaluation Guide for Pedestrian Safety About the Evaluation Guide | 8 You’ll enjoy using this Because you can contribute to an evaluation of a pedestrian safety intervention by: evaluation guide if you’re a:
Member or employee of a • Identifying evidence-based pedestrian safety policies, practices, environmental strategies, and educational and community group or advocacy promotional eforts organization • Increasing equitable access to safe pedestrian accommodations among vulnerable populations (e.g., lower- income communities, people with disabilities) • Educating city council representatives, other elected and appointed ofcials, and community leaders on the benefts of pedestrian safety initiatives
Media and communications • Identifying existing media (e.g., television, radio, social media) from which evaluation data can be accessed professional • Assessing reach to vulnerable subpopulations (e.g., children, elderly, people with disabilities, lower income populations) • Assessing gains in public support for pedestrian safety initiatives • Identifying education messages, audiences, and communication methods for dissemination of evaluation fndings
Community member • Communicating pedestrian safety concerns • Volunteering to serve on committees or support data collection, interpretation, or dissemination eforts • Helping to garner community support and recruiting participants for the evaluation
There are also many other potential users of this guide, given that: • Traditional designs for communities and transportation systems focused on automobiles are increasingly being replaced with “smart growth,” “new urbanist,” and “Complete Streets” approaches that balance the needs of drivers with those of pedestrians, bicyclists, and other road users (e.g., wheelchairs, scooters, strollers). • Interdisciplinary policies, practices, environmental strategies, and educational and promotional eforts to increase pedestrian safety are gaining momentum. • Technology and social media are altering the communication environment and require new approaches to education and promotion. • There is increasing recognition of the infuence of social determinants on systems and outcomes that are central to social, health, and economic equity.
FOOTNOTES 5. U.S. Department of Transportation National Highway Trafc Safety Administration. 2014 Pedestrian Safety Enforcement Operations: A How-To Guide Retrieved from: https://www.nhtsa.gov/sites/nhtsa.dot.gov/fles/812059-pedestriansafetyenforceoperahowtoguide.pdf 6. Boston Department of Transportation. 2001 Access Boston 2000-2010 Pedestrian Safety Guidelines for Residential Streets: Safety Guidelines for Residential Streets. Retrieved from: http://www.bu.edu/police/fles/2016/07/pedestrian_safety_guidelines.pdf 7. World Health Organization 2010 Pedestrian safety: A road safety manual for decision-makers and practitioners. Retrieved from: http://apps.who.int/iris/bitstream/10665/79753/1/9789241505352_eng.pdf 8. Leah Shahum. July 21, 2016. Vision Zero, Equity & Law Enforcement. Retrieved from: http://visionzeronetwork.org/vision-zero-equity-law-enforcement/.
Evaluation Guide for Pedestrian Safety About the Evaluation Guide | 9 CREATE YOUR EVALUATION PLAN
The Evaluation Guide for Pedestrian Safety is organized into six sections that correspond with the primary elements of the Centers for Disease Control and Prevention’s (CDC) Framework for Program Evaluation9:
In this area of the guide, you’ll fnd sequential, step-by-step guidance and instructions that can help you create an evaluation plan for a pedestrian safety intervention.
FOOTNOTES 9. Centers for Disease Control and Prevention (CDC), Program Performance and Evaluation Ofce (PPEO). A Framework for Program Evaluation. Retrieved from: https://www.cdc.gov/eval/framework/index.htm
Evaluation Guide for Pedestrian Safety Create Your Evaluation Plan | 10 RESOURCE CENTER
The Resource Center provides a variety of resources to support your evaluation planning process; these are referenced throughout the “Create an Evaluation Plan” section of the guide. You can navigate to each resource using the icons below.
Glossary – Visit the Glossary to look up new terms and their definitions. Throughout the guide, words in the glossary are light blue.
Tips for Success – Check out these helpful tips to enhance your evaluation planning process.
Examples from the Field – Learn about noteworthy field work in each section of the guide.
Templates – Download forms and templates that can help you create elements of your evaluation plan.
Tools & Resources – Find helpful tools and resources that can inform and inspire your evaluation efforts.
Evaluation Guide for Pedestrian Safety Resource Center | 11 SECTION 1: EVALUATION PURPOSE AND PARTNERS
Congratulations! You’ve decided to evaluate your pedestrian safety intervention.
Your evaluation will help to… improve pedestrian safety in your community or state AND contribute to a growing pedestrian safety movement nationally and internationally BY identifying what works to efectively prevent pedestrian injuries and fatalities.
Put your feet to the pavement!
This section highlights your frst steps to design your evaluation plan — convene your partners to discuss the fundamental components of your evaluation (population, intervention, audiences) in order to create your evaluation purpose statement and design your collaborative.
Step 1: Convene partners to plan the evaluation Partners provide a variety of perspectives, helpful ideas, and useful resources to ensure the success of the evaluation.
Start your journey!
Reach out to existing colleagues and make a list of their contacts and networks. AND Investigate previous pedestrian safety initiatives and evaluation eforts in your community or state.
This information can help you determine what individuals and agencies to engage as well as well as what successes or challenges emerged as part of previous related eforts.
Evaluation Guide for Pedestrian Safety Section 1: Evaluation Purpose and Partners | 12 What partners? Invite people from all walks of life to the table, including funders, decision-makers, advocates, implementers, evaluators, and members of the population of interest to discuss their respective interests and concerns related to pedestrian safety.
Identify those with wisdom and past experience working on pedestrian safety as well as those who are new to pedestrian safety, who ofer fresh perspectives on how to create positive change in your community or state.
Key partners for evaluating pedestrian safety initiatives include representatives from: • Local, regional, and state government agencies • Academic institutions • Health care organizations • Civic, community, and advocacy groups • Funding organizations • Ofces of elected and appointed ofcials and other policy- and decision-makers
How to engage partners? Identify partners’ interests and concerns related to pedestrian safety.
Varying viewpoints from diverse partners across diferent sectors and disciplines (e.g., design and planning, transportation, health, economics, and environmental science) can inform and enhance the breadth and depth of the purpose of your evaluation.
Tip for Success
Engagement of members of the population of interest may require a strong partnership commitment to hiring residents, people who grew up in the community, and/or those who have the cultural and linguistic characteristics of the individuals living in this population.
Evaluation Guide for Pedestrian Safety Section 1: Evaluation Purpose and Partners | 13 Consider the “added value” of your pedestrian safety initiative to a wide range of potential partners concerned with diferent assets or goods afecting sustainability in your community or state, such as:
Example Relationship to pedestrian safety
Environmental protection of More pedestrians and fewer drivers reduce the carbon footprint. renewable and non-renewable resources(naturalcapital)
Health promotion and education Moderate-intensity physical activity, such as walking, improves health and well-being. (humancapital) Healthy, knowledgeable, and skilled people contribute to a productive workforce.
Networks, trust, and norms of More pedestrians and fewer drivers increase opportunities for social interaction, stimulate the local economy, reciprocity(socialcapital) and enhance crime prevention.
Infrastructure and technology Pedestrian-oriented community and street design increases walking and decreases driving. (manufacturedcapital)
Wealth accumulation Walking is the simplest and least expensive mode of transportation (individual). (fnancialcapital) Pedestrian infrastructure costs much less than automobile infrastructure (community).
Source: Adapted from Forum for the Future, https://www.forumforthefuture.org/project/fve-capitals/overview.
Generate a list of these individuals or organizations and their assets.
Template
Partner Assessment and Engagement Plan template A template to guide partner engagement and assessment of assets for evaluation.
Evaluation Guide for Pedestrian Safety Section 1: Evaluation Purpose and Partners | 14 Partners’ assets can ofer insight into potential roles and responsibilities of partners as well as in-kind resources to support planning and implementation of the evaluation.
Types of Partners Examples Assets
Government agencies • Public health Pedestrian Safety Action Plans (local,regional,andstate) • Design and planning • Access to city ofcials and city council agendas • Transportation • Access to community and business leaders • Transit services • In-kind support (e.g., staf support, meeting space, data) • Highway safety • Best practices in road design, trafc safety, and law enforcement • Trafc engineering Educational and Promotional Eforts • Public works • Access to community events or calendars • Law enforcement • Access to communication/ public afairs staf • Best practices in health education and injury prevention • Existing networks and systems for community education and outreach
Academic institutions • Centers for applied science/ • Knowledge of theory, best practices, and evaluation and research methods practice across disciplines (e.g., • Ability to collect and analyze data public health, planning, law. • Connections with students or volunteer support environment)
Health care organizations • Emergency response • Data on pedestrian injuries and fatalities • Hospitals
Civic, community, and • Advocacy organizations • Authentic voice on pedestrian concerns and danger areas advocacy groups • Coalitions • Knowledge of and infuence on civic and community leaders • Business districts • Access to community listservs and grassroots outreach channels • Walk/ bike clubs • Source of volunteer support for events and outreach • Youth groups and senior citizen centers • Municipal advisory groups • Neighborhood organizations
Evaluation Guide for Pedestrian Safety Section 1: Evaluation Purpose and Partners | 15 Types of Partners Examples Assets
Funding organizations • Government (e.g., federal or • Source of funding for pedestrian safety projects as well as events or campaigns state agencies, local budget and • Links to technical assistance and resources tax incentives) • Opportunities for community development and capacity building • Non-proft (e.g., national or state associations, community development corporations) • Private (e.g., foundations, businesses)
Ofcesofelectedand • City or County Councils • High visibility “champions” appointedofcialsand • Local, state, and national-level • Formal commitments, such as city council resolutions other policy- and decision- elected and appointed ofcials makers (mayors, county executives, school district administrators)
Source: Adapted from Watch for Me – NC Bicycle and Pedestrian Safety, Education, and Enforcement Campaign: Project Summary and Evaluation Final Report. 2014. http://www.watchformenc.org/wp-content/themes/WatchForMeNC_Custom/documents/WFM_FinalReport_2014.pdf
Tip for Success
Create a brief partnership profle to share with potential new partners or to use as a tool for recruitment of new partners.
Example from the Field
New Orleans, LA partnership profle An example of a partnership profle.
Evaluation Guide for Pedestrian Safety Section 1: Evaluation Purpose and Partners | 16 Example from the Field
NC Watch 4 Me Community and Partner Profles Examples of diferent sized communities across North Carolina (e.g., 100,000+ population, < 25,000) participated in the Watch for Me NC Campaign including descriptions of partnership profles and key outcomes.
With your partners, decide whether it makes sense to conduct the evaluation yourselves or to hire an external evaluator.
Step 2: Identify your population of interest and your vision for change To ensure your evaluation plan has a clear focus and direction, defne the population of interest and the pedestrian safety vision for this population.
Defne your population of interest Thepopulationofinterestforpedestriansafetyinitiativesmaybedefnedinseveralways,including: • Geographically, such as single or multiple states, single or multiple counties, metropolitan areas, cities or municipalities, zip codes or school districts, census tracts or neighborhoods. • By institutions or organizations, such as government agencies, worksites, churches, schools. • By socially-defned or virtual groups, such as walking clubs or internet communities. • A combination of the groups above.
Consider priority subpopulations in this population as well. For instance, lower-income neighborhoods and people of color are disproportionately impacted by pedestrian fatalities and serious injuries.1, 2
Example from the Field
NC Watch 4 Me Scaling and Focusing Intervention for Target Populations Lessons learned for scaling and focusing interventions to target populations.
Evaluation Guide for Pedestrian Safety Section 1: Evaluation Purpose and Partners | 17 Revisit your partnership’s pedestrian safety vision and mission Refer to your partnership’s vision and mission to align partners’ intentions for increases in pedestrian safety in the population of interest.
Your partnership’s vision and mission may continue to evolve as new partners join or as the evaluation plan unfolds.
If you and your partners need to create a vision and mission, start by discussing how the population of interest will look diferent (e.g., improved health and quality of life, increased walking for transportation and use of public transportation, increased educational attainment and productivity) and what factors or conditions (e.g., mixed-use development, pedestrian-oriented street design, trafc calming, awareness of the benefts of walking) need to be changed in order to achieve these population changes.
To increase partnership momentum to evaluate your intervention, align your vision and mission with other national, state, or local partners and initiatives as a movement (e.g., Vision Zero, http://visionzeronetwork.org/).
Step 3: Consider the intervention stage and corresponding types of evaluation Identify your intervention stage: Planning – You and your partners are in the process of designing your intervention goals, objectives, activities, partners’ roles and responsibilities, and resources needed to carry out the intervention. Implementation – You and your partners are engaged in intervention delivery. Enforcement/maintenance – You and your partners are trying to sustain the intervention beyond the original timeline for intervention delivery.
FOOTNOTES 1. Morency P, Gauvin L, Plante C, Morency C. (2012). Neighborhood Social Inequalities in Road Trafc Injuries: The Infuence of Trafc Volume and Road Design. American Journal of Public Health 102(6):1112-9. DOI: 10.2105/AJPH.2011.300528. 2. Coughenour C, Clark S, Singh A, Claw E, Abelar J, Huebner J, et al. (2017). Examining racial bias as a potential factor in pedestrian crashes. Accident Analysis and Prevention 98:96-100. https://doi.org/10.1016/j.aap.2016.09.031.
Evaluation Guide for Pedestrian Safety Section 1: Evaluation Purpose and Partners | 18 Depending on your intervention stage, your evaluation plan may address one or more of the following four evaluation types: 1. Formative evaluation Planning stage ! Formative evaluation
Formative evaluation is frequently referred to as “community assessment”, or the examination of a community’s assets, needs, current resources, strengths, and challenges.
Asset mapping documents the resources and supports that already exist in your community.
Needs assessment identifes the changes need to occur in the community to improve health.
A comprehensive formative evaluation can address all of the following: • decisions about where to focus resources and interventions in order to maximally beneft the community; • understanding of the relationship between pedestrian safety interventions, pedestrian and driver behaviors, and health and related outcomes; • characteristics of the partnership and the population prior to intervention to identify factors for tracking or monitoring change; and • contextual factors that can infuence the intervention or evaluation (e.g., concentrated poverty or areas with a small local tax base, rural communities or areas with a small population density).
2. Process evaluation Implementation stage ! Process evaluation
Process evaluation helps you and your partners determine how well the intervention is working.
Factors such as feasibility, cost, reach, and unanticipated barriers can positively or negatively afect implementation fdelity and the population’s satisfaction with the intervention.
Process evaluation involves an assessment of how well intervention activities are carried out (e.g., policies, media, partner or community meetings), including evaluation of both the partnership and the pedestrian safety intervention.
3. Impact evaluation If intervention implementation is partially or fully complete, impact evaluation demonstrates how and to what degree intervention objectives have been attained.
Evaluation Guide for Pedestrian Safety Section 1: Evaluation Purpose and Partners | 19 4. Outcome evaluation If intervention implementation is fully complete, outcome evaluation demonstrates how and to what degree intervention goals have been attained.
Type of Evaluation Example Questions Example Types of Documentation
Formative evaluation • Will state, regional, and local departments • Interviews with appointed ofcials and staf in departments of of planning and transportation require planning and transportation developers and engineers to adhere to • Policy scans to identify existing guidelines pedestrian level-of-service (LOS) guidelines to prevent pedestrian injuries and fatalities?
Process evaluation • What resources (personnel, space, • Review of meeting minutes, time sheets, and budget or expense equipment) and costs (in-kind and reports monetary) are incurred as a result of • Implementation tracking forms or checklists developing, implementing, and enforcing pedestrian LOS guidelines for developers and engineers?
Impact evaluation • Will the pedestrian LOS guidelines • Inventory of new or modifed developments or streetscapes decrease the number of collisions involving adhering to the pedestrian LOS guidelines pedestrians in three years? (policy • Crash reports for these areas objective) • Focus groups with local residents to assess related consequences • What were some of the intended and unintended consequences?
Outcome evaluation • Will the pedestrian LOS guidelines reduce • Crash reports or crash maps over 5-10 year time frames the proportion of pedestrian injuries by • Hospital records 75% in fve years? (policy goal)
Evaluation Guide for Pedestrian Safety Section 1: Evaluation Purpose and Partners | 20 Multi-component and complex pedestrian safety interventions You and your evaluation partners may have diferent policy, practice, environmental, programmatic, or promotional components in various intervention stages, thus multiple types of evaluation may be occurring simultaneously.
“Downstream” and “upstream” intervention components You and your evaluation partners are likely to evaluate your policy, practice, environmental, programmatic, or promotional components or your “downstream” intervention strategies.
To fully understand the change process and outcomes, it is also helpful to evaluate your complementary “upstream” strategies and activities leading to these primary intervention components (e.g., community development and organizing, advocacy).
Visit Intervention Strategies for a list of potential pedestrian safety strategies.
Step 4: Determine evaluation audiences and the relevance of the evaluation to these audiences Who will use the evaluation fndings? Your audiences represent the same interest groups as your partners from Step 1, including funders, decision-makers, advocates, implementers, evaluators, and members of the population of interest.
Examples of how fndings will be used Refect on fndings from a formative evaluation to work with your partners to develop or refne intervention approaches. Identify ways you and your partners may use fndings from a process evaluation to improve the delivery of the intervention. Consider how funders and decision-makers may use the results of your impact and outcome evaluation to assess the return on their investment or allocation of funds, respectively.
How will fndings be used? Most audiences will be interested in the results of an impact and outcome evaluation to determine the “value” of the pedestrian safety intervention strategies for reducing pedestrian injuries and fatalities, as well as improving health and quality of life.
Evaluation Guide for Pedestrian Safety Section 1: Evaluation Purpose and Partners | 21 Perceived Purpose of an Partners Interests Evaluation
Evaluators and researchers To determine the efectiveness • Behavior, health outcomes of local policy, environment, and • Reliable, valid tools system changes • Evaluation design
Policymakers and To identify the changes with • What works, where, when, how and why practitioners the greatest impact, relevance, • Resources and assets feasibility, and sustainability • Costs and challenges
Advocates and community To inform local decision-making, • Local representation and participation leaders document successes, and improve • Health equity quality of life • Improved quality of life
Research and evaluation partners: Assess the efectiveness of pedestrian safety action plans and educational and promotional intervention approaches on improvements to a range of short-term, intermediate, and long-term impacts and outcomes.
Use reliable and valid quantitative tools and measures, as well as a rigorous study design and execution to ensure confdence in the fndings from the evaluation.
Policymakers and practitioners: Identify the intervention approaches with the greatest impact, relevance, feasibility, and sustainability (RE-AIM framework) by assessing what works, where it works, when it works, how it works, why it works (or why not), and how much it costs. Specifcally, policymakers often seek evidence that their constituents are being served, stories of positive examples of change, and publicity that will get them re-elected.
Incorporate mixed-methods (which combine quantitative and qualitative data) and ensure that the data is representative of the population of interest and corresponding subpopulations to ensure confdence in the fndings from the evaluation.
Evaluation Guide for Pedestrian Safety Section 1: Evaluation Purpose and Partners | 22 Advocates and community leaders: Determine the practical implications of fndings from the use of diferent intervention approaches to inform state or local decision-making, highlight state or local successes, and secure more funding to sustain the changes over time.
Track intended and unintended results, practical considerations (resources, costs), assets used, and challenges encountered using simple, efcient measures serving multiple purposes (advocacy, marketing, cost-beneft analysis) that are easily translated to the interests of local audiences (decision-makers, business owners).
These interests are not mutually exclusive, yet it can sometimes be difcult to meet the needs of all partners. Therefore, it is important to be clear about what your evaluation is or is not going to accomplish.
Revisit the list of partners’ interests and concerns related to pedestrian safety from Step 1 for additional potential uses of the evaluation fndings.
Step 5: Create a clear, concise purpose statement Initially, your evaluation purpose statement can be general, such as: “the purpose of this evaluation is to measure the impact of the pedestrian safety initiative.”
From your work in Steps 1-4…
Your purpose statement should have evolved to include the following elements: • A specifc reference to your population and subpopulations of interest; • A connection to the partnership’s vision and mission; • Relevant formative, process, impact, and/or outcome evaluation components; and • Uses of the evaluation fndings by priority audiences.
Consider the rangeoffactorsorconditionsafectingpedestriansafetyinthepopulationofinterest,yourpriorityfactorsand conditions, and “low hanging fruit”, or factors and conditions that can be more readily addressed by your partnership, including: • Partnership strengths, weaknesses, opportunities, and threats; • Organizing and advocacy activities; • Changes in policies, practices, and environments afecting pedestrian safety;
Evaluation Guide for Pedestrian Safety Section 1: Evaluation Purpose and Partners | 23 • Educational and promotional eforts increasing pedestrian safety knowledge and skills in the population of interest; and • Health behaviors, health outcomes, and related social, economic, and environmental outcomes.
To increase health equity, pay particular attention to how health disparities, inequities, and social determinants of health may afect these factors and conditions.
Tools & Resources
WHO Commission on SDOH Provides background on equity, inequity, and disparity.
Unnatural Causes documentary Provides background on equity, inequity, and disparity.
NACCHO Health Equity and Social Justice Committee Provides background on equity, inequity, and disparity.
RWJ Commission on SDOH Provides background on equity, inequity, and disparity.
Evaluation Guide for Pedestrian Safety Section 1: Evaluation Purpose and Partners | 24 Example from the Field
Federal Highway Administration, Ofce of Safety’s Pedestrian Safety Strategic Plan Example Pedestrian Safety Strategic Plan.
Work with partners to obtain buy-in for the evaluation purpose across multiple sectors and disciplines (e.g., public health, transportation, planning, law enforcement, education, academia).
Update or refne the evaluation purpose after following the steps in subsequent sections, adding new partners, or adapting the intervention.
For instance, in Section 2, specifcation of your intervention goals and objectives as well as your associated intervention strategies and activities can help to refne the evaluation purpose statement.
Step 6: Design your collaborative, including a governance structure and process, partner responsibilities, and budget and time constraints A clear governance structure and processes for collaborative engagement will support partnership cohesiveness and efectiveness throughout evaluation planning and implementation.
Tools & Resources
Governance Principles Provides guidance on developing a governance structure and process for partnerships.
Evaluation Guide for Pedestrian Safety Section 1: Evaluation Purpose and Partners | 25 Governance structure Refect on the history of your partnership, the purposes for partners’ participation, the maturity of the initiative, and the degree of formality of the governance mechanisms for interaction.
Establish a governance structure that supports shared decision-making and accountability, including: • Identifying a lead agency (fscal accountability and coordination). • Assigning a representative set of core partners to lead the partnership, such as a steering committee (shared decision-making authority). • Gathering input and feedback from an extended network of partners varying in skills, expertise, or resources. This extended network of partners can also be organized into subcommittees or workgroups to facilitate progress on multiple evaluation goals and objectives at the same time.
Considerthefollowingoptionsfororganizingyourpartnersanddeterminethebestftfor your collaborative work, including: • A hierarchical organizational structure with regular committee and subcommittee meetings involving all partners. • A management team of core partners that meets regularly and invites partners to participate as needed or available. • A “revolving” partnership that is project-oriented, permitting partners to spend time focused solely on what interests them and to participate in focused meetings with relevant partners to discuss these specifc projects or activities. • An informal collaboration with open meetings as needed among staf and/or volunteers.
Governance processes Several processes can facilitate good governance practices in your partnership, including: • Developing strong leadership and group management skills among partners. • Forming groups (e.g., committees, task forces) to distribute the workload. • Creating partnership principles or a partner orientation manual. • Document and share the partnership’s history. • Establish mechanisms for all members to voice their opinions and share their expertise. • Require partners to have consistent meeting attendance. • Increase partners’ cultural competency. • Address the needs of representatives of lower-income neighborhoods, including selecting convenient times
Evaluation Guide for Pedestrian Safety Section 1: Evaluation Purpose and Partners | 26 and places for meetings as well as providing meals, childcare, and transportation assistance to attendees. • Providing training and orientation to new partners. • Pooling resources across the partnership (e.g., staf, funding, skills). • Maintaining records on the partnership’s processes, decisions, and accomplishments. • Developing succession plans to support leadership transitions.
Theeforttodelineaterolesandresponsibilitiesisanongoingprocessthroughthelifeofthepartnershipforthefollowing primary reasons: • As you and your evaluation partners move through design and planning stages into implementation of the evaluation plan, partners’ respective roles and responsibilities will likely shift over time. • To promote shared governance, regular leadership transitions can foster greater inclusion and representativeness of partners. • Leadership, staf, or volunteer turnover is common in partnerships, thus it is helpful to have multiple partners assigned to diferent roles and responsibilities to increase institutional memory and partnership sustainability.
With these changes in mind, work with your partners to assign roles and responsibilities for evaluation planning and implementation
Template
Evaluation Partner Planning Grid Template A template to guide assignment of roles and responsibilities for evaluation.
Critical evaluation skills and capacities of partners include previous experience in research methods or evaluation (e.g., design, data collection methods, analysis and summary).
Supplemental skills and capacities may include having talented networkers, diplomatic personalities, good listening skills, passion about pedestrian safety issues, perseverance, dedication, adaptability.
Assess your partners’ evaluation skills, readiness, and capacity for evaluation as well as assets and available funds or resources to support evaluation eforts.
Identify potential needs for training or technical assistance.
Evaluation Guide for Pedestrian Safety Section 1: Evaluation Purpose and Partners | 27 Tip for Success
Ensure each organizational partner has more than one person involved in the partnership to maintain institutional memory in the event of staf turnover.
Example from the Field
FL Pedestrian and Bicycle Strategic Safety Plan - Partner Roles An example of the types of partners and their roles in a pedestrian safety evaluation.
NY DOT Steps to form evaluation team An example of the steps taken in to form a successful evaluation team.
NC Watch 4 Me Partner Input An example of how successfully integrate partners into the overall evaluation planning process.
Local champions are sometimes absent from initial planning eforts. You and your evaluation partners may consider investing time in recruiting these partners for a variety of reasons.
Evaluation Guide for Pedestrian Safety Section 1: Evaluation Purpose and Partners | 28 These individuals can be very valuable to generate enthusiasm for your work behind the scenes, in the public eye, or with populations of interest.
These individuals are visionary, charismatic, energetic, passionate, well-known and respected, well-connected with a strong network of resources, trusted by the partners and the population of interest, competent, persistent, and politically savvy or in a position of power or infuence.
Frequently, these “sparkplugs” are asked to: • engage many diferent communities and audiences; • inspire movement from vision to action; • leverage funding; • stand up to established thinking and lead advocacy eforts; • maintain and support direct and indirect participation of the community; and • connect the right people to the right opportunities.3
Example from the Field
NC Watch 4 Me Community Champion An example of how to successfully utilize community champions in an evaluation.
Once you and your evaluation partners have delegated responsibilities to diferent individuals, you are ready to begin to outline tasks and timelines for these respective duties. A template is available to assist you and your partners through this process.
Template
Gantt Chart and Timeline of Evaluation Activities Template A template to guide the development of a timeline for evaluation.
FOOTNOTES 3. Baker EA, Wilkerson R, Brennan LK. Identifying the Role of Community Partnerships in Creating Change to Support Active Living. American Journal of Preventive Medicine2012;43(5S4):S290-S299.
Evaluation Guide for Pedestrian Safety Section 1: Evaluation Purpose and Partners | 29 Tip for Success
Institute shared leadership or periodic leadership changes to bring new energy to the evaluation, overcome bureaucratic or regulatory roadblocks, or realign partners with the vision over time.
Enhance partnership sustainability by increasing consistency and follow through among leaders and staf, ofering revolving partnership membership, creating long range plans, or involving representatives from local government.
In addition, you and your evaluation partners will need to agree on a budget for the expenses associated with the evaluation activities and revisit this budget as new partners join or the evaluation plan is adapted.
Tools & Resources
Sample Budget Provides guidelines for developing a budget.
Evaluation Guide for Pedestrian Safety Section 1: Evaluation Purpose and Partners | 30 SECTION 2: DESCRIBING THE INTERVENTION
This section provides steps on how to create an intervention description and logic model to ensure partners’ have a mutual understanding of the intervention and a “map” to focus the evaluation on the intervention inputs, activities, outputs, and outcomes.
Step 1: Describe your Pedestrian Safety Intervention The description of your pedestrian safety intervention should include: • Intervention goals and objectives • Populations of interest • Risk and protective factors that will be addressed by the intervention
Intervention goals and objectives Identifying and articulating your pedestrian safety intervention’s goals and objectives early in the evaluation planning process can ensure you and your evaluation partners have a clear understanding of what you are evaluating.
Begin by having partners clarify what the intervention is to achieve in their own words to ensure multiple perspectives are captured and the evaluation plan is comprehensive. Engaging partners in this manner also ensures that evaluation fndings will be useful and applicable to partners and audiences (described in Section 6).
Consider the following questions with your evaluation partners: 1. What are the intervention goals and objectives? 2. Are these goals and objectives timely? 3. Who will be interested in the goals and objectives being achieved? 4. What evaluation resources are needed/ available to assess these goals and objectives? 5. How can the evaluation take into consideration the historical, political, and environmental contexts infuencing these goals and objectives?
Evaluation Guide for Pedestrian Safety Section 2: Describing the Intervention | 31 Ifyouandyourevaluationpartnersdecidetorefnethegoalsandobjectivesforevaluationpurposes,rememberthefollowing: Goals are: Objectives are: • Broad • Narrow • General intentions • Precise • Abstract • Concrete • Intangible • Tangible • Generally difcult to measure • Measurable • Time bound
Goals are driven by the ultimate vision of a successful intervention when all conditions are optimal. Goals describe the long-term, desired condition once the intervention is complete.
In your evaluation plan, your goals should include: 1. Your population of interest and relevant subpopulations; AND 2. Your desired intervention impact; OR 3. Your outcomes intended to be achieved.
The following are a few examples of sources of pedestrian safety goals that can be useful to you and your evaluation partners:
Sources of Description Examples Intervention Goals
Pedestrian Safety Action Evaluators commonly adapt and • Identify and prioritize candidate project corridors through a data-driven Plans(PSAPs) modify existing intervention goals process to reduce fatally and severe injury pedestrian crashes on all public outlined in the PSAP to align with roads throughout the state the evaluation efort. • Establish a governance and management structure to facilitate coordinated implementation of pedestrian safety initiatives statewide and gauge the success of the Pedestrian Safety Action Plan
Pedestrian safety education Review intervention • Mobilize the local community and increase awareness and action related to campaigns and promotions implementation plans for pedestrian safety educational campaigns and • Inform the public or policymakers about specifc pedestrian safety issues promotions. Evaluators commonly and concerns use these plans to create, modify, or adapt goals.
Evaluation Guide for Pedestrian Safety Section 2: Describing the Intervention | 32 Sources of Description Examples Intervention Goals
Literature review and Conduct a simple, or extensive, • Foster behavioral change among users of public-right-of-way to promote an evidence based strategies literature review to identify environment of mutual respect, courtesy and acceptance anticipated evidence-based • Increase the amount and quality of pedestrian space in the downtown area outcomes that can potentially be seen in your demographic.
Organization’s strategic plan Align the intervention goals • Improve transportation system infrastructure (through the implantation of with the overall strategic plan, strategic countermeasures and construction of new transportation facilities) organizational mission, or agency to optimize the safety for all users. vision. • Manage the objectives of efcient automobile travel and pedestrian safety mobility through land use strategies.
Review intervention objectives to ensure they are measurable and describe the desired outcome as well as how and when the results will be achieved. Likewise, make sure the objectives are clear and concise so the defnition of “success” is obvious to everyone involved. Objectives should be measured against pre-established and data-driven benchmarks (e.g., data from NHTSA’s Fatality Analysis Reporting System) set by your team (elaborated in Sections 4 and 5).
The objectives should be created using the SMART objective framework outlined below:
Specifc A specifc objective will identify the setting and activity the caused the desired change. Additionally, it will indicate how the change was implemented and clearly demonstrate what was done to facilitate the impact.
Measurable A measurable objective requires a quantifable activity that resulted in the desired change. It implies that baseline data is required so that result can refect the positive and/or negative impact of the proposed intervention.
Achievable An achievable objective is feasible and considers the availability of resources, the scope of the intervention, and is attainable within a bounded timeframe.
Relevant A relevant objective relates to the goals and refects program activities appropriately. The evaluation objective has an overall efect on the desired change.
Time Bound Identifes when the objective will be accomplished using a specifc, reasonable timeframe
Evaluation Guide for Pedestrian Safety Section 2: Describing the Intervention | 33 The SMART objective framework can be applied to objectives associated with process evaluation, impact evaluation, and outcome evaluation that will guide and inform the evaluation plan.
Populations of Interest The populations of interest can include pedestrian sociodemographic groups (e.g., older pedestrians 55+ years, children aged 5-9 years, those in poverty)and residents or commuters in a specifc location (e.g., street, corridor, campus, neighborhood, county). If the populations of interest are not clearly specifed in the intervention plan for each of the goals and objectives, your evaluation plan should specify the sociodemographic group(s) and location(s) of the intervention to guide the data collection methods (Section 4) and the unit of analysis (Section 5) in the evaluation plan.
Tailoring your evaluation plan to your priority population or subpopulations can increase confdence that your results are generalizable to these and other similar populations.
Below are some things to consider with respect to your priority population or subpopulations: • What are key characteristics of the population and subpopulations? • What are the rates of pedestrian injuries and fatalities in the population and subpopulations? • What factors afect pedestrian safety in your population and subpopulations and what is their independent or collective impact (good and bad)? • What related policies, practices, services, facilities, or educational and promotional eforts are currently being provided? Does this intervention complement or duplicate any of these? • Does the intervention address pedestrian safety priorities and needs identifed by the population of interest? • What are national, state, and local policy priorities for pedestrian safety and how do these correspond to your population or subpopulations?
Evaluation Guide for Pedestrian Safety Section 2: Describing the Intervention | 34 Thetablebelowprovidessomeexamplecharacteristicstoincludeinyourpopulationprofle:
Characteristics of population profile
Geography What geographic areas will the evaluation plan cover (e.g., street, neighborhood, city, county)?
Size How many people are impacted by this intervention (e.g., population density)?
Age What is the age distribution of this population?
Gender How many males versus females?
Income What is the income distribution of this population? What is the percent below the federal poverty level (FPL)?
Ethnicity and religion What is the racial and ethnic composition of this population? What are this population’s cultural norms and values?
Population trends or context What communication approaches are most appropriate for this population (e.g., social media, billboards, TV)? What is the historical context afecting your population or subpopulations? What political or economic infuences infuence your population’s access to information and resources?
Language and literacy What proportion of the population speaks English as a second language? What is the reading level or what literacy considerations need to be taken into consideration for this population?
Risk and protective factors Risk and protective factors identifed during the intervention planning phase will focus the evaluation on known safety problems and allow the evaluation partners to select an evaluation design and methods to best understand the infuences of risks and protective factors in the context of the intervention (see Sections 3 and 4).
Refer to the intervention plan to identify risk and protective factors. Typically, these are grouped into two categories: personal factors and environmental factors. These two categories take a multi-level approach to understanding what will have a positive or negative infuence on pedestrian safety in your population of interest.
Evaluation Guide for Pedestrian Safety Section 2: Describing the Intervention | 35 Tabled below are examples of common personal and environmental risk and protective factors applicable to pedestrian safety intervention evaluations:
Categories Types Examples of Risk and Protective Factors
Personal Knowledge and skills • Understanding of pedestrian safety laws near school zones
Experience and history • Drug/alcohol use by motorists and pedestrians • Literacy rates of program participants
Support and services • Roadway nighttime lighting, lack of signage and walkways
Environmental Access, barriers, and opportunities • Land use (commercial complexes located immediately across from residential areas with high-volume multi-lane roads separating them) • Designs (multiple turn lanes and wide vehicle turning radii) • Wide roads (four or more lanes)
Policies and living conditions • Enforcement of speed limits in high motor vehicle volume areas
Tip for Success
To read more about risk and protective factors, read “Section 2. Understanding Risk and Protective Factors: Their Use in Selecting Potential Targets and Promising Strategies for Intervention”
Step 2: Create a Logic Model A logic model is often used in an evaluation to visually depict the “theory of change” behind how an intervention is intended to infuence outcomes. Generally, a logic model is read by column from left to right and it includes directional arrows and lines to illustrate causal relationships between the intervention inputs, activities, and outputs and the short, intermediate, and long-term outcomes. Although the logic model is frequently a linear concept map, it can be complex in terms of the number of relationships among inputs, activities, and outputs and the resulting outcomes as well as the potential feedback and infuence of the outcomes on the inputs, activities, and outputs.
Evaluation Guide for Pedestrian Safety Section 2: Describing the Intervention | 36 Logic models should refect process evaluation, impact evaluation, and outcome evaluation elements. The process component of the model will describe what is needed in the planning phase of the intervention, while the outcome elements in a logic model demonstrate the intended efect or goal with respect to a given time period.
Other elements of the logic model include ‘assumptions’ and ‘external environment’
Here is an example of a simplifed logic model that illustrates the theory of how specifc pedestrian safety interventions leads to intended short-term and long-term outcomes. The goal of the simplifed logic model is to communicate with other stakeholders and partners the intent of the interventions and create the framework for the evaluation.
Activities Short-Term Outcomes Long-Term Outcomes
Educational Campaigns and Promotions / Encouragement • Increase knowledge and awareness of • Reduced pedestrian crash rate • Implement walking program or club to support pedestrian safety pedestrian safety programs • Reduced intersection pedestrian • Implement wellness program that educates attendees about the • Improved pedestrian behaviors crash rate (per crossings) safe walking • Reduced pedestrian injuries • Reduced pedestrian deaths • Reduced percent of pedestrian fatalities • Improved livable, walkable communities
Enforcement • Change in motorist cited for not yielding • Issue citations for both the driver and pedestrian at crosswalks to pedestrians • Provide High-Visibility Enforcement (HVE) on driver compliance • Change in motorist cited for speeding with pedestrian right-of-way laws in pedestrian/school zones
Engineering • Decrease in vehicle volume • Create longer delays for pedestrians at intersections • Reduction in the number of travel lanes • Create pedestrian facilities (sidewalks, walkways, adequate lighting) • Improved pedestrian crossing behaviors • Constructing medians to control turning movements. • Improved in motorist driving behaviors • Install pedestrian signals (i.e., walk/don’t walk messages, • Improved visibility of pedestrians and symbolic hand/walking man messages) motorists • Decrease in average vehicle speeds
Evaluation Guide for Pedestrian Safety Section 2: Describing the Intervention | 37 Activities Short-Term Outcomes Long-Term Outcomes
Land Use Design, Planning, & Policy • Change in walking distance • Reduced pedestrian crash rate • Convert auto-oriented strip development into more accessible • Increase in number of route choices • Reduced intersection pedestrian land use patterns more suitable for pedestrians • Decrease in the number of wide cross crash rate (per crossings) • Improve existing local street connectivity and circulation by streets • Reduced pedestrian injuries adding sidewalks, paths, stairs/ramps, gates, etc. to link dead- • Reduced pedestrian deaths end streets and cul-de-sacs • Reduced percent of pedestrian fatalities • Improved livable, walkable communities
Advocacy • Increase knowledge of pedestrian safety • Decision-maker engagement and support (elected ofcials, issues and concern among elected/ appointed ofcials, community leaders) through presenting to appointed ofcials and community city/ county/ state councils/ committees, adopting principles for leaders practice among appointed ofcials, etc.
Collaboration • Increase knowledge, skills, and abilities • Provide training and other types of technical assistance to of individuals implementing pedestrian partners to engage in partnership and community activities safety programs
Equity • Decrease pedestrian crash rates in low • Support policies for the development, funding, or maintenance income geographical areas of active transportation and pedestrian safety facilities and design features in afordable housing/ housing authority locations
Evaluation Guide for Pedestrian Safety Section 2: Describing the Intervention | 38 SECTION 3: EVALUATION DESIGN
This section provides steps to develop evaluation questions in order to specify your evaluation design. It is important to work closely with your evaluation partners during the design phase to ensure there is buy-in for the approach that you choose. You should refer to your evaluation purpose statement (see Section 1) as you develop your evaluation questions.
Step 1: Develop evaluation questions aligned with the evaluation purpose statement With your evaluation partners convened,
A. Brainstorm a list of evaluation questions aligned with your partnership’s evaluation purpose statement. Several approaches can be used to structure a brainstorming session with your partners, including more formal (nominal group technique, Delphi method) or informal methods.
In the idea generation phase, encourage your partners to consider the following categories of evaluation topics: • Partnership strengths, weaknesses, opportunities, and threats; • Organizing and advocacy activities; • Changes in policies, practices, and environments afecting pedestrian safety; • Educational and promotional eforts increasing pedestrian safety knowledge and skills in the population of interest; and • Health behaviors, health outcomes, and related social, economic, and environmental outcomes.
You can also refer partners to the following resources generated in Sections I and II to stimulate ideas, including: • Evaluation purpose statement; • Population and subpopulations of interest; • Partnership’s vision and mission; • Intervention goals and objectives; • Partnership’s logic model; and • Evaluation audiences and relevant of the evaluation fndings to these audiences.
Evaluation Guide for Pedestrian Safety Section 3: Evaluation Design | 39 Tip for Success
Make sure partner concerns are included to increase applicability of the evaluation results to pedestrian safety improvements.
Example from the Field
Sample Evaluation Questions Sample process and outcome evaluation questions for PSAPs or education and promotion campaigns.
Sample Process and Outcome Evaluation Questions for PSAPs or Education and Promotion Campaigns4
Process Questions Outcome Questions (logicmodelinput,activities,outputs) (logicmodelshort-,intermediate,long-termoutcomes)
• What intervention components were adopted, implemented, or • Did pedestrian safety training signifcantly improve knowledge and skills enforced fully, partially, or not at all? Why? among participants (e.g., law enforcement ofcers, trafc engineers, city • Were participants satisfed with the intervention components? planners)? Which ones? Why? • Did outreach to city council members (e.g., data on pedestrian fatalities, • Was there variation in implementation across subpopulations? citizens’ testimonies on walkability) signifcantly infuence their support of Which subpopulations? local ordinances and regulations to improve pedestrian safety? • What were assets or facilitators of intervention delivery? How can • Did the Safe Routes to School education campaign signifcantly increase assets or facilitators be harnessed? awareness of safe walking routes among students and safe driving practices • What were barriers or challenges to intervention delivery? How can among parents? barriers be overcome? • Did trafc calming devices signifcantly reduce trafc speeds? • Who engaged, dropped out, or did not engage? Why? • Did the worksite “distracted driving” campaign signifcantly increase safe driving behaviors among employees? • Did the pedestrian safety initiative signifcantly reduce pedestrian injuries and fatalities?
FOOTNOTES 4. Adapted from: HM Treasury 2011. The Magenta Book: Guidance for evaluation. Retrieved from: https://www.gov.uk/government/uploads/system/uploads/attachment_ data/fle/220542/magenta_book_combined.pdf
Evaluation Guide for Pedestrian Safety Section 3: Evaluation Design | 40 B. Prioritize the list of evaluation questions using criteria agreed upon by partners. In the prioritization phase, the following questions may stimulate your partnership’s selection of criteria for prioritization: • Are the evaluation questions important to all partners or subsets of partners? • Are the evaluation questions relevant to the audiences for the evaluation? • Are the evaluation questions aligned with the evaluation purpose statement and the intervention goals, objectives, and logic model? • Are the evaluation questions feasible to address within the current timeline, funding, and resources? • Will the evaluation questions inform either actionable improvements to the partnership or the intervention or evidence-based practices for the feld?
As partners go through the prioritization process, be sure to track and explain the reasons why evaluation questions may be combined, refned, or discarded in relationship to the evaluation criteria. It is critical to share this rationale with partners to ensure all individuals feel their input and ongoing participation is needed and valued, particularly for those with little to no evaluation experience.
Review your fnal list of evaluation questions with all partners. Check that each question is specifc, measurable, and achievable and that the partners responsible and the timeline are assigned, consistent with the SMART guidelines described in Section 2.
Template
Worksheet to Prioritize Evaluation Questions template A template for rating the relevance of each evaluation question and making decisions about including vs. discarding questions based on specifc criteria.5
FOOTNOTES 5. Adapted from: Salabarría-Peña, Y, Apt, B.S., Walsh, C.M. Practical Use of Program Evaluation among Sexually Transmitted Disease (STD) Programs, Atlanta (GA): Centers for Disease Control and Prevention; 2007. Retrieved from: https://www.cdc.gov/std/program/pupestd/Step3_0215.pdf
Evaluation Guide for Pedestrian Safety Section 3: Evaluation Design | 41 Step 2: Identify the population or subpopulations, intervention goals and objectives, and types of evaluation for each evaluation question Now that you and your evaluation partners have identifed and prioritized the evaluation questions, it is time to identify the population or subpopulations, intervention goals and objectives, and types of evaluation corresponding to each question.
A simple table can be used to link all of these elements as follows:
Evaluation question Population or subpopulations Intervention goals and objectives Types of evaluation
From Section 2, Step 1 Updated from Section 1 From Section 2 Updated from Section 1
Your partnership can draw on the initial populations and subpopulations and types of evaluation identifed in Section 1, including formative, process, impact, and outcome evaluation, as well as the intervention goals and objectives from Section 2.
You and your evaluation partners will need to identify whether the evaluation questions are aimed at assessment (pre-intervention, formative evaluation), implementation (intervention delivery, process evaluation), or outcomes (during or immediately following the intervention, impact, or outcome evaluation), or some combination of these. Again, impact and outcome evaluation correspond to the outcomes identifed in your intervention objectives and goals, respectively.
Crafting evaluation questions that bring together formative and impact or outcome types of evaluation set the stage for evaluation designs with both baseline and follow-up measures. Combining these methods will increase confdence that changes in the outcomes being measured are, to some degree, due to the intervention strategies being evaluated.
Attribution Contribution
Caused the observed outcomes Helped to cause the observed outcomes • Are the outcomes of interest attributable to the intervention? • Is the intervention contributing to the outcomes of interest? • Are the outcomes of interest changing as a result of the • Are the outcomes of interest changing? intervention? • Is there evidence that the intervention helped achieve (or was part of what • Did the intervention cause the outcomes of interest? caused) the outcomes of interest?
Evaluation Guide for Pedestrian Safety Section 3: Evaluation Design | 42 When you evaluate pedestrian safety interventions, such as educational and promotional campaigns and PSAPs, it is challenging to determine if the intervention components caused the change in the outcomes being measured. This is due to a number of factors, including multiple complementary intervention strategies, longer intervention duration, and fuctuating environmental, social, and cultural factors afecting pedestrian safety that can be challenging to account for in the evaluation. While you and your evaluation partners are designing an evaluation plan to demonstrate the changes in outcomes are fully attributed to your pedestrian safety intervention, it is more likely that the intervention components contributed to the changes in outcomes. See fgure6 highlighting diferences between attribution and contribution in evaluation.
Consider the following example: Sample Evaluation Population or Intervention Goals Types of Evaluation Question Subpopulations and Objectives
Did outreach to city City council Goal: Ormative Evaluation: council members (e.g., members To develop and enforce • Conduct a survey of city council members’ knowledge and awareness of rates of pedestrian data on pedestrian local policies in the next injuries and fatalities in their jurisdiction and potential policies to improve pedestrian safety. • Implement a policy scan to determine existing pedestrian safety ordinances and regulations fatalities, citizens’ 12 months to reduce and interview staf from local planning, transportation, and law enforcement agencies to testimonies on pedestrian injuries and assess implementation and enforcement of these policies. walkability) signifcantly fatalities in the next 3 • Identify baseline rates of pedestrian injuries and fatalities prior to intervention infuence their support years. implementation. of local ordinances and Objective: Process Evaluation: regulations to improve • Assess whether all city council members were present to receive data on pedestrian fatalities To increase city council pedestrian safety? and citizens’ testimonies on walkability. members’ knowledge • Assess whether competing priorities or other barriers interfered with city council members’ and awareness of opportunity to adopt new or modifed policies. escalating rates of • Assess whether staf from local planning, transportation, and law enforcement agencies had pedestrian injuries and the knowledge, skills, and resources to implement and enforce the new or modifed policies. fatalities and associated Impact Evaluation: ordinances and • Re-administer the survey of city council members’ knowledge and awareness and assess regulations to improve changes after 6 months of intervention implementation. pedestrian safety in the Outcome Evaluation: next 6 months. • Assess the number of new or modifed pedestrian safety ordinances and regulations adopted by city council and implemented and enforced by local planning, transportation, and law enforcement agencies after 12 months of intervention implementation. • Assess changes in rates of pedestrian injuries and fatalities after 3 years of intervention implementation.
FOOTNOTES 6. A. Almquist (2011). CDC Cofee Break: Attribution versus Contribution. National Center for Chronic Disease Prevention and Health Promotion, Division for Heart Disease and Stroke Prevention, Evaluation and Program Efectiveness Team. Retrieved from: https://www.cdc.gov/dhdsp/pubs/docs/april_2011_cb.pdf
Evaluation Guide for Pedestrian Safety Section 3: Evaluation Design | 43 Alignment of process and impact/ outcome evaluation questions creates the opportunity to examine exposure to the intervention and/or dose of the intervention hand-in-hand with the changes observed, increasing the ability to make causal inferences about the intervention’s infuence on the outcomes. In other words, the process evaluation data can provide useful descriptive data about how the intervention succeeded or failed to afect the outcomes observed.
Step 3: Specify the evaluation design to maximize causal inferenceand internal validity for each evaluation question The next step is to create your evaluation design(s) with respect to one or more of your evaluation questions and the associated types of evaluation. As noted in the previous step, you want to maximize your ability to say that the pedestrian safety intervention caused the changes in outcomes observed. Although many pedestrian safety evaluations will not be able to show full causality, or attribution, your evaluation can show that the combination of intervention activities contributed to the observed outcome of interest. Causal inferences increase confdence in the internal validity of the relationship of the intervention to the outcomes observed. Rigorous evaluation designs help to minimize threats to internal validity.
A. Select your evaluation design There are three basic categories of evaluation design: experimental, quasi-experimental, and non-experimental. The key characteristics of each design are shown in the following table:
Experimental Designs Quasi-experimental Designs Non-experimental Designs
Attribution Attribution or Contribution Contribution
• Randomization • No randomization • No randomization • Includes a control group • Includes a comparison group • Does not include a comparison group • Helps establish cause and efect • Individuals in comparison and intervention • Most feasible/ practical to implement • Most rigorous type of design groups should have similar characteristics • Does not establish cause and efect other than intervention exposure
Evaluation Guide for Pedestrian Safety Section 3: Evaluation Design | 44 Tools & Resources
Evaluation Design Illustrations This table shows descriptions, examples, and graphic depictions of each evaluation design along with causality and strength of attribution.
Evaluation Design Illustrations7 Evaluation Design Causal Evidence Description Design Illustration (examples)
Experimental Individuals from the evaluation (randomized controlled population are randomly trial or RCT) assigned to experimental (intervention) or control/ comparison groups for group equivalence and data is collected from both groups before and after the intervention.
Quasi-experimental Individuals in the experimental (e.g., non-randomized (intervention) group are trial, before-and-after compared to individuals from study, time series) another, similar “comparison group” or data is collected from individuals in the experimental group before and after the intervention. Baseline observations are the “comparison group.”
FOOTNOTES 7. Adapted from: Measurement, Learning & Evaluation (MLE) Project (evaluation component of the Urban Reproductive Health Initiative). Retrieved from: https://www.urbanreproductivehealth.org/toolkits/measuring-success/types-evaluation-designs
Evaluation Guide for Pedestrian Safety Section 3: Evaluation Design | 45 Evaluation Design Causal Evidence Description Design Illustration (examples)
Non-experimental, Only the experimental observation, or (intervention) group is evaluated descriptive and data is collected either (e.g., post-test only, before or after the intervention. case study)
Non-experimental Observe the intervention group Time Series Design multiple times before and after (e.g., longitudinal study) the intervention and analyze trends before and after.
B. Minimize threats to internal validity Although you’ve learned about the many types of evaluation designs you and your evaluation partners can use, you’re probably wondering how to determine the evaluation design you should use.
The short answer: Choose the design that can best provide the data you need to: 1. Answer your evaluation questions; 2. Complete your evaluation within the time allotted; and 3. Utilize the resources at your disposal.
Evaluation Guide for Pedestrian Safety Section 3: Evaluation Design | 46 However, there are likely several designs that could provide these data, and you want to choose the design that will maximize your causal inference.
Therefore, your next step is to consider the strengths and weaknesses of each evaluation design in minimizing threats to internal validity. Your fnal decision will need to strike a balance between getting the best possible data and making the most efcient use of available resources within your timeline.
There are three basic conditions necessary to establish causality, including: • A relationship must be established between the intervention and the outcomes. • The intervention must precede the outcomes observed. • The outcomes cannot be attributed to other explanations.
With these conditions in mind, there are seven basic threats to internal validity and their relationship to evaluation designs:
Threat to internal validity Relationship to evaluation design
History: An unanticipated event occurring during intervention Evaluation designs with more than one group (e.g., control group, comparison implementation that is not measured or accounted for in the group) minimize the history threat to internal validity because the exposure to analysis. unanticipated events is relevant to both groups and diferences between the groups can be attributed to the intervention.
Maturation: The outcomes observed may be attributed to normal Evaluation designs with more than one group (e.g., control group, comparison developmental processes in the population or subpopulations of group) minimize the maturation threat to internal validity because the interest as a function of time. developmental processes are relevant to both groups and diferences between the groups can be attributed to the intervention.
Statistical regression: A tendency of people who have extreme Evaluation designs with more than one group (e.g., control group, comparison scores on measures (e.g., surveys, direct observation) to regress to group) minimize the statistical regression threat to internal validity because the mean on subsequent administration of those measures. the tendency to regress to the mean is relevant to both groups and diferences between the groups can be attributed to the intervention.
Selection: The population or subpopulation members assigned to Evaluation designs with only one group (i.e., intervention group) or those intervention and control or comparison groups should be equivalent with random assignment of individuals to group (i.e., intervention group and before the intervention is implemented. control or comparison group) eliminate or minimize the selection threat to internal validity because single group or the random assignment to group eliminates the second group or limits diferences between the groups.
Evaluation Guide for Pedestrian Safety Section 3: Evaluation Design | 47 Threat to internal validity Relationship to evaluation design
Experimental mortality: The diferential loss of population or Evaluation designs with random assignment of individuals to group subpopulation members assigned to intervention versus control or (i.e., intervention group and control or comparison group) minimize the comparison groups. experimental mortality threat to internal validity because random assignment increases the equivalence of groups, thus diferences between the groups can be attributed to the intervention.
Testing: The baseline assessment (pre-intervention) may sensitize Evaluation designs with more than one group (e.g., control group, comparison population or subpopulation members in unanticipated ways and group) minimize the testing threat to internal validity because exposure to the afect their performance on the follow-up assessment(s). baseline assessment is relevant to both groups and diferences between the groups can be attributed to the intervention.
Instrumentation: The measures changed from the baseline Evaluation designs with more than one group (e.g., control group, comparison assessment to the follow-up assessment. group) minimize the instrumentation threat to internal validity because the changes in measures are relevant to both groups and diferences between the groups can be attributed to the intervention.
Tools & Resources
Evaluation Design: Strengths and Weaknesses This table shows strengths and weaknesses of each design, along with a description and example pedestrian safety intervention evaluations for each design.
Evaluation Guide for Pedestrian Safety Section 3: Evaluation Design | 48 Evaluation Design: Strengths and Weaknesses8
Design (examples) Strengths Weaknesses
Experimental • Can be used when individual • Requires recruiting more participants for detecting signifcant change (during data Cluster randomization is inappropriate or analysis) compared to individual RCTs Randomized impossible • Requires complex data analysis methods Controlled Trial • Can evaluate in real-world setting
Description: An experimental cluster RCT design is when study participants are grouped (clustered) on the basis of some characteristic (e.g., city, zip code, age, travel mode) and then randomized as a group to either the intervention or control group.
Example: The Advocacy for Pedestrian Safety Study was a mixed-methods, multi-center cluster randomized controlled trial that evaluated uptake of efective pedestrian safety interventions in high risk communities. This evaluation measured efects and factors related to the success or failure of the advocacy initiative aimed at improving pedestrian safety in high risk communities in the UK. The intervention was directed at local politicians who represented electoral wards housed within local authorities. Local authorities were randomized to intervention or control groups, stratifed by study center area and local authority size.9
Quasi- • Can easily be used with surveys • Can show only short-term changes experimental • Can account for the possibility that • Requires a control group that’s similar to the intervention group Pre-Post with some other factor occurred at the Control Group same time as the intervention
Description: The quasi-experimental pre-post with control group design consists of both the intervention and control group receiving pre-test and post-test. The comparison group does not receive the intervention in between the two tests.
Example: To evaluate a comprehensive pedestrian safety countermeasure program in Miami-Dade County, Florida, consisting of 16 education, enforcement, and engineering treatments deployed at four high-collision zones, pre- (1998-2001) and post-treatment (2002-2004) data were collected. Data was collected to compare local collision trend in neighboring counties for comparison.10
FOOTNOTES 8. Adapted from: University of Albany Center for Problem-Oriented Policing. Assessing Responses to Problems: An Introductory Guide for Police Problem-Solver. Appendix D: Summary of Evaluation Designs’ Strengths and Weaknesses. Retrieved from: http://www.popcenter.org/tools/assessing_responses/7 9. Lyons RA, Kendrick D, Towner EM, Coupland C, Hayes M, Christie N, et al. The advocacy for pedestrian safety study: cluster randomised trial evaluating a political advocacy approach to reduce pedestrian injuries in deprived communities. PLoS One. 2013;8(4):e60158. Available at: http://journals.plos.org/plosone/article?id=10.1371/journal. pone.0060158 10. Zegeer, C., D Henderson, R Blomberg, L Marchetti, S Masten, Y Fan, L Sandt, A Brown, J Stutts, and L Thomas. Evaluation of the Miami-Dade Pedestrian Safety Demonstration Project. National Highway Trafc Safety Administration. Washington, DC, 2008.
Evaluation Guide for Pedestrian Safety Section 3: Evaluation Design | 49 Design (examples) Strengths Weaknesses
Quasi- • Usually convenient and lower-cost • Can show only short-term changes experimental • Is simple and quick to implement • Cannot account for pre-existing trends Pre-Post (without • Can easily be used with surveys • Cannot account for the possibility that some other factor occurred at the same time a control group) • Can provide a reasonable estimate as the intervention (least rigorous in showing causal link between intervention and of the post-intervention change outcomes) • Is weak at ruling out other alternative explanations
Description: The quasi-experimental pre-post without a control group design consists of one intervention group that receives a pre-test followed by and intervention and then a post-test.
Example: A quasi-experimental pre-post design was used to evaluate pedestrian and driver behavior at crosswalks in diferent types of locations and communities to determine whether efectiveness of Yield-to-Pedestrian Channelizing Devices was impacted by community type (urban, suburban, small city, college town); location type (intersection or mid-block); or intersection trafc control (all-way stop control, or partial stop control). Similar analyses were conducted for other sites in the same communities to examine the presence of spillover efects.11
Quasi- • Is simple and straightforward • Can use when baseline equivalency is already established or assumed experimental • Can use with one group (no • Can be a low-cost way to assess diferences between multiple versions of Post-only comparison group) or two groups interventions (with a comparison group) of participants
Description: The post-only design is one of the simplest designs. It consists of one or more groups completing a post-test after the intervention has been implemented.
Example: To evaluate a new online version of an existing pedestrian safety education intervention, a quasi-experimental post-only design can be used to compare the efects of each version on mean knowledge scores from an online survey administered immediately after the intervention.
FOOTNOTES 11. Pennsylvania Department of Transportation Bureau of Planning and Research. C. Strong, M. Kumar. Western Transportation Institute College of Engineering Montana State University. Safety Evaluation of Yield-to-Pedestrian Channelizing Devices: Final Report. 2006 Retrieved from: https://www.dot.state.pa.us/public/pdf/YTPCDFinalReport.pdf
Evaluation Guide for Pedestrian Safety Section 3: Evaluation Design | 50 Design (examples) Strengths Weaknesses
Quasi- • Is easy to use with data routinely • Is hard to use if special data collection methods, (e.g., surveys), are used to measure experimental collected over many time periods the problem Interrupted Time • Can rule out pre-existing trends and • Cannot account for the possibility that some other factor occurred at the same time Series alternative explanations as the Intervention • Takes a long time to establish results • Is hard to interpret when there are few events per time period before the intervention
Description: The simplest type of time series designs is the interrupted time series. This design is often used to evaluate the impact of a population- wide policy or intervention. It involves a single treatment group which is measured many times before and after the start of the intervention. It is called “interrupted” time series because the researcher graphs the data before and after the intervention, and looks for an interruption in the line or curve where the intervention was introduced.12
Example: A quasi-experimental interrupted time-series design was used to evaluate change in crash frequency across varying degrees of pedestrian countdown timer treatment density during 120 months of a large study in Detroit, Michigan. This evaluation design allowed evaluators to assess whether change in crash frequency depended upon of the degree to which the countdown timers penetrated the treatment unit. BE.13
Non-experimental • Is easy to use with data routinely • Is difcult to use if data collection methods (e.g., surveys) change over time Multiple Time collected over many time periods • Often needs qualitative data to explain Series • Can rule out preexisting trends and • Takes a long time to establish results many other alternative explanations • Can be hard to interpret when there are few problem events per time-period • Results easily presented (graphs) before the intervention and interpreted (patterns) • Can forecast short-term trends
Description: Multiple time series designs (or multiple baseline design) incorporates a baseline and an intervention condition across multiple participants, behaviors, or contexts. The greater the number of replications, the more confdent one can be that the treatment produced the observed changes.
Example: A non-experimental multiple time series design can be used to evaluate trends in pedestrian injury rates in intervention areas before and after a community intervention. Data can be collected retrospectively for a fve-year period, 2.5 years before intervention, and 2.5 years after the intervention, from state database records. Comparison of monthly pedestrian injury rates before and after the intervention can show any changes in injury trends.
FOOTNOTES 12. O’Connor, E., J. Bellamy, B. Spring. Evidence-Based Behavioral Practice Online Training Course. Critical Appraisal: Time Series Designs Retrieved from: http://ebbp.org/course_outlines/critical_appraisal/#C 13. Huitema, RV Houten, H Manal. Time-series intervention analysis of pedestrian countdown timer efects. Accident Analysis & Prevention 2014. 72:23–31. Retrieved from: http://europepmc.org/abstract/med/25003967
Evaluation Guide for Pedestrian Safety Section 3: Evaluation Design | 51 Design (examples) Strengths Weaknesses
Non-experimental • Can provide a detailed description of problem or • Takes a long time Case Study initiative, operations and contextual information, such • The amount of data can become unwieldy as the historical perspective • Need skills in multiple data collection techniques • Can be used to integrate data from multiple • Insiders can be biased sources (e.g., documents, interviews, participation, • Outsiders can be naïve observation, videos) • Can be used to report various characteristics (e.g., participation rates, representativeness, partnership infuence, how resources are leveraged) • Can be both quantitative and qualitative data that details facilitators, barriers, and gaps
Description: A non-experimental case study is an in-depth study of a particular person, group, program, policy, problem, decision, or organization over a period of time in order to provide a detailed description.
Application: Case studies are appropriate for assessing changes in public health capacity in sub-population groups. Case studies are especially applicable when the intervention is unique, when an existing intervention is used in a new setting, when a unique outcome is being assessed, or when an environment is unpredictable. Case studies can also allow for an exploration of community characteristics and how these may infuence intervention implementation, as well as identifying barriers to and facilitators of change. Examples of case studies in pedestrian safety.14
FOOTNOTES 14. Compiled from Federal Highway Administration Ofce of Natural and Human Environment 2005. Pedestrian and Bicycle Data Collection in United States Communities: Quantifying Use, Surveying Users, and Documenting Facility Extent. Retrieved from: http://www.pedbikeinfo.org/pdf/PlanDesign_Tools_FHWACaseStudies.pdf
Evaluation Guide for Pedestrian Safety Section 3: Evaluation Design | 52 Step 4: Determine sampling strategies to maximize external validity in the selected population or subpopulations for each evaluation question To complement your evaluation design, you and your evaluation partners should determine your sampling plan. A sampling plan can help maximize the applicability of your evaluation fndings to similar populations and subpopulations. The applicability of evaluation fndings is commonly referred to as external validity and it can apply to people, places, and time periods.
A. Identify your sampling methods In essence, your evaluation questions guide the selection of a sample of participants or data that represents your population or subpopulations of interest to ensure that the evaluation fndings can be generalized to similar populations or subpopulations.
There are two main types of sampling methods: • Probability, or random assignment of population or subpopulation members to intervention and control or comparison groups • Non-probability, or non-random assignment
This graphic15 illustrates how the diferent types of probability and non-probability sampling methods relate to one another and the descriptions as well as relative strengths and weaknesses are described in the following table.
FOOTNOTES 15. J. Dudovskiy 2011 Research-Methodology Website. Retrieved from: http://research-methodology.net/wp-content/uploads/2012/04/Sampling-in-primary-data-collection1.png
Evaluation Guide for Pedestrian Safety Section 3: Evaluation Design | 53 Tools & Resources
Sampling Methods: Comparing Strengths and Weaknesses This table shows descriptions of sampling methods along with strengths, weaknesses, and examples.16, 17
Sampling Method Description Strengths Weaknesses
Probability Sampling Random selection of individuals • Representative sample of population • Time consuming that provides an equal chance of • Generalizable to population • Expensive being selected.
Technique Description Example Why Choose?
Simple random Ensures that each participant has Use a basic lottery process an online When an unbiased sample that is an equal likelihood or probability random number generator to assign representative of the population is of assignment to experimental participants to intervention or control needed. (intervention) or control/ group. comparison groups
Systematic random A consistent rule is used to assign Every other participant who completes When a fairly representative sample is participants to experimental a baseline survey is assigned to available. (intervention) or control/ intervention vs. group. comparison groups
Stratifed random Ensures that diferent subgroups Divide population into meaningful When a larger unbiased sample that of participants have an equal groups based on certain criteria (e.g., is representative of the population is likelihood or probability of age, zip code) and then select a random needed. assignment to experimental sample from each group. (intervention) or control/ comparison groups
FOOTNOTES 16. http://study.com/academy/lesson/cluster-random-samples-defnition-selection-examples.html 17. J. Dudovskiy 2011 Research-Methodology Website. Retrieved from: http://research-methodology.net/sampling-in-primary-data-collection/multi-stage-sampling/
Evaluation Guide for Pedestrian Safety Section 3: Evaluation Design | 54 Technique Description Example Why Choose?
Random cluster Sample is made up of diferent Advocacy groups are divided into When you don’t have access to the groups within a population. This clusters based on non-overlapping entire population, sample clusters are difers from stratifed because primary advocacy topic each supports. geographically convenient, or clusters are you use the entire group, not All members of each group are included natural divisions. just select individuals within the in data collection. group. Clusters must be mutually exclusive.
Multi-stage Sample is determined in two or Divide communities into high, med, and When you need to include a higher more steps: randomly choosing low SES and then randomly choosing number of clusters. clusters and then randomly communities within each cluster. choosing participants from each cluster.
Sampling Method Description Strengths Weaknesses
Non-probability Some participants may have a • Less resource intensive • High probability of bias in sample Sampling greater chance of being selected • Les rigid • Lower generalizability • Variety of sampling techniques
Technique Description Example Why Choose?
Quota Participants are selected based on Participants from each community When probability sampling is not possible specifc criteria were recruited to match population and consideration for population characteristics (e.g., age, sex, race, proportions is needed. SES) of communities in which the countermeasures were implemented.
Evaluation Guide for Pedestrian Safety Section 3: Evaluation Design | 55 Technique Description Example Why Choose?
Purposive Sample cases are chosen To determine the travel patterns of May be the only appropriate method deliberatively to represent pedestrians and bicyclists between available if there is a limited number of characteristics known or suspected the ages of 6 -18 years in three primary data sources to be of key relevance to the communities. Rather than applying evaluation questions. random sampling and choosing cases • Extreme case – that may not be available, you can use sample unusual cases purposive sampling to ask community • Heterogeneous – partners in each of the communities sample a wide variety of cases for a list of parent/ child pairs that are • Homogeneous – interested in participating. Depending sample similar cases on evaluation goals, a specifc type of • Critical case – purposive sampling should be applied. sample important cases • Typical case – sample average cases • Theoretical – sample based on theory that is developed during sampling
Volunteer Participants self-select • Snowball – Participants are asked to When there is a need for a quick and refer others they know to also take inexpensive sample part in the evaluation. • Self-selection – Participants see social media request to take part in survey.
Haphazard Opportunistic participant The frst 20 adults seen walking through When target population is generalizable (Convenience) recruitment based on populations the park are invited to complete a and other sampling techniques are too readily available or easy to recruit pedestrian safety survey. resource-intensive, not cost-efcient, through word-of-mouth (e.g., such or not necessary to answer evaluation as snowball sampling) questions.
In creating your sampling plan, your partners should prioritize the sampling methods that best match your evaluation questions, design, and audiences as well as your partners’ skills and experience, timeline, and resources available to recruit participants and collect data.
Evaluation Guide for Pedestrian Safety Section 3: Evaluation Design | 56 B. Determining your sample size Next, you and your evaluation partners should decide on the size of the sample or samples necessary to make causal inferences about your population or subpopulations of interest based on the evaluation fndings from your sample.
The goal is to increase statistical power in order to increase confdence that the evaluation fndings are detecting an intervention efect when the intervention efect truly exists.
Tools & Resources
G*Power A tool to compute statistical power analyses for many diferent t tests, F tests, χ2 tests, z tests and some exact tests. G*Power can also be used to compute efect sizes and to display graphically the results of power analyses.
Power and Sample Size .com Free and open source online calculators.
PS An interactive computer program for performing statistical power and sample size calculations.
PowerUp! A tool that provides convenient excel-based functions to determine minimum detectable efect size and minimum required sample size for various experimental and quasi-experimental designs.
PowerUp!R R package version of PowerUp! and additionally includes functions to determine sample size for various multilevel randomized experiments with or without budgetary constraints.
R Package pwr A package of tools with functions for basic power calculations using efect sizes and notations from Cohen (1988).
Evaluation Guide for Pedestrian Safety Section 3: Evaluation Design | 57 Tools & Resources
Russ Lenth’s power and sample-size page Software is intended to be useful in planning statistical studies.
WebPower A free collection of tools for conducting statistical power analysis online.
SampSize (app for Android, iOS iPhone, iPad) The design of clinical trials by calculating the sample size using inputs provided by the user.
These power calculations should account for precision of the intervention efect estimates, systematic errors in the data collected or analyzed, and loss of participants in follow-up assessment over time (attrition).
Seek out referrals for statistical expertise from government agencies or academic institutions when more rigorous sampling and power techniques are needed.
Evaluation Guide for Pedestrian Safety Section 3: Evaluation Design | 58 SECTION 4: METHODOLOGY AND DATA COLLECTION
This section provides steps to help you identify data sources and design data collection methods to answer your evaluation questions.
Step 1: Identify data sources or design data collection methods There are two types of data that you can use in your evaluation: quantitative data and qualitative data. These data types can be used to provide insight to the evaluation questions using a variety of evaluation methods.
1. Quantitative data and methods Quantitative data are numerical data that can be counted (or quantifed) and statistically analyzed. The data refer to the specifc assessment measures to capture sociodemographic information, geographic characteristics, knowledge and attitudes, perceptions and beliefs, behaviors, policy or environmental attributes, quality of implementation, exposure to the intervention, etc. The process of collecting this data using quantitative methods is intended to uncover numerical patterns and trends, yet the methods may be applicable to both quantitative and qualitative data. For instance, a survey (method) can have quantitative data (measures that are numeric – income, Likert scales) and qualitative data (measures that are non-numeric – race/ethnicity, open-ended questions).
Evaluation Guide for Pedestrian Safety Section 4: Methodology and Data Collection | 59 Common types of quantitative data and methods that can be used to evaluate outcomes corresponding to pedestrian safety interventions include: Common Quantitative Description Examples Data and Method Types
Pedestrian Counts Physical counts of pedestrians at any given • National Bicycle and Pedestrian Documentation Project provides Surveys sidewalk, path, crosswalk, or roadway. Counts detailed guidance and templates on how to collect pedestrian can be used: count data. • As a proxy measure for exposure when • Pedestrian and Bicycle Data Collection in United States analyzing pedestrian-involved crashes at Communities also provides information on how to collect data for specifc locations pedestrian safety programs. • To measure changes in pedestrian volume • Pedestrian Safety Knowledge (child/oralinstrument)18 is a survey before and after a safety intervention is instrument used to assess children’s knowledge of pedestrian implemented. safety, which include route selection, proper looking before Questionnaires that ask predetermined and while crossing, and how to walk safely on streets without questions on knowledge, attitudes, opinions, sidewalks. experiences, and behaviors of pedestrians • Pedestrian Behavior Questionnaire19 is a survey administered to and/or drivers. parents to report children’s weekly walking habits.
Trafc,LawEnforcement, Trafc, law enforcement, and injury State and national dataset include: and Injury Surveillance surveillance datasets are comprehensive • Crash and roadway data (Police Reports, Fatality Analysis Reporting Datasets repositories of data that are collected, System (FARS), Inventories/Roadway Safety Audits and Reviews) compiled, and validated by national and state • Vehicle Registration agencies. These datasets – which include • Citation/Adjudication data collected by trafc safety organizations, • State Level Hospital Inpatient Discharge Data police departments, and public health • State Level Emergency Department Data agencies – can include a variety of variables • EMS that can inform evaluations of pedestrian • Trauma Registry safety interventions. • Death Certifcates
FOOTNOTES 18. Schwebel, D. C., & McClure, L. A. (2014). Training Children in Pedestrian Safety: Distinguishing Gains in Knowledge from Gains in Safe Behavior. The Journal of Primary Prevention, 35(3), 151–162. http://doi.org/10.1007/s10935-014-0341-8 19. Stavrinos D, Byington KW, Schwebel DC. The efect of cell phone distraction on pediatric pedestrian injury risk. Pediatrics. 2009;123:e179–185
Evaluation Guide for Pedestrian Safety Section 4: Methodology and Data Collection | 60 Tools & Resources
Pedestrian Safety Evaluation: Qualitative Data Collection Methods Provides greater detail on specifc qualitative data collection methods that can be used to inform pedestrian safety interventions.
2. Qualitative data and methods Qualitative data are descriptive, non-numerical data that approximate or characterize – but do not quantify – the attributes, characteristics, and properties of a thing or phenomenon. Qualitative data can provide contextual information that can convey the “how” and “why” of a phenomena or issue though the expressed thoughts, feelings, attitudes, and values of a person or group. Qualitative data describe whereas quantitative data defne.
Examples of qualitative data collection methods that can be used to evaluate pedestrian safety interventions include:
Focus Groups An interactive discussion between a homogenous sample of six to eight individuals. Focus groups are usually facilitated by a trained moderator who focuses on a specifc set of topics to capture social trends and group perspectives.
Key Informant Interviews A semi-structured, one-on-one conversation designed to gain insight on a given topic. The interviewer will guide the interviewee through a discussion of their own life experience, perspectives, and opinions to further understand or create new knowledge about a specifc subject.
Observations A research method aimed to systematically observe and record interactions, events, locations, etc. between individuals and their environment in a natural state. Observations provide allow evaluators to describe and understand people’s behavior in context.
Case Studies An in-depth study of a documented event aimed to narrow down a board topic and ofer a real-world application of a specifc concept or theme. Case studies provide supplemental
Document Analysis A form of research that systematically reviews a document, policy brief, public record, etc. and then interprets the data to measure the impact of the fle.
Inventories/Roadway A collection of all road-related information that defnes and monitors the state-owned highway network, maintaining Safety Audits and Reviews an inventory of the roadway features, conditions, and characteristics. Road Safety Audit (RSA) is the formal safety performance examination of an existing or future road or intersection by an independent, multidisciplinary team. It qualitatively estimates and reports on potential road safety issues and identifes opportunities for improvements in safety for all road users (FHWA).
Evaluation Guide for Pedestrian Safety Section 4: Methodology and Data Collection | 61 Tools & Resources
Pedestrian Safety Evaluation: Qualitative Data Collection Methods Provides greater detail on specifc qualitative data collection methods that can be used to inform pedestrian safety interventions.
3. Basic Diferences between Quantitative and Qualitative Data BelowisasummaryexplainingthekeydiferencesbetweenQualitativeandQuantitativeData20
Qualitative Quantitative
Purpose To describe a situation, gain insight to particular practice To measure magnitude
Format No pre-determined response categories Pre-determined response categories, standard measures
Purpose In-depth explanatory data from a small sample Wide breadth of data from large statistical representative sample
Data In-depth explanatory data from a small sample Wide breadth of data from large statistical representative sample
Analysis Draws out patterns from concepts and insights Test hypotheses, uses data to support conclusion
Result Illustrative explanation and individual responses Numerical aggregation in summaries, responses are clustered
Sampling Theoretical Statistical
Cost With small samples, this data can be cheaper to collect Existing publicly available data sets have no data collection costs With large populations, this data can be more expensive to collect
FOOTNOTES 20. Nigatu, Tilahun. “Qualitative Analysis”. 2009. Presentation.
Evaluation Guide for Pedestrian Safety Section 4: Methodology and Data Collection | 62 4. Benefts of Combining Quantitative and Qualitative Data (Mixed method evaluations) A mixed method evaluation is an evaluation approach in which both quantitative and qualitative data are collected, analyzed, and integrated to address one or more evaluation questions.21
Although quantitative and qualitative data have individual strengths, oftentimes both are needed to understand a complex problem and strengthen an evaluation. Quantitative data can provide statistically reliable numerical information, while qualitative data can provide richer and deeper insights into the phenomenon under study. Therefore, a mixed method evaluation provides an efective way to combine and cross-apply quantitative and qualitative data to present a reliable and valid set of evaluation results.
The table below details the strengths and limitations of quantitative only, qualitative only, and mixed method evaluation approaches.
Strengths and Limitations of Evaluation Approaches Strengths Limitations
Quantitative Only • Can incorporate large sample sizes • May be difcult to understand context of fndings • Data can be trended across multiple years • May lack specifcity for the needs of the evaluation • Data obtained from large datasets may not be up to date (data may be lagged by one or more years) • Lacks the richness and detail of descriptive information based on human experience
Qualitative Only • Allows for the examination of phenomena in rich detail • Often utilizes a relatively small sample size • Provides insights based on human experience • Findings are only generalizable to the study population • Data can uncover subtleties not addressed by • Lacks the validation of statistical analysis quantitative data sources
Methods • Combines inductive and deductive reasoning and • May require more resources to conduct both evaluation methods thinking through the triangulation of evaluation • Training is necessary for evaluators to cross over and collect/ fndings analyze both data types • Strengthen the reliability of data • Can help ensure evaluation fndings are robust, valid, and credible • Reduces evaluator bias when interpreting results
FOOTNOTES 21. (From John Creswell’s 2013 article: http://digitalcommons.unl.edu/cgi/viewcontent.cgi?article=1047&context=dberspeakers)
Evaluation Guide for Pedestrian Safety Section 4: Methodology and Data Collection | 63 Asyoucreateyourevaluationplananddeterminethespecifcdatacollectionmethodsyouwilluse,youmuchconsiderthefollowing: • The purpose of the evaluation and the end users of the evaluation fndings Determine which evaluation and data collection methods would be most appropriate to answer the evaluation questions and yield credible results. Also, consider the methods would obtain the most buy-in from the primary audience or the end-users. • The availability of data and resources for the evaluation Determine if you will have to collect your own data (primary data) or if you can utilize existing data sources (secondary data). The time and resources available to collect data should balance the strengths and limitations of each data source. Some methods are more costly to collect, analyze, and interpret.
Step 2: Create a matrix to connect your evaluation questions, indicators, and data sources A data collection matrix is a helpful planning tool that can help you visualize how all the data you have or need can ft together to answer your evaluation questions.Abasicdatacollectionmatrixcontainsseven(7)mainelements:
Data Collection Matrix Elements Defnition
1. Evaluation questions A high-level question aimed to understand the value, impact, and signifcance of the intervention. A well- written evaluation question will serve as the guiding framework for your evaluation design and identify will what you want to understand about the intervention.
2. Indicators Specifc qualitative and quantitative data points that are used to operationalize the outcomes or activities of the intervention. It is recommended to have more than one indicator for each outcome or activity being evaluated.
3. Data sources Identifes from whom or what agency the data will come from for the evaluation.
4. Data collection method The method at which the data will be captured. This includes, but not limited to, surveys, focus groups, interviews, observations, etc.
5. Responsible party The individual or entity responsible for collecting/ providing the needed data. For example, an external evaluator will be the responsible party that will conduct interviews and collect primary data for the evaluation.
6. Timing Related to the frequency at which the data will be collected (e.g. daily, quarterly, and annually). Serves as a timeline of data collection for your evaluation.
7. Analysis plan A roadmap for how you plan to organize and assess the data to make sense of the information and identify fnal conclusions to answer the evaluation question.
Evaluation Guide for Pedestrian Safety Section 4: Methodology and Data Collection | 64 Below is an example of a data collection matrix that is used to plan the collection of evidence of the impact of a pedestrian safety intervention.
Example Data Collection Matrix
Evaluation Question: What is the impact of the infrastructure changes at the crosswalk on pedestrian and trafc volumes and behavior?
Indicator Data Source Data Collection Responsible Party Timing Analysis Plan Method
Pedestrian crossing FARS, police records Secondary data Evaluator Annually, August Time series crashes source
Non-pedestrian FARS, police records Secondary data Evaluator Annually, August Time series crossing crashes source
Drivers Yielding Audit/observation Observation Department of Pre-intervention Pre/post analyses During ‘WALK’ signal Transportation Post-intervention
Pedestrians Crossing Pedestrian counts Observation Department of Pre-intervention Pre/post analyses During Beginning of Transportation Post-intervention WALK Cycle
Perceived pedestrian Interviews Qualitative Evaluator Post-intervention Emerged themes safety at the from coding interview crosswalk transcripts
Tip for Success
To fnd out more on how to develop evaluation questions see Section I: Focus the Evaluation.
Evaluation Guide for Pedestrian Safety Section 4: Methodology and Data Collection | 65 SECTION 5: DATA ANALYSIS
In this section, a cooking analogy will be used to simplify and explain the data analysis process, including the partners conducting the analyses, the variables or themes used in the analyses, steps to prepare and methods to analyze the data, interpretation of the results, and the culminating plan.
Step 1: Convene data analysis partners Convening your partners responsible for data analysis is much like gathering your “chefs,” “cooks,” “dishwashers,” and “servers.”
The “chefs” represent partners with the big picture in mind. • These partners are able to connect the evaluation purpose, intervention goals and objectives, evaluation design, data collection methods, and indicators and standards to the data analysis plan. • Refer to the list of partners with data analysis roles from Section 1.
The “cooks” and “dishwashers” are responsible for implementing the analysis plan. • This implementation requires several technical skills, including mathematics, statistics, and probability theory; economic theory; systems theory; and information technology, data management, data coding, and programming. • While existing partners may possess some or all of these technical skills, it is helpful to be prepared to recruit consultants or contractors to support implementation.
The “servers” refer to the partners responsible for translation and dissemination of fndings (see Section VI). • At some level, all partners should be involved in the interpretation of the quantitative and qualitative fndings to ensure utility, propriety, feasibility, and accuracy of the evaluation.22 • Utility standards ensure that an evaluation will serve the information needs of intended users. • Feasibility standards ensure that an evaluation will be realistic, prudent, diplomatic and frugal. • Propriety standards ensure that an evaluation will be conducted legally, ethically and with due regard for the welfare of those involved in the evaluation, as well as those afected by its results. • Accuracy standards ensure that an evaluation will reveal and convey technically adequate information about the features that determine worth or merit of the program being evaluated.
FOOTNOTES 22. https://www.cdc.gov/eval/standards/index.htm
Evaluation Guide for Pedestrian Safety Section 5: Data Analysis | 66 Template
Data Analysis Planning Grid Template A template to guide data analysis activities for evaluation.
Step 2: Identify variables and themes Variables or themes are the key “ingredients” in quantitative and qualitative data analysis “recipes,” respectively.
A variable or theme can represent an attribute of a person, place, thing, or idea.
Variables can be quantitative (numeric) or qualitative (categorical, non-numeric); alternatively, themes are all qualitative.
Themes are typically descriptive in nature and, therefore, more complex than variables to analyze.
An original set of variables or themes is derived from measures already available in your secondary data sources or those obtained through your primary data collection tools.
Analysis variables can be created to summarize one or more of the original variables. Through data reduction approaches, themes can also be transformed into variables, if desired.
Examples of analysis variables derived from original variables designed to characterize intervention exposure and impact include: • “High-risk population reach” = the proportion of racial, ethnic, and/or lower-income populations exposed to the intervention; • “Intervention dose” = scale or size of the intervention setting (e.g., # of feet of roadway) + stage of implementation (e.g., policy adopted, funds allocated, environment modifed, enforcement underway) + quality of implementation (e.g., high, low); and • “High-risk population impact” = high-risk population reach + intervention dose.
Evaluation Guide for Pedestrian Safety Section 5: Data Analysis | 67 Many types of variables or themes may be included in quantitative and qualitative data analysis refecting populations or samples, interventions, outcomes, and contextual conditions. See the table below for examples of variables in each of these categories.
Examples of Variables Variable Types Quantitative (Numeric) Qualitative (Categorical, Non-Numeric)
Variables Demographic: age, income Demographic: gender, race/ethnicity, ability status Geographic: population density within 1 mile radius of Geographic: urban, rural, suburban; areas (e.g., counties, street improvement metropolitan areas, zip codes, census tracts or block groups); corridors (e.g., continuous roads or streetscapes); intersections
Intervention Inputs: number of partners, amount of funding, Inputs: types of partners (e.g., urban planning, health, Variables monetary value of in-kind resources transportation, residents, businesses), types of funding (e.g., grants, Activities: number of policy, practice, or environmental tax dollars, budget appropriations), types of in-kind resources (e.g., changes planned, implemented, or enforced; number of meeting space, equipment, supplies) educational programs or campaigns implemented Activities: types of policy, practice, or environmental changes Outputs: planned or implemented, Organizational: number of new positions or Outputs: committees to support pedestrian safety in the region Organizational: new positions (e.g., Pedestrian/Bike Coordinator), or state new decision-making bodies (e.g., Balanced Transportation Policy: number of cities and counties adopting Committee), new or modifed practices (e.g., use of form-based ordinances for street trees zoning codes) Environmental: # of feet of streetscapes with widened Policy: new or modifed plans (e.g., pedestrian master plan, sidewalks transportation management plan, security master plan), new or Educational or Promotional: media hits modifed policies (e.g., mixed-use zoning regulation, complete streets ordinance, trafc calming policy, speed limits) Environmental: new or improved sidewalks (e.g., presence or absence), quality of installation of trafc calming devices (e.g., high, low) Educational or Promotional: types of classes (e.g., pedestrian safety training ) or events (e.g., pedestrian summit)
Evaluation Guide for Pedestrian Safety Section 5: Data Analysis | 68 Examples of Variables Variable Types Quantitative (Numeric) Qualitative (Categorical, Non-Numeric)
Outcome Knowledge/ awareness/ skills: percent of city council Knowledge/ awareness/ skills: population or subpopulation Variables members with increased knowledge of pedestrian exposure to campaign messages (e.g., exposed, unexposed) injuries and fatalities Behaviors: walking (e.g., walking for leisure, walking for Behaviors: walking (e.g., pedestrian counts, energy transportation), use of public transit (e.g., high, low, driving (e.g., expenditure), proportion of the population using public types of distracted driving, speeding) transit, driving (e.g., percent decrease in distracted driving) Health: perceived general health status (e.g., excellent, fair, poor) Health: pedestrian injuries, pedestrian fatalities, quality- Social: perceived safety of walking during the day or night adjusted life years saved Economic: per capita return on investment, cost per quality-adjusted life year Social: crime rates, unemployment rates, poverty rates
Contextual Political climate: polling data on public support for Political climate: changes in elected or appointed ofcials Variables pedestrian safety initiative Economic climate: budget freeze Economic climate: local revenue allocated to pedestrian Social determinants: poverty and income inequality (e.g., federal safety improvements poverty levels), education (e.g., academic achievement), quality of Social determinants: poverty and income inequality (e.g., housing, civic participation, social cohesion, discrimination GINI coefcient), education (e.g., high school graduation rate), employment (e.g., unemployment rate), language and literacy (percent with English as second language)
Consider how your variables may align with the following categories:
Independent variables – variables related to populations or subpopulations, interventions, and contextual factors;
Dependent variables – also known as “outcome variables”; and
Confounding variables – variables related to populations or subpopulations and contextual factors.
Evaluation Guide for Pedestrian Safety Section 5: Data Analysis | 69 Step 3: Organize data and ensure data quality Good kitchen organization is necessary to ensure food is prepared safely and properly. Similarly, organizing data and ensuring data quality are necessary precursors to data analysis.
A. Organize data To organize your quantitativedata,thereareawiderangeofdatabasemanagementsystems–relational,network,fat,and hierarchical – that can be customized in order to perform the following tasks relevant to your evaluation data: •Defningdata–Creating data defnitions and relationships to organize the data (e.g. participant ID link to survey responses, street segment ID link to items from an environmental audit, city ID link to policy assessment data, county ID link to sociodemographic variables, and latitude and longitude coordinates link to spatial data for mapping) as well as setting parameters for valid values for each variable. • Entering data – Designing forms for inputting new data, modifying existing data, or deleting unwanted data. • Downloading or reporting data – Developing usable formats for transferring data for further processing in data analysis applications or sharing with diferent audiences. • Administration – Registering and monitoring users, enforcing data security and privacy, monitoring system performance, maintaining data integrity, and recovering information corrupted by an event.
To organize and properly name and store your qualitativedatarecords(e.g.,audiofles,transcripts,consentforms,observationnotes/ rubrics),thereareafewessentialcomponents,including: • Identifying metadata to archive records, including data collection methods (e.g., interviews, focus groups, observations), dates or times of data collection, names of those responsible for data collection, individuals or groups of participants, and names of those responsible for coding or reviewing the analysis. • Centralizing and organizing records in a location with proper fles names based on metadata (e.g., “
B. Ensure data quality Several steps can be taken to ensure data quality during data collection (Section 4), such as using tools with established validity and reliability and engaging multiple people in data collection to establish inter-observer reliability.
Evaluation Guide for Pedestrian Safety Section 5: Data Analysis | 70 Likewise, good data analysis involves several complementary steps to ensure data quality prior to analysis, including: using multiple data analysts, creating a data analysis protocol, training data analysts, reviewing data validity and reliability, addressing missing information, and addressing bias.
Using multiple coders or data analysts is particularly helpful for qualitative data analysis to reconcile diferences or subjective interpretations of the data. While multiple coders or analysts require more organization, time, and money, the resulting themes or codes are typically higher quality, given the opportunity to discuss varying interpretations and generate consensus on the most appropriate themes or codes.
Your eforts to maximize data quality help to minimize threats to internal and external validity, and can increase confdence in the fndings from the data analysis (causal inference).
You and your partners should discuss the following data quality control procedures to ensure that the evaluation fndings demonstrate changes in the outcomes of interest attributed to the intervention as opposed to alternative explanations.
Quality Data Quality Control Procedures Considerations Quantitative Data Qualitative Data
Validity If data collection tools did not have established validity, Assess credibility, transferability, and confrmability of the data as conduct an analysis of the validity of the tools used compared to similar other populations and contexts.23 (construct, content, and criterion validity). Review each variable to ensure the values are appropriate (i.e., no outliers).
Reliability If data collection tools did not have established Assess dependability of the data on individual circumstances.24 reliability, conduct an analysis of the reliability of the tools used.
Missing With sufcient sample sizes, and representativeness, N/A information or consider imputation, interpolation, or modeling loss to follow up techniques to recapture this data.
Bias Report any potential sources of bias in sampling, data collection, or data analysis.
FOOTNOTES 23. Lincoln, YS. & Guba, EG. (1985). Naturalistic Inquiry. Newbury Park, CA: Sage Publications. 24. Adapted from: University of Albany Center for Problem-Oriented Policing. Assessing Responses to Problems: An Introductory Guide for Police Problem-Solver. Appendix D: Summary of Evaluation Designs’ Strengths and Weaknesses. Retrieved from: http://www.popcenter.org/tools/assessing_responses/7
Evaluation Guide for Pedestrian Safety Section 5: Data Analysis | 71 Quality Data Quality Control Procedures Considerations Quantitative Data Qualitative Data
Unintended Ensure the analysis data set is complete, including variables for population and sample characteristics and contextual consequences conditions. and extraneous variables (confounding, modifying)
Measurement Outcome data must be collected after intervention delivery has begun. timing Account for multiple intervention components or changes in dose of the intervention.
Measure Avoid changes in measures from baseline to follow up. adaptation
Step 4: Agree on data analysis methods In this step, partners should agree on the “recipes” to be used. A recipe provides the fnished product, ingredients, and directions. Similarly, the data analysis method identifes the goal/ objective, the set of variables/ themes, and the step-by-step methods for carrying out the analysis.
Your analytical methods may be used to describe, compare, or predict variables/ themes or relationships among variables/ themes proposed in the goals, objectives, or evaluation questions.
A. Describe variables or themes For quantitative data, there are three primary ways to describe variables, including: • Descriptive statistics summarize the distribution of the data using measures of central tendency and spread. Measures of central tendency include the mean, median, and mode. Measures of spread include range, variation, and standard deviation. Descriptive statistics can be easily summarized in tables or graphs. • Frequencies are counts of occurrences for a particular variable or theme. Frequencies may summarize several data points, or cases, at once and are often reported as a percent. Displaying data across diferent variables in cross tabulations can further illustrate the data distribution. • Geocoded data use geographic coordinates (latitude and longitude) to spatially orient the data. These data are often shown in maps.
Evaluation Guide for Pedestrian Safety Section 5: Data Analysis | 72 For qualitative data, there are two main approaches to identifying and describing themes, or coding data from qualitative sources, including: • Deductive data analysis involves interpretive techniques designed to extract and summarize diferent themes and the frequency of those themes in a dataset (e.g., transcripts from an interview or focus group, elements of a policy identifed through policy analysis) using preset codes derived from the intervention goals and objectives or the evaluation questions. • Inductive data analysis uses focused coding procedures to identify indigenous or emergent themes, or ideas and concepts derived from the data; themes are inductively organized into categories, or sensitizing concepts, to create new themes or codes or to replicate existing themes or codes.25, 26
For deductive qualitative data analysis, software applications (e.g., AtlasTI, NVivo) are frequently used by qualitative analysts to assist with both the organization and coding of qualitative data.
B. Compare or predict variables or relationships among variables Most data analysis methods used for comparison or prediction fall into the universe of inferential statistics.
In contrast to descriptive statistics that represent observed properties of the data, inferential statistics are used to make inferences about population data based on observations in a sample data set and assumptions refecting diferent models, parametric (continuous data) or non- parametric (categorical data).
These models are used to assess the probability that the observations for the sample are true for the population; this is referred to as the level of signifcance, p, and p < .05 is a common value that is used to identify an acceptable probability.
Simple inferential statistical analysis methods are commonly used to compare variables to one another or to compare variables by diferent samples.
• Correlations (parametric) are used to determine the degree of correspondence between two variables; these analyses do not assess causality (i.e., one variable causes the other variable to occur). Correlations are used to determine whether the correspondence between or among variables is strong or weak and positive or negative.
FOOTNOTES 25. Patton M. Qualitative research and evaluation methods. 3rd ed. Thousand Oaks, CA: Sage; 2002. 26. Bowen G. Grounded theory and sensitizing concepts. International Journal of Qualitative Methods 2006;5(3).
Evaluation Guide for Pedestrian Safety Section 5: Data Analysis | 73 • T-tests (parametric) or Chi-Square Analyses (non-parametric) are used to determine whether there is a signifcant diferent between two groups on a specifed variable. T-tests compare the means of the two groups and Chi-Square Analyses compare proportions for the two groups.
More complex comparisons examine relationships between independent and dependent variables.
Again, dependent variables typically refect the outputs as well as short-term, intermediate, and long-term outcomes from the logic model.
And, independent variables represent all the other factors (i.e., populations and samples, intervention, contextual conditions) that may explain any changes in the dependent variables over time.
• Analysis of Variance (ANOVA; parametric) assesses relationships of independent variables to dependent variables by examining diferences in means among populations or evaluation samples or subsamples (e.g., higher or lower income groups, exposed or unexposed to pedestrian safety campaign messages, residents living in a mixed-use versus traditional development, pre- versus post- intervention). Relationships of multiple independent (e.g., two-way ANOVA) or dependent variables (Multiple Analysis of Variance, MANOVA) can also be assessed through this analysis method.
Simple and complex analysis methods are also designed to predict relationships among variables or to incorporate mixed-methods analysis, or triangulation, refecting a range of diferent data sources.
• Regression analysis is used to determine whether one variable (predictor) can be used to predict the outcome of another variable (criterion). The output from a regression analysis identifes statistically signifcant predictors of the criterion as well as the amount of variation in the criterion that is explained by the set of predictors. • Health impact assessment (HIA) is a process to evaluate the potential health efects of a plan, project, or policy before it is built or implemented; the major steps include screening, scoping, assessment, recommendations, reporting, and monitoring and evaluation.27 • Spatial Analysis uses geometric (points, lines, shapes) and geographic (latitude and longitude coordinates) data to analyze variables and their connection to places.
FOOTNOTES 27. https://www.cdc.gov/healthyplaces/hia.htm
Evaluation Guide for Pedestrian Safety Section 5: Data Analysis | 74 • Model building is used to go beyond simple comparisons and predictions with a small set of variables to examine complex relationships among a large set of variables. Some prominent analytical methods include: • Hierarchical linear modeling is designed to work with nested data (e.g., analyzing people within organizations within communities). • Structural equation modeling identifes direct and indirect pathways among independent and dependent variables and tests these relationships as well as the ft of the entire model of paths and relationships. • Economic modeling includes cost-efectiveness analysis, cost-beneft analysis, and cost utility analysis. • Systems modeling includes Markov decision-choice models, agent-based models, system dynamics models, and theoretical mathematical models.Step 5: Interpret results
Step 5: Interpret results Near the end of your food preparation steps, you will also pay attention to cues that your snack or meal is ready to serve (e.g., whether the food has had sufcient time to cook or bake). Similarly, you will want to review and compare your analysis results to standards in the feld as part of interpreting your fndings.
A. Review your data to identify patterns Patterns in your data may be related to observed relationships among independent and dependent variables, trends for diferent subpopulations within the overall population, or extraneous variables that alter anticipated relationships of independent and dependent variables.
B. Refer to indicators and standards As noted in previous sections, indicators are used to quantify and compare data across settings and populations.
Tools & Resources
The Good Indicators Guide: Understanding how to use and choose indicators Provides general guidance on indicators.
References to available national, state, or local standards allows you and your partners to refect on an evidence base for assessing your intervention’s progress toward achieving the goals and objectives (i.e., exceeded standards, met standards, did not change, or changed in the wrong direction).
Evaluation Guide for Pedestrian Safety Section 5: Data Analysis | 75 Several national and international standards are available to guide this process; some already depict numeric indicators and current standards: • Healthy People 2020 Leading Health Indicators for Injury and Violence, https://www.healthypeople.gov/sites/default/fles/HP2020_LHI_ Injury_Viol.pdf • National Association of County and City Health Ofcials and Safe States Alliance, https://c.ymcdn.com/sites/safestates.site-ym.com/ resource/resmgr/imported/FINAL%20LHD%20IVP%20Standards%20and%20Indicators_071211.pdf) • Smart Growth America’s “Dangerous by Design” - https://smartgrowthamerica.org/dangerous-by-design/
Others provide descriptions or recommendations for indicators and standards: • Centers for Disease Control and Prevention, https://www.thecommunityguide.org/ and http://www.cdc.gov/mmwr/preview/mmwrhtml/ su6004a13.htm • American Public Health Association, https://www.apha.org/~/media/fles/pdf/topics/injury_and_violence_prevention_policy_lessons_from_ the_feld.ashx • Association of State and Territorial Health Ofcials and Safe States Alliance, http://www.astho.org/programs/prevention/injury-and- violence-prevention/ • Robert Wood Johnson Foundation’s Public Health Law Research, http://publichealthlawresearch.org/topic/injury-and-violence-prevention • World Health Organization, http://www.who.int/violence_injury_prevention/publications/39919_oms_br_2.pdf.
Your indicators and standards are used to judge and communicate intervention success. For instance, if the intervention exceeded or met the standards, these fndings can be used to advocate for long-term investment of personnel time and other resources into the intervention to increase its sustainability.
Evaluation Guide for Pedestrian Safety Section 5: Data Analysis | 76 Placeyourfndingsinatabletoconnectindicatorsandstandardsfromthefeldtoyourfndingsandjudgetheefectivenessofyour intervention.Thesejudgmentswillinformthecontentforyourdisseminationeforts.
Intervention Goal/ Outcomes Evaluation Data Source/ Data Indicator/ Analysis Results Objective (Logic Model) Question(s) Collection Method Standard(s)
1.
2.
3.
Evaluation Guide for Pedestrian Safety Section 5: Data Analysis | 77 Step 6: Create a data analysis plan Your data analysis plan is your “cookbook,” combining all of the recipes that your partners will use to address your evaluation purpose (Section 1) and questions (Section III).
Your data analysis plan should also incorporate your variables, data quality control procedures, and partners’ responsibilities and timelines.
You may consider organizing your plan according to the goals and objectives of the intervention (Section 2) and/or the related evaluation questions (Section 3).
Here is an example outline for your data analysis plan: Goal 1 Objective 1 " Evaluation question, design, and populations/samples " Variables, data sources, and data collection methods " Analysis method " Indicators and standards from the feld " Analysis results " Tasks, responsibilities, due dates
Goal 1 Objective 2 " Evaluation question, design, and populations/samples " Variables, data sources, and data collection methods " Analysis method " Indicators and standards from the feld " Analysis results " Tasks, responsibilities, due dates
Evaluation Guide for Pedestrian Safety Section 5: Data Analysis | 78 SECTION 6: REPORTING AND DISSEMINATION This section provides steps to report and disseminate evaluation fndings to various audiences and consider how they will use the evaluation fndings when determining the appropriate communication product and dissemination method.
Step 1: Identify audiences and develop a dissemination plan Early in the evaluation planning process, it is important to identify the audiences and key partners that will be interested in the evaluation results. Identifying both the internal and external audiences will inform the reporting styles, messages, and format for sharing the key evaluation fndings. Internal audiences can include internal agency partners, intervention implementers, or community members. External audiences can include political decision makers, funders, or advocacy groups. It is important to understand how your audiences prefer to receive evaluation fndings so that you can deliver them in formats that are engaging, comprehensible, and digestible. A well-written dissemination plan should outline specifc strategies used to communicate pedestrian safety evaluation fndings to various stakeholder groups. It is important to map out how you and your evaluation partners plan to share your fndings to ensure you’re sending information to the appropriate audience in their preferred format. Below is a summary of the dissemination plan components. • Audience: Who is going to receive evaluation report? • Product: What kinds of information will be presented
Tip for Success
See Step 2: Selecting reporting styles to share evaluation fndings
• Delivery: How will the information will be delivered (e.g., print, electronic, verbal, visual-aids? • Key Messages: What are the main highlights that you plan to present and how should the messages be framed? • Release Date: When will this information be disseminated? • Person Responsible: Who is responsible for developing and releasing this information • Follow-up Activities: Notes for yourself that tracks the progress of each report being disseminated • Budget: How much will it cost to implement the dissemination plan?
Evaluation Guide for Pedestrian Safety Section 6: Reporting and Dissemination | 79 Step 2: Select key messages for specifc purposes Now that you know how you are going to disseminate your evaluation fndings, it’s time to identify how to select and frame key evaluation fndings for diferent audiences and purposes. This section will help you identify specifc evaluation fndings that can help you and your evaluation partners to: • Obtain funding and resources • Discuss impacts of specifc policies and practices • Conduct continuous quality improvement to enhance the implementation of a program, policy, or system • Reach new populations • Overcome barriers or resistance to the intervention • Increase intervention efectiveness • Inform intervention sustainability
Step 3: Select communication methods and create evaluation products Onceyouandyourevaluationteamhaveidentifedkeyaudienceswithwhomyoucansharethe evaluationfndings,nextstepsinclude: • Selecting communication methods and reporting styles to share evaluation fndings; • Presenting your evaluation fndings efectively: data visualization techniques; and • Framing evaluation fndings and developing recommendations for next steps.
Selecting reporting styles to share evaluation fndings There are many types of reporting styles and approaches that you and your evaluation partners can use to share your evaluation fndings. Ultimately, they depend on your audience. The following section will describe fve common types of evaluation reports.
Traditional/comprehensive reports Traditional reports tend to be formal and comprehensive in nature and should follow the standard format of reporting in your agency. The document provides a detailed summary of the evaluation goals and objectives, methodology, and fndings, and is illustrated with facts and fgures to showcase the data. Oftentimes these reports are used to reference the historical context, disseminate information across the agency, and add knowledge to reporting systems and future programming eforts. Although these reports are valuable to an organization for the reasons listed above, it can take a lot of efort and resources to complete the report, which can be costly.
Evaluation Guide for Pedestrian Safety Section 6: Reporting and Dissemination | 80 Executive Summaries An executive summary is an abbreviated version of a traditional report, meaning it summarize the same sections found in the comprehensive report. Often, executive summaries are written after the full report is completed and excerpts copied and pasted in from the larger report and restricted to create the condensed report. More information about executive summaries is discussed later in the section.
Issue Briefs and Fact sheets Issue briefs and facts sheets are short, written documents that state the issue at hand, outline recommended action steps, and provide supplemental information to support fndings and recommendations. Issue briefs are usually one to two pages and include a list of references and contact information for the author. Issue briefs and fact sheets can be useful when trying to enlist support from elected ofcials or other decision-makers.
Presentations to Program Leadership and Stakeholders Often, you will have a specifc audience including upper-level leadership that is interested in the results of your evaluation. Because traditional reports can be overwhelming to read and upper management will not have time to digest all of the information in a dense report, you will need to translate your written evaluation report into a short and concise presentation. Remember, all of the information in the report is relevant, however, the purpose of the presentation is to showcase the high-level issues, actions, fndings, and recommendations for next steps.
Website Posts Posting evaluation fndings on a credible website is an excellent way of communicating information to a large audience. You can upload all reports online and create interactive visuals to display data. Potential postings include: • Evaluation reports • Video presentation • PowerPoint Presentations • Newsletter and brochures • Meeting schedules and agenda • Press releases and other forms of media
Data Dashboards Data dashboards is an online or computer desktop tool that is used for visualizing real-time or archived data to external or internal partners. The goal of the data dashboard is to create an interactive interface where the user can create customized data reports to meet their specifc needs.
Evaluation Guide for Pedestrian Safety Section 6: Reporting and Dissemination | 81 Presenting your evaluation fndings efectively: Data visualization techniques Data visualization is a technique that is used to enhance the quality of evaluation reports and communicates data/information using engaging images that resonate with the audience. It is important to display your data and complement your text with visual and graphic depictions of your fndings. This section of the guide provides you with examples of how to display evaluation fndings using the diferent data visualization techniques such as: • Performance charts: to compare current data points with targets • Line graphs: correlations, time-series • Comparisons • Qualitative: word clouds, call out boxes, meaningful quotations • Pedestrian safety infographics
Performance Charts Bar Chart Pie Chart Shows frequency by category visually instead of using numbers; Shows relative frequency by category visually instead of using bar charts are great for comparing the relative frequency of numbers; pie charts are great for showing importance of each diferent categories category relative to the whole
Evaluation Guide for Pedestrian Safety Section 6: Reporting and Dissemination | 82 Histogram Time Series A bar chart for grouped numerical data in which the frequencies or These trends may also appear for two variables, but the horizontal percentages if each group are representative as individual vertical variable must be a measurement of time. Time series graphics and bars; histograms are great for showing multiple variables on one analyses are great for illustrating historical trends and predicting chart future values and trends over time.
Line Graphs
Line Graph Scatter Plot Line graphs are used often to show trends over time. Shows the actual data as individual (ungrouped) data points for two variables. Scatter plots are used in analysis to draw the best ft line and to show detail of data distribution, spread, and trend.
Evaluation Guide for Pedestrian Safety Section 6: Reporting and Dissemination | 83 Comparisons Qualitative Data In an evaluation, it is always a good idea to show comparison data Since qualitative data focus on themes and categories rather than to demonstrate change before and after the intervention was numbers, graphs, tables, and charts are not a good method to implemented. It is also helpful for presenting diferences between display this type of data. Instead, use the following: two variables. • Word clouds: An image of a group of words relevant to the evaluation fndings. The size of the word demonstrates the Comparisons over time can be represented using an interrupted frequency of its representation in the dataset. There are time series graph. In these graphs, data can be trended and many free resources available online that can help you with compared to illustrate changes that have occurred in a population visualization for qualitative data. over multiple months or years after an intervention takes place • Call out boxes: A graphic element designed to highlight small (e.g., after a policy is adopted). portions of information.
There are a number of ways to show comparison data. All you have • Quotations: Identifying good quotations from interviews or to do is select a data visualization method of your choice and put focus groups to emphasize key points is a good way to validate old and new data side by side to compare. and communicate your evaluation fndings. Find quotes that are insightful, direct, and explanatory by nature. • Stories: In a full report, one or more short stories can be very helpful. These often resonate with policy makers. Include photos.
Evaluation Guide for Pedestrian Safety Section 6: Reporting and Dissemination | 84 Infographics Infographics are a visual representation of information, data, or knowledge displayed in an easy-to-read format. Infographic presents vital information in an attractive and digestible design to educate the reader on the evaluation fndings. Tips for developing infographics?
NY PSAP- Vision Zero: Year Three Report
Evaluation Guide for Pedestrian Safety Section 6: Reporting and Dissemination | 85 Framing evaluation fndings and developing recommendations for next steps Given that evaluation is about improvement, creating recommended next steps is often an important way to start using evaluation fndings. Framing is an important consideration in considering audiences for dissemination. Individuals can interpret the same data in diferent ways depending on the mental model through which they perceive information. It is often good practice to frst share preliminary evaluation fndings and a draft version of recommendations with program implementers and partners to ensure that the fndings are presented in the proper context and recommendations are infused with their valuable feedback. Engaging evaluation partners in the framing of evaluation fndings and development of recommendations: increases the credibility of the evaluation fndings; ensures recommendations are feasible and realistic to implement; and enhances buy-in and likelihood that recommendations will be implemented.
Step 4: Monitor Dissemination Eforts Keeping track of dissemination eforts and estimating the number of individuals or groups that are reached is important to determine if the right people are receiving the accurate and timely information to make decisions and/or programmatic improvements.
A Dissemination Tracking Table can help you to identify the number of individuals that have received a report of your evaluation results and can help you track any next steps or actions that are identifed based on the evaluation fndings. Monitoring this information can potentially be difcult to maintain, however, tracking this information will help you identify which dissemination methods are most successful.
Audience Product Delivery Key Messages Release Date Responsibility Follow-Up
Evaluation Guide for Pedestrian Safety Section 6: Reporting and Dissemination | 86 Glossary Term Definition Achievable Objective An achievable objective is feasible and considers the availability of resources, the scope of the intervention, and is attainable within a bounded timeframe. Activities Found in a logic model, activities describe specific events that are planned to achieve the goals and objectives outlined in the intervention. Activities should be tightly linked to the resources that support the implementation of the proposed strategy, as well as, connected to the intended outcome the activity will achieve. Analysis Plan A roadmap for how you plan to organize and assess the data to make sense of the information and identify final conclusions to answer the evaluation question. Asset mapping Techniques for identifying community resources, such as people, places, services, or other goods (Community Tool Box, http://ctb.ku.edu/en/table-of-contents/assessment/assessing-community-needs-and- resources/identify-community-assets/main). Assumption Hypotheses about factors or risks which could affect the progress or success of an intervention or hypothesized conditions that bear on the validity of the evaluation; assumptions are made explicity in theory- based evaluation. (Development Assistance Committee. (2002). Glossary of key terms in evalaution and results-based management. Paris, France: OECD.) Attribution Attribution involves drawing causal links and explanatory conclusions between observed changes and specific interventions. (Iverson, A 2003, Attribution and aid evaluation in international development: a literature review, prepared for CIDA Evaluation Unit, International Development Research Centre, May.) Additional resources: DeGroff A, Schooley M, Chapel T, & Poister TH. (2010). Challenges and strategies in applying performance measurement to federal health programs. Evlauation and Program Planning, 33, 365-372. Patton MQ. (2008). Utilization-Focused Evaluation Methods.(4th ed.). Thousand Oaks, CA:Sage Publications, Inc. Patton MQ. (2008). Qualitative Research & Evaluation Methods.(3rd ed.). Thousand Oaks, CA:Sage Publications, Inc. W.K.Kellogg Foundation. Knowledge Center. Retrieved from: https://www.wkkf.org/resource- directory/resource/2010/w-k-kellogg-foundation-evaluation-handbook
Audience Refers to whom or what organization intended to receive information (e.g. intervention, evaluation report, executive summary, etc.) from the evaluation. Bias The extent to which a measurement, sampling, or analytic method systematically underestimates or overestimates the true value of an attribute. (US Environmental Protection Agency. (2007). Program Evaluation Glossary. Office of the Administrator/ Office of Policy/ Office of Strategic Environmental Management/ Evaluation Support Division.) Capacity The ability of community members to make a difference over time and across different issues. (http://ctb.ku.edu/en/table-of-contents/overview/model-for-community-change-and-improvement/building- capacity/main) Capacity-Building The intentional, coordinated and mission-driven efforts aimed at strengthening the management and governance of public health agencies to improve their performance and impact (Brownson, EBPH). Case Studies An in-depth study of a documented event aimed to narrow down a board topic and offer a real-world application of a specific concept or theme. Categorical data Measures that place data into a limited number of groups or categories. (US Environmental Protection Agency. (2007). Program Evaluation Glossary. Office of the Administrator/ Office of Policy/ Office of Strategic Environmental Management/ Evaluation Support Division.) Causal inference Judgment about the relationship of causes to the effects they produce; a cause is termed “necessary” when it (causality) always precedes an effect even if it is not the sole cause or the effect is not the sole result; a cause is termed “sufficient” when it inevitably initiates or produces an effect; any given causal factor may be necessary, sufficient, neither, or both (Brownson, EBPH). Coalition Group of individuals and/or organizations that join together for a common purpose (Brownson, EBPH). Confounding An error that distorts the estimated effect of an exposure on an outcome, caused by the presence of an extraneous factor associated with both the exposure and the outcome (Brownson, EBPH). Contextual factors Indicators associated with the surroundings within which a health issue occurs, including assessment of the social, cultural, economic, political, and physical environment (Brownson, EBPH). Continuous data Quantitative data with an infinite number of attributes. (US Environmental Protection Agency. (2007). Program Evaluation Glossary. Office of the Administrator/ Office of Policy/ Office of Strategic Environmental Management/ Evaluation Support Division.) Continuous quality An approach to quality management that emphasizes the importance of maintenance and sustainability of improvement the quality improvement intervention. Contribution (analysis) An approach for assessing causal questions and inferring causality in real-life program evaluations. It offers a step-by-step approach designed to help arrive at conclusions about the contribution a program has made (or is currently making) to particular outcomes. This approach is designed to reduce uncertainty about the contribution the intervention is making to the observed results through an increased understanding of why the observed results have occurred (or not!) and the roles played by the intervention and other internal and external factors. Additional resources: “Contribution Analysis” (Better Evaluation): http://www.betterevaluation.org/en/plan/approach/contribution_analysis “Contribution Analysis: A new approach to evaluation in international development” (USAID Learning Lab): http://mail.usaidlearninglab.org/sites/default/files/resource/files/mod17_ausaid_fiji_case_article.pdf Countermeasure A measure or action taken against an unwanted action or situation. Cross-tabulation A type of table in a matrix format that displays the (multivariate) frequency distribution of the variables; the tables provide a basic picture of the interrelation between two variables and can help find interactions between them. (Gokhale, D. V.; Kullback, Solomon (1978). The Information in Contingency Tables. Marcel Dekker.) Data Collection Matrix A two-dimensional table that serves as a planning tool to map out anything related to how the data will be captured and analyzed. A basic data collection matrix contains (7) elements: evaluation question, indicators, data source, data collection method, responsible party, timing, and analysis plan. Data Collection Method The method at which the data will be captured. This includes, but not limited to, surveys, focus groups, interviews, observations, etc. Data Source Identifies from whom or what agency the data will come from for the evaluation. Data Visualization Data visualization is a technique that is used to enhance the quality of evaluation reports and communicates data/information using engaging images that resonate with the audience Deductive reasoning A reasoning process in which multiple, general premises are assumed to be true and are then combined to generate a specific conclusion. Delivery Refers to how the method in which results of the evaluation will be disseminated to an external audience. Delphi method Originally developed at the RAND Corporation, an iterative circulation of questions and responses that are progressively refined in light of responses to each round of questions by a group of experts (preferably, participants’ identities should not be revealed to each other); the aim is to reduce the number of viable options or solutions, perhaps to arrive at a consensus judgment on an issue or problem, or a set of issues or problems, without allowing anyone to dominate the process (Brownson, EBPH). Dependent variable Whatever behavior or outcome you are trying to change (e.g., program, intervention) as a result of the presence of the independent variable(s). If you are evaluating a number of different methods or conditions, each method is an independent variable. These variables are called "dependent" because changes in these variables depend on the action of the independent variable (or something else). Adapted from http://ctb.ku.edu/en/table-of-contents/evaluate/evaluate-community- interventions/experimental-design/main Dissemination Process of communicating either the procedures or the lessons learned from a study or program evaluation to relevant audiences in a timely, unbiased, and consistent fashion (Brownson, EBPH). Dissemination Plan A roadmap of how you plan to share your evaluation findings internally and externally. Distal or long-term Changes associated with an intervention’s goals, often measured in terms of morbidity, mortality, quality of outcome life, or related changes (Brownson, EBPH). Distal Outcomes Outcomes that are expected to occur long after implementing the intervention. Related to Long-term outcomes. Document Analysis A form of research that systematically reviews a document, policy brief, public record, etc. and then interprets the data to measure the impact of the file. Downstream Interventions and strategies focus on providing equitable access to care and services to mitigate the negative intervention strategies impacts of disadvantage on health. http://nccdh.ca/glossary/entry/upstream-downstream Ecological framework Model relating individual, interpersonal, organizational, community (including social and economic factors), and health policy factors to individual behavior change and their direct effect on health (Brownson, EBPH). Education Programs aimed at achieving changes in motorist and pedestrian behavior or attitude. Education efforts can also improve the ability of drivers and pedestrians to use and respond to the roadway environment safely and correctly. Emergency Services Emergency response services and programs designed to manage pedestrian injuries after a crash occurred. Encouragement Programs and efforts aimed to promote walking and engage the public in pedestrian safety programs. Enforcement Law enforcement agency efforts to promote compliance with laws, ordinances, and regulations related to pedestrian safety. To teach motorists and pedestrians about safe driving and crossing practices. Engineering Modification to the roadway environment and improve the existing transportation infrastructure and factor in safety when designing new transportation infrastructure. Traffic engineers use road safety audits, street redesign and the use of engineering countermeasures to improve pedestrian safety. Environmental Factors External elements in the environment that may have an influence on risk and protective factors for your target population (i.e., the structure of a road, design of a crosswalk, etc.). Evaluation Assessment of the effectiveness of a pedestrian program by involving procedures that are useful, feasible, ethical, and accurate. Evaluation design The types and sequencing of data collection methods and intervention approaches used to evaluate a policy or program, including experimental, quasi-experimental, and non-experimental studies (Brownson, EBPH). Evaluation Plan A plan and process that attempts to systematically and objectively determine the relevance, effectiveness, and impact of activities in the light of their objectives (Brownson, EBPH). Evaluation Question A high-level question aimed to understand the value, impact, and significance of the intervention. A well- written evaluation question will serve as the guiding framework for your evaluation design and identify will what you want to understand about the intervention. Executive Summary An executive summary is an abbreviated version of a traditional report, meaning it summarize the same sections found in the comprehensive report. Often, executive summaries are written after the full report is completed and excerpts copied and pasted in from the larger report and restricted to create the condensed report. Experimental study Evaluation in which the investigators have full control over the allocation and/or timing of intervention design delivery and evaluation observations; the ability to allocate individuals or groups to intervention or control conditions randomly is a common requirement of an experimental study (Brownson, EBPH). Exposure The state of being exposed to a specific agent or concept. External environment Factors over which you have little or no control may affect your program’s outcomes. These external factors – such as the political and economic circumstances or social influences — can help or hinder an intervention's success. In a logic model, elements of the external environment may also be referred to as "external factors" or "surrounding circumstances." (Innovation Network) External validity Evaluation is externally valid, or generalizable, if it can produce unbiased inferences regarding a target population (beyond the subjects in the study); this aspect of validity is only meaningful with regard to a specified external target population (Brownson, EBPH). Focus Group An interactive discussion between a homogenous sample of six to eight individuals. Focus groups are usually facilitated by a trained moderator who focuses on a specific set of topics to capture social trends and group perspectives. Focus Groups An interactive discussion between a homogenous sample of six to eight individuals. Focus groups are usually facilitated by a trained moderator who focuses on a specific set of topics to capture social trends and group perspectives. Follow-up Activities A section on the dissemination plan that allows you to take notes and track the progress of each report being disseminated. Formative evaluation Type of evaluation conducted in the early stages of an intervention to determine whether an element of a program or policy (e.g., materials, messages) is feasible, appropriate, and meaningful for the target population (Brownson, EBPH). Frequency The count of cases corresponding to the attibutes of an observed variable. (US Environmental Protection Agency. (2007). Program Evaluation Glossary. Office of the Administrator/ Office of Policy/ Office of Strategic Environmental Management/ Evaluation Support Division.) Goal Long-term outcomes partners hope to achieve. Governance The structures, processes, rules and traditions through which decision-making power that determines actions is exercised, and so accountabilities are manifested and actualized (Zadek & Radovich, 2006 https://www.hks.harvard.edu/m-rcbg/CSRI/publications/workingpaper_23_zadek_radovich.pdf) Health disparities Differences in the incidence and prevalence of health conditions and health status between groups, based on race/ethnicity, socioeconomic status, sexual orientation, gender, disability status, geographic location, or some combination of these. Health equity The opportunity for everyone to attain his or her full health potential; no one is disadvantaged from achieving this potential because of his or her social position or other socially determined circumstance (Whitehead M. et al). Health inequities Systematic and unjust distribution of social, economic, and environmental conditions needed for health (Whitehead M. et al). Impact evaluation Assessment of whether intermediate objectives of an intervention have been achieved. Indicators may include changes in knowledge, attitudes, behavior, or risk-factor prevalence (Brownson, EBPH). Implementation fidelity The degree of fit between the developer-defined elements of an intervention and its actual implementation in a given organization or community setting. (Backer, T.E. (2001). Finding the balance: Program fidelity and adaptation in substance abuse prevention. Rockville, MD: SAMHSA.) Independent variable The variable (e.g., program, methods, conditions) that the evaluator wants to evaluate. They are called variables because they can change. They are independent because their existence does not depend on whether something else occurs: you choose them, and they stay consistent throughout the evaluation period. Adapted from http://ctb.ku.edu/en/table-of-contents/evaluate/evaluate-community- interventions/experimental-design/main Indicator An indicator is a measure used to express the behavior of a system or part of a system, including the following characteristics: performance measurement, progress toward goals or objectives, evidence of results achieved, uniform measurement for comparison, and modifiable over time (Flowers J., 2005; Brizius & Campbell, 1991). Indicators Specific qualitative and quantitative data points that are used to operationalize the outcomes or activities of the intervention. It is recommended to have more than one indicator for each outcome or activity being evaluated. Inductive reasoning A logical process in which multiple, specific premises are assumed to be true and are then combined to generate a generalized conclusion. Inferential statistics Statistical analysis using models to confirm relationships among variables of interest or to generalize findings to an overall population (www.cdc.gov/eval/guide/glossary/index.htm). Inputs Resources and processes that support the intervention design, planning, and implementation efforts. An example of inputs that are resources includes staff, target audience, money, space, time, partnership meetings, and technology. Intermediate An intermediate outcome is identified shortly after the intervention has ended. Evaluators will be able to observe some type of change in behavior or norms, which may take some time to see. Intermediate outcomes tend to measure change 3 to 6 months after a participant has completed the intervention. Intermediate Objectives An intermediate outcome is identified shortly after the intervention has ended. Evaluators will be able to observe some type of change in behavior or norms, which may take some time to see. Intermediate outcomes tend to measure change 3 to 6 months after a participant has completed the intervention. Intermediate outcome Changes associated with an intervention’s objectives, often measured in terms of knowledge, attitude, or behavior changes (Brownson, EBPH). Internal validity Degree to which the causal inference drawn from a study is warranted when account is taken of the study methods, the representativeness of the study sample, and the nature of the population from which it is drawn; index and comparison groups are selected and compared in such a manner that the observed differences between them on the dependent variables under study may, apart from sampling error, be attributed only to the hypothesized effect under investigation (Brownson, EBPH). Issue briefs/Fact sheets Written document that states the issue at hand, outlines recommended action steps, and that provides supplemental information to support findings and recommendations. Issue briefs are usually one to two pages and it includes a list of references and the contact information of the author. Key Informant Interviews A semi-structured, one-on-one conversation designed to gain insight on a given topic. The interviewer will guide the interviewee through a discussion of their own life experience, perspectives, and opinions to further understand or create new knowledge about a specific subject Key Messages Major takeaway points from your evaluation findings that should be disseminated to those invested in the outcomes of the intervention. Logic Model A framework used to depict how the interventions are supposed to function and the theory in which it will work. Logic models can include process and outcomes elements, similar to creating SMART objectives. The process component of the model will describe what is needed in the planning phase of the intervention (e.g., resources, program events or strategies, deliverables/ products from the activities). Whereas the outcome elements in a logic model demonstrate the intended effect or goal with respect to a given time period.
Long-Term Long-term outcomes will depict the ultimate goal of the intervention. These outcomes have shown to be sustainable within the priority population and are often measured 6 months to a year after the intervention has been completed. Long-Term Objectives Long-term outcomes will depict the ultimate goal of the intervention. These outcomes have shown to be sustainable within the priority population and are often measured 6 months to a year after the intervention has been completed. Mean A measure of central tendency, the arithmetic average; a statistic used primarily with interval-ratio variables following symmetrical distributions. (US Environmental Protection Agency. (2007). Program Evaluation Glossary. Office of the Administrator/ Office of Policy/ Office of Strategic Environmental Management/ Evaluation Support Division.) Measurable Objective A measurable objective requires a quantifiable activity that resulted in the desired change. It implies that baseline data is required so that result can reflect the positive and/or negative impact of the proposed intervention. Median A measure of central tendency, the value of the case marking the midpoint of an ordered list of values of all cases; a statistic used primarily with ordinal variables and asymmetrically distributed interval-ratio variables. (US Environmental Protection Agency. (2007). Program Evaluation Glossary. Office of the Administrator/ Office of Policy/ Office of Strategic Environmental Management/ Evaluation Support Division.) Mission A statement that captures the enduring focus of your partnership — why your partnership exists and what needs it fulfills in your community (National Institutes of Health, 2002). Mixed method An evaluation approach in which researchers collect, analyze, and integrate both quantitative and qualitative data in a single study to address evaluation questions. Mode A measure of central tendency, the value of a variable that occurs most frequently; a statistic used primarily with nominal variables. (US Environmental Protection Agency. (2007). Program Evaluation Glossary. Office of the Administrator/ Office of Policy/ Office of Strategic Environmental Management/ Evaluation Support Division.) Morbidity Any departure, subjective or objective, from a state of physiological or psychological well-being. In practice, morbidity describes instances of disease, illness, injury, and disability. Mortality A measure of the occurrence of deaths or fatalities in a defined population. Needs assessment Systematic procedure that makes use of epidemiologic, sociodemographic, and qualitative methods to determine the nature and extent of health problems, experienced by a specified population, and their environmental, social, economic, and behavioral determinants (Brownson, EBPH). New Urbanist Promotes the creation and restoration of diverse, walkable, compact, vibrant, mixed-use communities composed of the same components as conventional development, but assembled in a more integrated fashion, in the form of complete communities. These contain housing, work places, shops, entertainment, schools, parks, and civic facilities essential to the daily lives of the residents, all within easy walking distance of each other. (http://www.newurbanism.org/newurbanism.html) Nominal group Structured, small-group process designed to achieve consensus; individuals respond to questions and technique prioritize ideas as they are presented (Brownson, EBPH). Non-parametric Mathematical procedures for statistical hypothesis testing which, unlike parametric statistics, make no statistics assumptions about the probability distributions of the variables being assessed. (Corder, G. W.; Foreman, D. I. (2014). Nonparametric Statistics: A Step-by-Step Approach. Wiley.) Objective Concise time- and action- specific, measurable statements that describe how a goal will be reached. Observations A research method aimed to systematically observe and record interactions, events, locations, etc. between individuals and their environment in a natural state. Observations allow evaluators to describe and understand people’s behavior in context. Observations A research method aimed to systematically observe and record interactions, events, locations, etc. between individuals and their environment in a natural state. Observations provide allow evaluators to describe and understand people’s behavior in context. Outcome evaluation Long-term measure of effects such as changes in morbidity, mortality, and/or quality of life (Brownson, EBPH). Outcome Objective Measures the intended effect of the program on the target population or at the end of the intervention. With a major focus on the intended audience, outcome objectives will determine the success of the intervention. Outcome objectives can be divided into three periods: Short-, Intermediate-, and Long-term. Outputs Found in a logic model, an output describes the tangible items or experiences an individual will encounter if they participated in the intervention at hand. Parametric statistics Data comes from a population that follows a probability distribution based on a fixed set of parameters. (Geisser, S.; Johnson, W.M. (2006) Modes of Parametric Statistical Inference, John Wiley & Sons.) Participatory Collaborative, community-based research method, designed to actively involve community members in approaches research and intervention projects (Brownson, EBPH). Pedestrian Any person on foot, walking, running, jogging, hiking, standing, sitting, lying down, or in a manually or mechanically propelled wheelchair (but not riding in or on a motor vehicle, railway train, streetcar, pedalcycle , animal, animal-drawn vehicle, or other vehicle) on a public road, in the public right of way, or in a parking lot. (homepage) Pedestrian Counts A method used to collect pedestrian data in specific communities or local areas. Pedestrian injury When a pedestrian sustains bodily harm in an unintentional motor vehicle traffic crash with one or more vehicles or pedalcycles. (homepage) Pedestrian safety An aspect of walkability that deals with the level of risk to pedestrians when attempting to walk along or across the network of roads in a community. (homepage) Pedestrian Safety Action A plan developed by community stakeholders intended to improve pedestrian safety in the community. Plans Pedestrian Safety Coordinated efforts designed to improve pedestrian safety by informing a defined population about a specific Education Campaign and pedestrian safety issue(s) targeting knowledge, attitudes, awareness, beliefs, behaviors, and/ or social norms Promotions related to pedestrian safety. These efforts can vary in complexity depending upon a variety of factors, such as duration, resources, and message. (adapted) http://www.cdpr.ca.gov/docs/specproj/h2o/workbook/section1/page5.html Pedestrian safety A pedestrian safety intervention aimed to improve awareness, education, and behaviors of drivers and education campaigns pedestrians in the community. and promotions Pedestrian Safety Activities outlined by a local or state Pedestrian Safety Action Plans or Pedestrian Safety Educational Intervention Campaigns and Promotions aimed to improve pedestrian safety. Pedestrian volume The number of pedestrians that occupy a given space Personal factors Internal elements that have an influence on risk and protective factors for an individual (e.g., education status, awareness of laws, behaviors etc.) Population of interest Group of people with diverse characteristics who are linked by geographical location or setting, social ties, common perspectives, and/ or joint actions (Brownson, EBPH). Primary Data Refers to data that were collected for your own evaluation, meaning data is collected directly from the source. Process evaluation Analysis of inputs and implementation experiences to track changes as a result of a program or policy. This occurs at the earliest stages of public health intervention and often is helpful in determining midcourse corrections (Brownson, EBPH). Process Objective Measures activities that are necessary to deliver the program effectively and efficiently. Process objectives tend to be short-term outcomes by nature and generally evaluates the operational components of implementing an intervention. Product Refers to the type of information that will be presented in the evaluation report and how it will be presented.
Protective Factors Any characteristics of an individual that decreases the likelihood of an adverse event or experiences that may threaten an individual's morbidity, mortality, and quality of life status. Proximal outcomes Outcomes that are expected to occur soon after implementing the intervention. Related to short-term and intermediate outcomes. Qualitative data descriptive, non-numerical data that approximates or characterizes – but does not measure – the attributes, characteristics, and properties of a thing or phenomenon. Qualitative Data Used to examine people’s experiences in detail using a specific set of techniques and tools to guide a participant through a discussion or some form of analysis. Quantitative data numerical data that can be counted (or quantified), verified, and statistically analyzed Quantitative Data Numerical data points that can be counted or quantified. The process of collecting and analyzing the quantitative data is intended to uncover numerical patterns and trends. Quantitative data collection tools (i.e. surveys) are designed with the aim to place data into categories or ranked order. Quasi-experimental Evaluation in which the investigators lack full control over the allocation and/or timing of intervention study design delivery and evaluation observations, but conduct the study as if it were an experiment, allocating subjects to groups; the inability to allocate individuals or groups to intervention or control conditions randomly is a common situation that may be best studied as a quasi-experiment (Brownson, EBPH). Range A measure of spread which gives the distance between the lowest and the highest values in a distribution; a statistic used primarily with interval-ratio variables. (US Environmental Protection Agency. (2007). Program Evaluation Glossary. Office of the Administrator/ Office of Policy/ Office of Strategic Environmental Management/ Evaluation Support Division.) Readiness An organization's or program’s ability to successfully implement an evaluation project or framework. Evaluation readiness has multiple components, including leadership support for evaluation, organizational culture in support of learning and improvement, evaluation skills and expertise, and resources. http://www.pointk.org/client_docs/tear_sheet_core-innovation_network.pdf RE-AIM framework Framework for consistent reporting of research results that takes Account of Reach to the target population; Effectiveness or Efficacy; Adoption by target settings or institutions; Implementation of consistency of delivery of intervention; and Maintenance of intervention effects in individuals and settings over time (Brownson, EBPH). Release Date Refers to the date the evaluation report will be sent out to a specified audience. Relevant Objective A relevant objective relates to the goals and reflects program activities appropriately. The evaluation objective has an overall effect on the desired change. Responsible Party The individual or entity responsible for collecting/ providing the needed data. For example, an external evaluator will be the responsible party that will conduct interviews and collect primary data for the evaluation. Risk Factors Any characteristics of an individual that increases the likelihood of an adverse event or experiences that may threaten an individual's morbidity, mortality status, and quality of life status. Sample A selected subset of a larger group or population (the “universe” population). http://www.oxfordreference.com/search?source=%2F10.1093%2Facref%2F9780195160901.001.0001%2Facr ef-9780195160901&q=sample Sampling Plan A sampling plan is a detailed outline of which measurements will be taken at what times, on which material, in what manner, and by whom. Sampling plans should be designed in such a way that the resulting data will contain a representative sample of the parameters of interest and allow for all questions, as stated in the goals, to be answered. The steps include: identify the parameters to be measured, the range of possible values, and the required resolution; design a sampling scheme that details how and when samples will be taken; select sample sizes; design data storage formats; and assign roles and responsibilities. http://www.itl.nist.gov/div898/handbook/ppc/section3/ppc33.htm Secondary Data Refers to data that were collected by someone else, rather than the user. Short-Term Outcome Short-term outcomes are those that demonstrate the immediate impact of an intervention on the target audience. For instance, short-term indicators can be related to changes in an individual’s knowledge, attitude, or skill level related to the intervention. Smart Growth An approach to development that encourages a mix of building types and uses, diverse housing and transportation options, development within existing neighborhoods, and community engagement. (https://smartgrowthamerica.org/our-vision/what-is-smart-growth/) SMART Objective A mnemonic acronym that explains how to create an objective. According to the mnemonic, objectives should be specific, measurable, achievable, relevant, and time-bound. Social determinants of Life-enhancing resources, such as food supply, housing, economic and social relationships, transportation, health education and health care, whose distribution across populations effectively determines length and quality of life (James S, 2002). Specific Objective A specific objective will identify the setting and activity the caused the desired change. Additionally, it will indicate how the change was implemented and clearly demonstrate what was done to facilitate the impact. Standard deviation The standard deviation of a set of numerical measurements (on an “interval scale”). It indicates how closely individual measurements cluster around the mean (www.cdc.gov/eval/guide/glossary/index.htm). Statistical power The likelihood that a study will detect an effect when there is an effect there to be detected (Brownson, EBPH). Strategy A way of describing how you are going to get things done; a good strategy will take into account existing barriers and resources (people, money, power, materials, etc.) and it will stay with the overall vision, mission, and objectives of the intervention. (http://ctb.ku.edu/en/table-of-contents/structure/strategic- planning/develop-strategies/main ) Surveys A method of collecting information from a target population by asking a series of questions with pre- identified responses or space to answer an open-ended question. Theory of change A comprehensive description and illustration of how and why a desired change is expected to happen in a particular context Time Bound Objective Identifies when the objective will be accomplished using a specific and reasonable timeframe. Timing Related to the frequency at which the data will be collected (e.g. daily, quarterly, and annually). Serves as a timeline of data collection for your evaluation. Traditional/comprehensiveTraditional reports tend to be formal and comprehensive in nature and should follow the standard format of reports reporting in your agency. The document provides a detailed summary of the evaluation goals and objectives, methodology, and findings and is illustrated with facts and figures to showcase the data. Traffic Records System The traffic records system inventory includes reliable state-level data sources that can help decision-makers use data to develop and evaluate engineering, enforcement, education, and emergency medical services safety countermeasures. Triangulation The compilation of results from multiple data sources used to provide additional context for data findings. Triangulation A technique used to analyze diverse sources of data (i.e., surveys, interviews, observation, etc.) in order to supplement missing contextual information and identify emerging themes. Upstream intervention Interventions and strategies focus on improving fundamental social and economic structures in order to strategies decrease barriers and improve supports that allow people to achieve their full health potential. http://nccdh.ca/glossary/entry/upstream-downstream Utility-Focused An approach based on the principle that an evaluation should be judged on the usefulness to its intended Approach users. Variable (or Theme) A quantitative or qualitative representation of an attribute of a person, place, thing, or idea. Variance A measure fo the spread of the values in a distribution; the larger the variance, the larger the distance of the individual cases from the group mean. (US Environmental Protection Agency. (2007). Program Evaluation Glossary. Office of the Administrator/ Office of Policy/ Office of Strategic Environmental Management/ Evaluation Support Division.) Vision A statement that captures the desired end state of your partnership and describes future direction and long- term focus (National Institutes of Health, 2002). Evaluation Templates: Section 1 Partner Assessment and Engagement Planning Grid Template
Partner Name Partner Knowledge/ Skills/ Interest or Perspective on Existing or Potential Role in the History or Rationale for (Organization) Abilities/ Resources Evaluation Evaluation Recruitment/ Engagement*
Evaluation Partner Roles and Responsibilities Individual(s) or Title or Role Responsibilities Organization(s) Administrative • Articulate a clear vision for change in the population and how the evaluation will assess the Leadership changes • Oversee the scientific rigor of the design, planning, and implementation of the evaluation • Identify, recruit, and foster regular communication with partners • Establish agreements or principles to maintain a fair and balanced collaborative effort • Serve as liaison to other individuals, organizations, and partnerships (e.g., elected or appointed officials, funding agencies) • Build trust and leverage support for the evaluation in the population of interest through “deep” connections to community leaders and representatives • Set meeting agendas and coordinate and facilitate partner meetings • Manage day-to-day operations of the partnership, including oversight of staff, finances, and operations • Support passionate, committed, culturally competent, and trusted staff • Enforce timelines and due dates • Assure the quality of data collection, analysis, reporting, and dissemination products Financial Management • Develop a budget for the evaluation • Track accounts payable (bills and expenses)
• Track accounts receivable (invoices and revenue) • Produce financial reports for partners and funders as needed Data Collection Data Analysis Dissemination
Data Collection Planning Grid Template
Evaluation Data Collection Method Data Collection Activities Person(s) Responsible Due Date Question 1.
2.
3.
4.
5.
6.
Partners, Communication Purpose(s), Format(s), and Dates*
Person(s) Purpose of Communication Audiences Possible Formats Due Date Notes Responsible Decide on evaluation design/ Partners Meeting agenda activities (consensus building) Upcoming evaluation activities Population of interest Advertisements (recruit) Progress of the evaluation Funders Progress report (report) Present initial/interim findings Policy- and decision- Policy brief (influence) makers Present complete/final All Infographics, Summary findings (recommend) report
Gantt Chart and Timeline of Evaluation Activities Template
Month Month Month Month Month Month Month Month Month Month Month Month Evaluation Activities Year Year Year Year Year Year Year Year Year Year Year Year Evaluation Planning Team Meetings X X Task X X Task X X Task X X Partner Engagement Team Meetings Task Task Task Data Collection Team Meetings Task Task Task Data Analysis Team Meetings Task Task Task Dissemination Team Meetings Task Task Task
Evaluation Partner Planning Grid Template Individual(s) or Title or Role Responsibilities Organization(s) Administrative • Articulate a clear vision for change in the population and how the evaluation will assess the Leadership changes • Oversee the scientific rigor of the design, planning, and implementation of the evaluation • Identify, recruit, and foster regular communication with partners • Establish agreements or principles to maintain a fair and balanced collaborative effort • Serve as liaison to other individuals, organizations, and partnerships (e.g., elected or appointed officials, funding agencies) • Build trust and leverage support for the evaluation in the population of interest through “deep” connections to community leaders and representatives • Set meeting agendas and coordinate and facilitate partner meetings • Manage day-to-day operations of the partnership, including oversight of staff, finances, and operations • Support passionate, committed, culturally competent, and trusted staff • Enforce timelines and due dates • Assure the quality of data collection, analysis, reporting, and dissemination products Financial Management • Develop a budget for the evaluation • Track accounts payable (bills and expenses)
• Track accounts receivable (invoices and revenue) • Produce financial reports for partners and funders as needed Data Collection Data Analysis Dissemination
Evaluation Templates: Section 2 SMART Objectives Template
When it comes time to define the goals and objectives of your CoP, this template will walk you through the process of developing specific, measurable, achievable, realistic, and time-based objectives. You may want to use the SWOT Analysis Template to better understand your domain and help you develop your CoP’s objectives.
Once you have created the goals of your CoP, it is time to think about objectives and activities needed to accomplish these goals. Here’s an example:
EXAMPLE 1: Increase the number of people in public health informatics who are from minority groups or vulnerable populations.
Not-so-SMART objective 1a: Recruit from historically Black colleges and other minority institutions. Key Component Objective Specific - What is the Inform minority students about the field of public health informatics specific task? and recruit them to join as professionals. Measurable - What are Number of minority institutions where recruitment activities are the standards or implemented; Number of new minority individuals who attend the parameters? Public Health Informatics Network (PHIN) conference Achievable - Is the task Yes, when connected to the training initiatives of national partner feasible? organizations. Realistic - Are sufficient Not at the local or state level (no time or resources to commit to this resources available? activity). Involve national organizations. Time-Bound - What are One year – between the 2008 and 2009 PHIN conferences the start and end dates?
SMART objective 1a: From August 2008-2009, establish recruitment initiatives at historically Black colleges and other minority institutions in conjunction with the training initiatives of national partner organizations (e.g., NACCHO, ASTHO).
You can use the following tables to help your CoP develop SMART objectives that will ensure you reach the goals set forth in your charter.
GOAL 1: Not-so-SMART objective 1a: Key Component Objective Specific - What is the specific task? Measurable - What are the standards or parameters? Achievable - Is the task feasible? Realistic - Are sufficient resources available? Time-Bound - What are the start and end dates?
SMART objective 1a:
For [Domain] Community of Practice use. 1/4 Not-so-SMART objective 1b: Key Component Objective Specific - What is the specific task? Measurable - What are the standards or parameters? Achievable - Is the task feasible? Realistic - Are sufficient resources available? Time-Bound - What are the start and end dates?
SMART objective 1b:
Not-so-SMART objective 1c:
Key Component Objective Specific - What is the specific task? Measurable - What are the standards or parameters? Achievable - Is the task feasible? Realistic - Are sufficient resources available? Time-Bound - What are the start and end dates?
SMART objective 1c:
GOAL 2: Not-so-SMART objective 2a:
Key Component Objective Specific - What is the specific task? Measurable - What are the standards or parameters? Achievable - Is the task feasible? Realistic - Are sufficient resources available? Time-Bound - What are the start and end dates?
SMART objective 2a:
For [Domain] Community of Practice use. 2/4 Not-so-SMART objective 2b:
Key Component Objective Specific - What is the specific task? Measurable - What are the standards or parameters? Achievable - Is the task feasible? Realistic - Are sufficient resources available? Time-Bound - What are the start and end dates?
SMART objective 2b:
Not-so-SMART objective 2c:
Key Component Objective Specific - What is the specific task? Measurable - What are the standards or parameters? Achievable - Is the task feasible? Realistic - Are sufficient resources available? Time-Bound - What are the start and end dates?
SMART objective 2c:
GOAL 3: Not-so-SMART objective 3a:
Key Component Objective Specific - What is the specific task? Measurable - What are the standards or parameters? Achievable - Is the task feasible? Realistic - Are sufficient resources available? Time-Bound - What are the start and end dates?
SMART objective 3a:
For [Domain] Community of Practice use. 3/4 Not-so-SMART objective 3b:
Key Component Objective Specific - What is the specific task? Measurable - What are the standards or parameters? Achievable - Is the task feasible? Realistic - Are sufficient resources available? Time-Bound - What are the start and end dates?
SMART objective 3b:
Not-so-SMART objective 3c: Key Component Objective Specific - What is the specific task? Measurable - What are the standards or parameters? Achievable - Is the task feasible? Realistic - Are sufficient resources available? Time-Bound - What are the start and end dates?
SMART objective 3c:
For [Domain] Community of Practice use. 4/4 Evaluation Templates: Section 3 WORKSHEET: Criteria to Use When Prioritizing the Evaluation Questions
List your evaluation questions in this column. Criteria: 1 = Important to partners 2 = Reflects purpose, goals and objectives of initiative 3 = Reflects key elements in logic model of initiative 4 = Answers will lead to actionable improvements 5 = Can be answered with available resources
Check if evaluation question meets each criteria. Evaluation Questions 1 2 3 4 5 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Adapted from: Salabarría-Peña, Y, Apt, B.S., Walsh, C.M. Practical Use of Program Evaluation among Sexually Transmitted Disease (STD) Programs, Atlanta (GA): Centers for Disease Control and Prevention; 2007. Retrieved from: https://www.cdc.gov/std/program/pupestd/Step3_0215.pdf Evaluation Templates: Section 4 Data Collection Planning Matrix
by Lori Wingate | June 2015
This material is based upon work supported by the National Science Foundation under grant number 1204683. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author and do not necessarily reflect the views of NSF.
An evaluation plan needs to describe what data will be collected, from what sources, and how, by whom, and when, as well as how the data will be analyzed. Placing this information in a matrix format helps ensure that there is a viable plan for collecting all the data necessary to answer each evaluation question and that all data collected will serve a specific, intended purpose. The table below may be copied into another document, such as a grant proposal, and edited/ expanded as needed.
Evaluation Question: Indicator Data Source Data Collection Responsible Timing Analysis Plan Method Party
DEFINITIONS Evaluation Questions are overarching questions about a project’s merit, worth, or significance. The number of evaluation questions depends on the scope and purpose of the evaluation; 3 to 7 questions is typical. Questions should address both project implementation and outcomes. Indicators are specific data points about an aspect of a project—basically, what will be measured in order to answer the evaluation questions. It is useful to use multiple indicators to address an evaluation question, including qualitative and quantitative data. Data Source identifies from whom or what entity data will be collected. Typical data sources for ATE evaluations include project personnel, students, graduates, faculty, project partners, business and industry representatives, institutional records, Web usage statistics, and teaching and learning artifacts. Data Collection Method is the means by which information will be gathered, such as surveys, focus groups, interviews, observations, and institutional database queries. Responsible Party is the individual or organization that will be tasked with collecting the needed information. In many cases, data collection requires cooperation among multiple entities. For example, an external evaluator may be responsible for an administering a survey, but a member of the project staff may need to supply the contact information. Timing identifies when and how frequently data will be collected (e.g., at events, quarterly, annually). It is important to identify approximately when data collection will take place to ensure the information will be obtained when needed for reporting purposes and decision making and that the data collection schedule is conducive to other things taking place in project’s context (e.g., other major data collection activities, semester schedules). Analysis Plan is the approach to be used for making sense of the data, basically how the information will be transformed to reach conclusions in relation to the evaluation questions.
evalu-ate.org | (269) 387-5895 | Western Michigan University
Evaluation Templates: Section 5 Data Analysis Planning Grid Template
Evaluation Person(s) Analysis to Be Performed Data to Be Analyzed and Summarized Due Date Question Responsible 1.
2.
3.
4.
5.
6.
Evaluation Templates: Section 6 Dissemination Planning Grid Template
Person(s) Purpose of Communication Audiences Possible Formats Due Date Notes Responsible Decide on evaluation design/ Partners Meeting agenda activities (consensus building) Upcoming evaluation activities Population of interest Advertisements (recruit) Progress of the evaluation Funders Progress report (report) Present initial/interim findings Policy- and decision- Policy brief (influence) makers Present complete/final All Infographics, Summary findings (recommend) report Tools and Resources: Homepage EVALUATION GUIDE FOR PEDESTRIAN SAFETY CHECKLIST: SUMMARY OF EVALUATION STEPS
SECTION 1: EVALUATION PURPOSE AND PARTNERS SECTION 4: METHODOLOGY AND DATA COLLECTION
☐ Step 1: Convene partners to plan the evaluation ☐ Step 1: Determine, collect, and use data sources that can answer your evaluation questions ☐ Step 2: Identify your population of interest and your vision for ☐ Step 2: Create a matrix to connect your evaluation questions, change indicators, and data sources ☐ Step 3: Consider the intervention stage and corresponding types SECTION 5: DATA ANALYSIS of evaluation ☐ Step 4: Determine evaluation audiences and the relevance of the ☐ Step 1: Convene data analysis partners evaluation to these audiences ☐ Step 5: Create a clear, concise purpose statement ☐ Step 2: Identify variables and themes
☐ Step 6: Design your collaborative, including a governance ☐ Step 3: Organize data and ensure data quality structure and process, partner responsibilities, and budget and time constraints SECTION 2: DESCRIBING THE INTERVENTION ☐ Step 4: Agree on data analysis methods
☐ Step 1: Create a Description of the Pedestrian Safety Interventions ☐ Step 5: Interpret results
☐ Step 2: Create a Logic Model ☐ Step 6: Create a data analysis plan SECTION 3: EVALUATION DESIGN SECTION 6: REPORTING AND DISSEMINATION
☐ Step 1: Develop evaluation questions aligned with the evaluation ☐ Step 1: Determine reporting needs based on various audiences purpose statement ☐ Step 2: Identify the population or subpopulations, intervention ☐ Step 2: Select and create appropriate evaluation reports goals and objectives, and types of evaluation for each evaluation question ☐ Step 3: Specify the evaluation design to maximize causal inference ☐ Step 3: Disseminate Evaluation Findings and internal validity for each evaluation question ☐ Step 4: Determine sampling strategies to maximize external ☐ Step 4: Monitor Dissemination Efforts validity in the selected population or subpopulations for each evaluation question
Strategies for Pedestrian Safety by Domain
Strategy Domain Collaboration To partner and organize Identify potential partners to join a partnership. to create change. Establish, define, and modify the organizational structure and function of a partnership or coalition. Establish or modify the type of leadership structure and/or governing process of a partnership, and also place specific leaders or champions in positions to help move the initiative forward. Obtain physical space, equipment, and supplies for individual partners or organizations. Gain commitment from organizations and residents to undertake actions to achieve a set of objectives. Obtain financial and non-financial resources for partners to carry out specific intervention strategies or projects.
Establish the norms, standards, rules, and procedures that a partnership and its members will follow. Hire individuals to take on specific roles in the partnership.
Initiate a planning process that includes identifying what the partnership wants to achieve, how it will achieve this through various strategies, and prioritizing these strategies.
Provide training and other types of technical assistance to increase the general knowledge, skills, and abilities of individual partners or partnership members to engage in partnership and intervention activities. Increase the capacity of partners to engage, mobilize, and empower residents and organizations to participate in civic activities, support pedestrian safety agendas, and perceive greater ownership of their localities, regions, or states.
Provide training and other types of technical assistance to increase the knowledge, skills, and abilities of leaders or champions to engage in partnership and intervention activities. Promote partnership/ collaboration through newsletters, press releases, websites, events, briefs, and marketing.
Advocacy To generate political will Increase decision-maker engagment and support (elected and community officials, appointed officials, community leaders) through support. presentations to city/ county/ state councils/ committees, adopting principles for practice among appointed officials, etc. Enhance community (individuals and organizations) outreach and engagement in active transportation and pedestrian safety projects through visioning, concept plans, design charettes, hosting workshops/ summits/ symposia, etc.
Encouragement To design campaigns, Design and implement social marketing campaigns host events, or provide (systematic application of marketing, along with other information to increase concepts and techniques, to achieve specific behavioral goals knowledge and for a social good; e.g., Take A Walk, Feet First, Travel Smart). awareness of pedestrian safety initiatives. Gain media coverage (radio, television, print). Take advantage of social media (blog, Facebook, Twitter).
Create billboards or environmental art. Host Walk to School Day/ Smart Commute Day/ Healthy Transportation Day or related events. Recruit residents and organizations to participate in a Smart Commute Challenge/ Car Free Challenge. Design events supporting safe places for active transportation and pedestrian safety (e.g., Sunday Parkways). Install point-of-decision prompts for active transportation and safety in the vicinity of schools, child care agencies, parks and recreation facilities, and worksites.
Convene networks and champions to generate support (e.g., Block Captain Network, Passport Neighborhood Coaches, Loud Voices Youth Group, Neighborhood Builders).
Provide walking tours of historical landmarks, unique neighborhoods, or destinations to increase awareness of places to walk. Provide recognition or awards to supporters (e.g., Active Business Award Luncheon). Create resource guides, toolkits, and manuals (e.g., Active Business Toolkit). Design pedestrian maps or transit guides. Offer materials or incentives for distribution (e.g., apparel, videos, calendars, banners, pedometers). Give business incentives to customers for active transportation (e.g., 25% discount).
Share newsletters or e-newsletters about pedestrian safety partnerships and projects. Provide brochures, flyers, and posters about pedestrian safety partnerships and projects. Education To increase educational Create community-based walking programs or clubs to outreach to localities, increase support and safety (Foot energy program/ Passport regions, and states. program/ Prescription program/ Walk-to-shop program/ Faithfully Fit program (non-denominational faith-based)/ Wheel It, Walk It program/ 10,000 Steps program/ Safe and Active Living United Districts (SALUD))
Create school-based walking programs or clubs to increase support and safety (Safe Routes to School program/ Walking School Bus program) Create worksite-based walking programs or clubs to increase support and safety (Fitness buddy program/ lunch and learn focused on walking) Create training programs to increase pedestrian safety (Walk safety training program/ Safety Walks program) Include pedestrian safety in other health or wellness programs (Healthy Mind, Healthy Body program) Land Use & Planning To design and develop School, child care agency, park or recreation facility, and communities worksite site selection, design, and construction standards (permission, protection, (e.g. require less acreage, maintenance and rehab, limit and use of land). vehicular access) New development/ redevelopment plans, funds, and maintenance to support active transportation in and around schools, child care agencies, parks and recreation facilities, and worksites Comprehensive/ general policy or plan incorporating active transportation or pedestrian safety provisions (i.e. land use plan) Land use/zoning regulation/policy to support urban growth boundaries or containment policies (policies that constrain urban sprawl and create incentives for development in the central portions of urban areas) Land use/zoning regulation/policy to support urban service boundaries (delineates the area beyond which certain urban services such as sewer and water will not be provided)
Small area plan incorporating active transportation or pedestrian safety provisions (e.g. neighborhood, corridor)
Sub-division ordinance (law), regulation (rule or restriction), or resolution (non-binding statement) to standardize procedures for land partition, use, sale, and development (e.g., land titles/ lots/ parcels, design of access roads, amount of open space, building orientation specifications, coordinated development patterns).
Policy/practice that supports the development or redevelopment of dense mixed use areas Land use/ zoning regulation/ permit to support creating residential areas with active transportation or pedestrian safety facility or design features Transportation plan incorporating active transportation and pedestrian safety provisions Pedestrian or sidewalk plan (i.e. identifying gaps in service for pedestrians and mapping future vision) Land use/ zoning regulation/ policy to support sidewalks
Private agreement to construct or allow construction of active transportation or pedestrian safety facility or design feature Engineering To design sites and Improvements to sidewalks, bike lanes or streets for traffic streets, intersections, calming on routes to schools, child care agencies, parks and sidewalks/crosswalks/si recreation facilities, and worksites gnage, or timed traffic Development of physical features to create or enhance safe lights. activity in schools, child care agencies, parks and recreation facilities, and worksites (e.g., placement of street trees, increased lighting, graffiti removal) Policy/practice to support roadway design standards reinforcing active transportation and pedestrian safety Policy/practice for street design supporting active transportation and pedestrian safety (e.g., Complete Streets)
New development/ redevelopment/ maintenance of public transit (e.g. bus, light rail, bike racks on buses, street car, trolley) New development/ redevelopment/ maintenance of pedestrian infrastructure (e.g., sidewalks, crosswalks) New development/ redevelopment/ maintenance of traffic calming measures (e.g. speed bumps, crossing islands, speed tables, traffic signals, on-street parking) New development/ redevelopment/ maintenance of streetscape feature (e.g. lighting, artwork, planter, landscaping, transit shelter) Pedestrian safety support signage (e.g. pedestrian crossing signal, crosswalk) Enforcement To develop active Policy/practice to enable safe routes to schools, child care transportation and agencies, parks and recreation facilities, and worksites pedestrian safety policies and enhance Policy to increase law enforcement to reduce traffic local law enforcement violations of these policies (speed Policy/practice that levies financial charges or that provides limits, yielding to tax exemptions directly or indirectly on services or products pedestrians in as incentives for active transportation or pedestrian safety crosswalks, and proper (e.g., tolls on roads paid by drivers for road maintenance and walking behaviors) and construction) community or site Capital improvement project approval by elected or enforcement of active appointed officials and allocation of funding to support transportation and implementation pedestrian safety. Policy/practice that establishes the formation of a formally- constituted body of people that meets regularly to make decisions regarding active transportation or pedestrian safety (e.g., Active Living Committee, Active Transportation Council, Pedestrian Advisory Group, Employee Wellness Committee) New local government positions (e.g., Bike/Ped Coordinator, Balance Transportation Manager) Policy to require security staff at schools, child care agencies, parks and recreation facilities, and worksites to reduce crime violations Policy/funding to support public transportation systems
Policy/funding to support active transportation Policy to restrict or minimize auto use (e.g., parking costs)
Policy disincentives for parking (e.g. require fewer spaces)
Transit incentive program (e.g. commuter choice, fare free transit policy, discount passes, van pools, park and ride lots)
Policy requiring community involvement in transportation planning (e.g. charettes, forums) Policy/practice or funding for pedestrian projects (e.g. municipal, state, federal enhancement funds, local bond measures) Policy to review design concepts or plans for new or redevelopment projects before implementation to ensure compliance with existing active transportation or pedestrian safety policies or guidelines Policy/practice that establishes the formation and/or support of an organized group of residents committed to increasing public safety and preventing crime within a community Policy/practice that establishes a process for interaction between the community and law enforcement to promote trust and help to increase safety and prevent crime (e.g., residents identify suspects, residents bring problems to the attention of the police)
Government contracts/formal agreements that integrate active transportation or pedestrian safety principles into the formal organization of a governmental body (e.g., Charter)
Government permits authority to local cities and counties to pass a local resolution or ordinance to support active transportation or pedestrian safety Policy/practice that supports development, funding, or maintenance of active transportation and pedestrian safety facilities and design features in affordable housing/ housing authority locations Tools and Resources: Section 4 Pedestrian Safety Evaluation: Quantitative Data Collection Methods
A. Primary quantitative data sources Pedestrian Counts The pedestrian count is a method used to collect pedestrian data in specific communities or local areas and are often collected by the Department of Transportation as part of the Roadway Safety Audit Reviews (RSARs)1. It can be collected manually or by automatically using advanced technology like infrared sensors and street video cameras. This data collection method provides information like pedestrian volume and behavioral data can provide insights into specific crash causes and potential countermeasures.
The information collected from pedestrian counts is used to: − Identify changes in pedestrian behaviors and trends, − Determine the effect facility construction has on levels and behavior of walking, − Describe the demographic characteristics of non-motorized transportation users, and − Provide insight on why pedestrian crashes are occurring and identify solutions for facility improvements.
Advantages/Strengths Challenges/Limitations − Provides location-specific data on pedestrian volume. − manual counts and video recording is labor-intensive and − Allows calculation of risk per pedestrian. expensive − Data from a few observation periods can be scaled up for daily, − lack of consistency that prevents comparisons weekly, monthly, or annual estimates.
Surveys Surveys are important evaluation tools that are used to gather the knowledge, attitudes, opinions, experiences, and behaviors of respondents. Surveys can be used to determine the degree to which the short-term outcomes (or more proximal objectives) of the PSAP interventions are accomplished. Surveys are a great data collection tool to consider for pedestrian education programs and to determine the quality of the walking environment and unmet pedestrian needs, fears, or other concerns
Advantages/Strengths Challenges/Limitations − Easy to compare and analyze − Possible response bias; wording can bias participants’ responses − Administer to any size sample − Possible sampling bias − Can collect a lot of data at once − Sometimes difficult to get a high response rate − Participant anonymity − Sample surveys/ questionnaires already exist − Inexpensive
1 Roadway Safety Audit Reviews (RSARs) involve the use of a multi-disciplinary team approach to review and evaluate a location, corridor, or area after it is built or before it is open to the public.
Survey Tool Description Community Pedestrian Behavior • Parents reported children’s weekly walking habits (e.g., to school, to friends’ houses, to walk dog) Questionnaire (parent)2 Pedestrian Safety • Children’s knowledge of pedestrian safety, which include route selection, proper looking before and while Knowledge (child/ oral crossing, and how to walk safely on streets without sidewalks. instrument)3 Neighborhood Environment • Neighborhood disorder subscale that assesses the frequency of events in neighborhoods such as: loitering for Children Rating Scales4 adults, gang activity, drunks or drug dealers hanging around, and disorderly or misbehaving groups of youths or adults. • Other subscales: neighborhood quality, neighborhood interaction, facility availability, fear of retaliation, stop misbehavior, assist. Neighborhood Environment • Measures perceived neighborhood safety, pedestrian infrastructure, and aesthetics. Walkability Survey (NEWS)5 Children's Leisure Activities • Two identical questionnaires were developed, one for parents (proxy report) and one for 10-to 12-year-old Study Survey (CLASS)6 children (self-report). • Proxy Report: Parents were asked to report the frequency of the activity and the total time their child spent in that activity. • Self-Report: A physical activity measure completed by the child. St. Louis Environment and • A detailed assessment of walking behavior, places to walk, barriers to being physically active, Physical Activity Instrument neighborhood infrastructure for walking and cycling, perceptions about places for walking, social assets, social support for physical activity, community assets, policy attitudes, and sedentary behaviors. Environmental Supports for • Assesses the physical and social environment, including perceptions of the community environment, Physical Activity safety, access to recreation and shopping destinations, and conditions of the neighborhood and facilities. Questionnaire Schools Combined Tool (Home to • National Safe Route to School Survey (1-item) School Journey) • Perceived traffic safety concerns – home, route, school (9-items) • Perceived personal safety concerns – home, route, school (10-items) Physical Activity School • PASS is a free, user-friendly, web-based, 8-item tool that assesses and increases awareness of evidence- Score (PASS) based physical activity practices at elementary schools
2 Stavrinos D, Byington KW, Schwebel DC. The effect of cell phone distraction on pediatric pedestrian injury risk. Pediatrics. 2009;123:e179–185 3 Schwebel, D. C., & McClure, L. A. (2014). Training Children in Pedestrian Safety: Distinguishing Gains in Knowledge from Gains in Safe Behavior. The Journal of Primary Prevention, 35(3), 151–162. http://doi.org/10.1007/s10935-014-0341-8 4 Coulton, C., Korbin, J., & Su, M. (1996). Measuring neighborhood context for young children in an urban area. American Journal of Community Psychology, 24, 5–32. 5 Saelens B.E., Sallis J.F., Black J.B., Chen D. Neighborhood-based differences in physical activity: An environment scale evaluation. Am. J. Public Health. 2003;93:1552–1558. doi: 10.2105/AJPH.93.9.1552 6 A. Telford, J. Salmon, D. Jolley, D. Crawford. Reliability and validity of physical activity questionnaires for children: The Children's Leisure Activities Study Survey (CLASS) Pediatric Exercise Science, 16 (2004), pp. 64–78.
B. Secondary Quantitative Data Sources Existing Data Systems: Existing data sources are usually state-level data sources and are great data tool to measure a baseline on pedestrian crashes and injury. These data sources are part of the traffic records system.
Advantages/Strengths Challenges/Limitations − Inexpensive or free − Available data may lag by one or more years − Usually accurate data without the time-consuming and − May not have data on behavior, knowledge, attitudes, and labor-intensive process of collecting the data themselves opinions − Usually statewide − Usually allows for historical comparisons or trend analysis to compare national, state, or county regions
Traffic Records System The traffic records system inventory includes reliable state-level data sources that can help decision-makers use data to develop and evaluate engineering, enforcement, education, and emergency medical services safety countermeasures. A valuable characteristic of the data sources included in the traffic record system is linkage variables that allow evaluators to integrate the data and conduct analyses that generate another level of insight and understanding that is limited in just any singular data system.
Together this traffic record data system can provide complete information and measures that are used to track pedestrian injury and fatal crashes with motor vehicles and other non-motor vehicle-related pedestrian injuries. The traffic record system includes data sources that can be grouped by: − Crash data − Roadway data − Vehicle data − Driver data − Citation and adjudications data − Injury Surveillance systems
Quantitative Description and Objective Advantages/Strengths Challenges/Limitations Data Collection Method Police Reports Crash, vehicle, and person-specific data about − Describe the pedestrian crash including − Usually, only traffic crashes on public
crashes involving pedestrians reported by Law the use of crosswalks roadways enforcement; City, County, and State police. − No medical outcomes − Information about the pedestrian after the crash occurred − Usually only include fatalities 30 days Crash and and Crash Data Roadway from the crash. Roadway Inventory Describe the infrastructure and roadway features in − Type of signaling and signage − Only information about the roadway and their Inventories, Roadway Safety Audits, and − Roadway features that can provide maybe not knowledge, attitudes, and Reviews information on crash risk factors when opinions of the pedestrians. combined with crash data.
− History of the roadway design Vehicle Registration (Registered motor vehicles that include types, − Good for education for both miles that − Only information on the vehicle
Vehicle configurations, and usage of the vehicle. were driven − May be difficult to access local data Citation/Adjudication The citation that was issued to Pedestrian and − The true number of citations issued not − Sometimes not a statewide system with Drivers for example jaywalking and not yielding to only those involved in a crash. many courts.
pedestrians. Adjudication is the convictions from − Citation codes can be different from
Citation Citation those citations from the courts. jurisdiction to jurisdiction. License Driver Previous driver history involved in pedestrian − Final driver conviction data that − No pedestrians crashes; driver behavioral data; such as speeding in accumulated points including Alcohol
Adjudication school zones; conviction data derived by the points and pedestrian offenses. Driver and & system; driver demographic EMS Patient information and transport times for those − Transport time information − Only initial diagnosis pedestrians transported by Emergency Medical Services (EMS). Pedestrian injuries treated by prehospital EMS agencies. Trauma Registry Collects information on seriously injured pedestrian − Can provide the severity of injury with − Only for pedestrian seen in a trauma patients treated in a trauma hospital and fits the the AIS and ISS score. center and fits the criteria to be included trauma criteria. in the registry. − Lack of detail on circumstances surrounding crash. State Level Emergency Pedestrian incidents severe enough to be admitted − Types of injuries on body regions − Lack of detail on circumstances Department Data to a hospital. Uses ICD-9-CM and ICD-10-CM coding − ED charges surrounding crash. scheme. Information on body regions and injury.
State Level Hospital Pedestrian incidents severe enough to be admitted − Types of injuries on body regions − Potential duplicate counts of individuals Inpatient Discharge Data to a hospital. Uses ICD-9-CM and ICD-10-CM coding − Hospital charges − Lack of detail on circumstances scheme. Information on body regions and injury. − Discharge status of the patient surrounding crash.
Death Certificates State Death Vital Records Uses ICD-9-CM and ICD- − State Vital Statistics are complete, − Information on the location of the crash Surveillance Systems 10-CM coding scheme to determine if the person population-based data sets event is often limited or incomplete. was a pedestrian. − Include both traffic and non-traffic − Information on the circumstances of the Injury pedestrians who died. death can be limited. Here is a list of variables to consider when designing pedestrian safety evaluations.
Data Sources Variables to Consider Crash and Roadway Police Reports Crash location, type of roadway, intersections, crosswalk, total lanes in roadway, Data contributing circumstances of driver, posted speed limit, non-motorist information including circumstances prior to crash and at time of crash, and condition at time of crash Fatality Analysis Reporting System (FARS) See about and they have a pedestrian selection of variables that has prior to crash. Inventories/Roadway Safety Audits and Reviews Type of roadway, number of lanes, type of signals and devices, lighting Vehicle Vehicle Registration Type of vehicle, number of miles within year, county Driver and Citation & Citation/Adjudication Citation issued, location of jurisdiction, result of conviction Adjudication License Driver Driver demographics, type of points on license, types of driving classes taken Injury Surveillance State Level Hospital Inpatient Discharge Data Patient demographics, the source of admission, the area of residence, principal and Systems secondary diagnoses (ICD-9-CM and ICD-10-CM diagnosis codes), procedures, type of discharge, the cause of Pedestrian injury (ICD-9-CM and ICD-10-CM E codes), the source of payment, the length of stay, charges, and hospital type. State Level Emergency Department Data Patient demographics, source of admission, area of residence, principal and secondary diagnoses (ICD-9-CM and ICD-10-CM diagnosis codes), procedures, type of discharge, cause of Pedestrian injury (ICD-9-CM and ICD-10-CM E codes), source of payment, length of stay, and charges, EMS Incident address and location type, times, destination, initial type of injury
Trauma Registry Severity injury score, ISS and or AIS Death Certificates ICD-10-CM diagnosis codes may contain BAC levels
Injury Surveillance Systems uses ICD9/ICD10 codes to indicate a pedestrian inpatient and death data- They are stratified in traffic and non-traffic. ICD-10-CM − Pedestrian, traffic: [V02–V04](.1,.9), V09.2 − Pedestrian, non-traffic: V01, [V02–V04](.0), V05, V06, V09(.0–.1,.3,.9)
ICD-9-CM − Pedestrian, traffic: E810-E819(.7) − Pedestrian, non-traffic: E800-807(.2), E820-E825(.7), E826-E829(.0)
National Data Sources There are also other national databases that provide additional information on pedestrian behavior, exposure, patterns, and trends.
Pedestrian and bicycle secondary data sources Provides pedestrian and bicycle commuting data for census block groups. However, these data only include regular American Community commuters, not those who occasionally walk or bike to work or those who walk or bike for non-work purposes. The Survey (ACS) U.S. Census Bureau surveys a small percentage of the population each year to develop 1-year, 3-year, and 5-year estimates. ACS data are accessible through American FactFinder. National Household Daily travel data is collected for all trips, modes, purposes, trip lengths, and areas of the country. Data are available at Travel Survey (NHTS) the person- and household-level. The NHTS is updated every five to seven years. Fatality Analysis The National Highway Traffic Safety Administration (NHTSA) provides public data regarding fatal injuries suffered in Reporting motor vehicle traffic crashes. Fatality data can be downloaded or queried online. System (FARS) General Estimates These data come from a nationally representative sample of police-reported motor vehicle crashes that resulted in System (GES) property damage, injury, or death. National Survey of National survey data were collected in 2002 to ascertain the scope and magnitude of bicycle and pedestrian activity Bicyclist and and the public’s behavior and attitudes regarding bicycling and walking. The project was jointly sponsored by NHTSA Pedestrian Attitudes and the Bureau of Transportation Statistics. and Behavior CDC’s WISQARS™ (Web-based Injury Statistics Query and Reporting System) is an interactive, online database that WISQARS provides fatal and nonfatal injury, violent death, and cost of injury data. That are national and state-specific with pedestrian breakouts in pedestrian traffic and non-traffic.
Pedestrian Safety Evaluation: Qualitative Data Collection Methods
Focus Group A focus group is an interactive discussion designed to collect in-depth, subjective information on thoughts, insights, beliefs, and perceptions for a particular topic with a pre-selected sample of individuals. This qualitative method is facilitated by a trained moderator who guides the group through a set of questions to capture social trends and norms. Focus groups tend to be between 60-90 minutes long with about six to eight participants. The interactive component of a focus group is unique to this method and is found to be conducive for creating a comfortable environment for participants to share, as well as, uncover a variety of insights on the issue at hand. Advantages Challenges - Able to capture data on social interactions - Requires a skilled moderator and reactions to a specific topic - Less controlled environment - Useful for exploratory, explanatory, and - Comfortability levels vary evaluative research - Participant dynamic can be - Can collect a large range of perspectives in uncontrollable time period - Can be costly - Pending issues can emerge from the data - Data analysis is complex and time- consuming
Key Informant Interview Key Informant Interviews can be used for exploratory, explanatory, policy, and general research activities. Although they are not ideal for measuring social interactions and group perspectives, it is important to note that the primary goal of a key informant interview is to capture personal reactions to a topic and share a testimonial of personal life experiences. Advantages Challenges − Easy to capture personal experiences, life - Difficult to establish rapport with stories, feelings, etc. participant under time constraints − Will provide contextual information to - Conversation-based approach requires a enhance quantitative data Acquire in-depth skilled interviewer information - Data collection period and volume can − An ideal approach to understanding be extensive-- dependent on sample sensitive topics size − Flexible structure that can lead to important concepts/ themes - Pending issues can emerge from the data
Observations An observational evaluation is a research method that collects data through recorded observation and a systematic review of people, places, and things and how they all interact with one another. The most vital component of this method is that all interactions occur organically, so it is the observer’s role to note behavior, norms, social situations, and environmental factors and provide an objective reflection of his or her empirical experience in a given setting. The list below outlines the many uses to collect data through the observation method, these include: - exploring a new topic - capturing a contextual information - describing a specific setting and the social and environmental interactions there - understanding unspoken norms and social cues - triangulating data Advantages Challenges - Provides context to behaviors - Time consuming - Documents unspoken social etiquette and - Data collection can be a challenge in the cultural norms field - A less intrusive qualitative method - Data can be subjective - Provides insight to natural interactions - Will require skilled observers
Case Studies A Case Study is described as a formal or informal tool used to critically assess a documented story or event. This method has been used in public health to identify practical strategies or concepts previously practiced by others and that can be actively applied elsewhere. For example, if you are interested in knowing more about the effects of a 15 mile per hour speed limit 10 miles away from a school district, you may want to review a case study to understand the cost and benefits of the intervention for your site. Because case studies are considered anecdotal evidence, they are often criticized for being considered the less credible source of educational research. However, case studies can be powerful evidence as it can bridge the gap between researchers and practice and can provide supplemental evidence when a more rigid, robust evidence is unavailable.
Advantages Challenges - Contains rich and detailed information that - Information is not generalizable is not usually collected from other methods - Focuses on a single unit or event - More likely to be conducted in rare and - Criticized for lack of rigor and results special cases demonstrating poor reliability and validity - Has the ability to capture complex issues with diverse variable to understand the phenomenon
Document Analysis A document analysis is a social research method used to identify and interpret patterns in data. Some define a document analysis as the opportunity for an evaluator to give the documents a voice by coding concepts and themes that emerge in the text or visual aids. Document analysis is good to do because it gives a historical context to and issue or provides valuable information that may lead to alternative outcomes. A documentation review can include the following sources: - Written documents (public records, white papers, policies, newsletters etc.) - Historical documents (reports, meeting minutes, attendance logs, financial documents, etc.) - Signs and posters - Sound recording - Media Advantages Challenges - Can potentially have a large sample size - Criticized for subjectivity - Inexpensive - Time consuming - Good source of background information and - Information may not be current or historical knowledge reliable
Inventories/Roadway Safety Audits and Reviews Roadway Inventories is a collection of all road-related information that defines and monitors the state- owned highway network, maintaining an inventory of the roadway features, conditions, and characteristics. These describe the signaling and roadway features. This will give existing information about signs and signaling. Road Safety Audits and reviews provide formal information on existing or future road or intersection provided by an independent, multidisciplinary team. It qualitatively estimates and reports on potential road safety issues and identifies opportunities for improvements in safety for all road users.
Advantages Challenges - Contains rich and detailed information on - Time consuming roadway - Collaboration with partners - Pre-data information
Additional Web-based Tools & Resources Additional Web-based Tools & Resources to Support Pedestrian Safety Evaluation Planning and Implementation
Title Description URL Homepage NHTSA National Priority Congress HR 22 Fixing America’s Surface Transportation Act’ or the ‘‘FAST Act’’ https://www.nhtsa.gov/sites/nhtsa.dot.gov/files/fast-act-titleiv-highway- Safety Program (See page 29.) traffic-safety.pdf NHTSA Countermeasures A reference to assist traffic safety and other professionals in selecting effective, https://www.nhtsa.gov/sites/nhtsa.dot.gov/files/812239_countermeasures_ that Work evidence-based 8thed_tt.pdf countermeasures for traffic safety problem areas. FHWA nonmotorized safety Website provides helpful resources to support the implementation of the Safety https://safety.fhwa.dot.gov/hsip/spm/ requirements for State DOTs Performance Management Final Rule that establishes the process for State DOTs and MPOs to establish and report their safety targets, and the process that FHWA will use to assess whether State DOTs have met or made significant progress toward meeting their safety targets. FHWA How to develop a Report presents an overview and framework for state and local agencies to http://safety.fhwa.dot.gov/ped_bike/ped_focus/docs/fhwasa0512.pdf Pedestrian Safety Action Plan develop and implement a Pedestrian Safety Action Plan tailored to their specific problems and needs. FHWA Pedestrian & Bicycle Report documents a comprehensive analysis of pedestrian crash data trends and https://safety.fhwa.dot.gov/ped_bike/pssp/fhwasa10035/ Safety website factors, a detailed review of more than 200 publications on pedestrian safety, and input from more than 25 expert stakeholder members. Section 1 Governance Principles Provides guidance on developing a governance structure and process for https://www.hks.harvard.edu/m- partnerships rcbg/CSRI/publications/workingpaper_23_zadek_radovich.pdf WHO Commission on SDOH Provides background on equity, inequity, and disparity www.who.int/social_determinants/thecommission/en/index.html
Unnatural Causes Provides background on equity, inequity, and disparity www.unnaturalcauses.org/ documentary NACCHO Health Equity and Provides background on equity, inequity, and disparity www.naccho.org/topics/justice/mission.cfm Social Justice Committee RWJ Commission on SDOH Provides background on equity, inequity, and disparity www.rwjf.org/pr/product.jsp?id=41008
Sample Budget Provides guidelines for developing a budget https://www.ncjrs.gov/pdffiles/bdgtdtl0.pdf
Section 2 How to Develop a Pedestrian Presents an overview and framework for state and local agencies to implement a https://safety.fhwa.dot.gov/ped_bike/ped_focus/docs/fhwasa0512.pdf Safety Action Plan Pedestrian Safety Action Plan Advancing Pedestrian and A reference guide for an integrated and comprehensive effort to improve https://www.nhtsa.gov/staticfiles/nti/pdf/812258-Peds_Bike_Primer.pdf Bicyclist Safety: A Primer for pedestrian and bicycle safety. Highway Safety Professions Road Diet (Roadway A technical assistance guide for roadway reconfiguration https://safety.fhwa.dot.gov/road_diets/info_guide/ch3.cfm Reconfiguration) Street Design: Part 1- An article that reviews street polices and discusses how to make the https://www.fhwa.dot.gov/publications/publicroads/10julaug/03.cfm Complete Streets transportation system more accessible for all travelers. Complete Streets An online guide to support planning, designing, and operating of roadways https://www.transportation.gov/mission/health/complete-streets efforts. Complete Streets Polices The National Complete Streets Coalition’s list of complete street polices. https://www.transportation.gov/mission/health/complete-streets-policies Title Description URL Section 2. Understanding Risk A section of a larger online tool aimed to support public health professional http://ctb.ku.edu/en/table-of-contents/analyze/choose-and-adapt- and Protective Factors: Their conduct needs assessment and design health interventions. community-interventions/risk-and-protective-factors/main Use in Selecting Potential Targets and Promising Strategies for Intervention Identifying the Components Worksheet that discusses the components of a logic model https://www.cdc.gov/std/Program/pupestd/Components%20of%20a%20Logi of a Logic Model c%20Model.pdf Introduction to Program A digital PDF that is a self-guide tutorial of how develop a program evaluation. https://www.cdc.gov/eval/guide/cdcevalmanual.pdf Evaluation for Public Health Programs: A Self-Study Guide Section 3 G*Power A tool to compute statistical power analyses for many different t tests, F tests, χ2 http://www.gpower.hhu.de/ tests, z tests and some exact tests. G*Power can also be used to compute effect sizes and to display graphically the results of power analyses. Power and Sample Size .com Free and open source online calculators. http://powerandsamplesize.com/
PS An interactive computer program for performing statistical power and sample http://biostat.mc.vanderbilt.edu/wiki/Main/PowerSampleSize size calculations PowerUp! A tool that provides convenient excel-based functions to determine minimum http://www.causalevaluation.org/ detectable effect size and minimum required sample size for various experimental and quasi-experimental designs. PowerUp!R R package version of PowerUp! and additionally includes functions to determine https://cran.r-project.org/web/packages/PowerUpR/index.html sample size for various multilevel randomized experiments with or without budgetary constraints. R Package pwr A package of tools with functions for basic power calculations using effect sizes https://cran.r-project.org/web/packages/pwr/index.html and notations from Cohen (1988). Russ Lenth's power and Software is intended to be useful in planning statistical studies. https://homepage.divms.uiowa.edu/~rlenth/Power/index.html sample-size page WebPower A free collection of tools for conducting statistical power analysis online. http://webpower.psychstat.org
SampSize (app for Android, An application to assist in the design of clinical trials by calculating the sample https://www.epigenesys.org.uk/portfolio/sampsize/ iOS iPhone, iPad) size using inputs provided by the user. Section 4 Steps in Conducting a A PowerPoint presentation that discusses the components of conducting a mixed http://digitalcommons.unl.edu/cgi/viewcontent.cgi?article=1047&context=d Scholarly Mixed Methods methods study. berspeakers Study CDC Coffee Break: Using A PowerPoint presentation that discusses the components of conducting a mixed https://www.cdc.gov/dhdsp/pubs/docs/cb_july_2012.pdf Mixed Methods in Program methods study. Evaluation Analyzing Quantitative Data Issue brief that discusses how to analyze quantitative data for an evaluation. http://c.ymcdn.com/sites/www.safestates.org/resource/resmgr/evaluation_r for Evaluations esources_webpage/quantitative_data_evaluation.pdf Mixed Methods in Program A PowerPoint presentation on mixed method evaluations https://www.cdc.gov/asthma/program_eval/webinar4b/4b_with_notes.pptx Evaluation Mixed Method Research A website discusses the components of mixed method http://resourcecentre.foodrisc.org/mixed-methods-research_185.html
Title Description URL Overview of Mixed Methods An online module that provides an overview of mixed method research. https://cirt.gcu.edu/research/developmentresources/research_ready/mixed_ methods/overview National Bicycle and Provide detailed guidance on how to conduct a pedestrian and bicycle count. http://bikepeddocumentation.org/application/files/7614/6671/7784/NBPD_I Pedestrian Documentation nstructions_2010.pdf Project Pedestrian and Bicycle Data Provides information on how to collect data for pedestrian safety programs. http://www.pedbikeinfo.org/pdf/PlanDesign_Tools_FHWACaseStudies.pdf Collection in United States Communities: Quantifying Use, Surveying Users, Documenting Facility Extent Program Evaluation Tip A tip sheet that guides users through the process of developing survey questions. https://www.cdc.gov/dhdsp/programs/spha/docs/constructing_survey_quest Sheet: Constructing Survey ions_tip_sheet.pdf Questions Simple, Inexpensive A journal article that reviews potential sampling methodologies for pedestrian https://dash.harvard.edu/bitstream/handle/1/12638506/PABS- Approach to Sampling for and bicycle surveys. approach.pdf?sequence=1 Pedestrian and Bicycle Surveys Consensus A comprehensive online report that provides information on pedestrian safety http://c.ymcdn.com/sites/www.safestates.org/resource/resmgr/ISW8_Repor Recommendations for surveillance. t_Final.pdf Pedestrian Injury Surveillance: Report from the Injury Surveillance Workgroup 8 (ISW8) Steps for Conducting Focus Provides information on how to conduct an In-depth Interview or a Focus Group http://www.orau.gov/cdcynergy/soc2web/content/activeinformation/resour Groups or Individual In-depth Discussion ces/soc_focusgroup-indepthinterview_steps.pdf Interviews Guidelines for Writing Case A step-by-step guide on how to write a case study. https://awc.ashford.edu/tocw-guidelines-for-writing-a-case-study.html Studies How to Analyze a Case Study An online tool that provides guidance on how to analyze a case study http://wps.prenhall.com/bp_laudon_essmis_https://awc.ashford.edu/tocw- guidelines-for-writing-a-case- study.html6/21/5555/1422312.cw/content/index.html HHS Informed Consent FAQs Provides information on the guidelines for informed consent. https://www.hhs.gov/ohrp/regulations-and-policy/guidance/faq/informed- consent/ Section 5 The Good Indicators Guide: Provides general guidance on indicators https://www.k4health.org/sites/default/files/TheGoodIndicatorsGuide.pdf Understanding how to use and choose indicators Choosing the Correct Tool to determine the appropriate statistical test for variable types. http://stats.idre.ucla.edu/other/mult-pkg/whatstat/ Statistical Test
Section 6 Evaluation Report Checklist A comprehensive checklist of what should be included in an evaluation report. https://wmich.edu/sites/default/files/attachments/u350/2014/evaluation- reports.pdf Data Collection Planning Provides detailed information on how to construct a data matrix http://www.evalu-ate.org/resources/tool-datamatrix/