RAND Suicide Prevention Program Evaluation TOOLKIT

CHILDREN AND FAMILIES The RAND Corporation is a nonprofit institution that helps improve policy and EDUCATION AND THE ARTS decisionmaking through research and analysis. ENERGY AND ENVIRONMENT

HEALTH AND HEALTH CARE This electronic document was made available from www.rand.org as a public service

INFRASTRUCTURE AND of the RAND Corporation. TRANSPORTATION

INTERNATIONAL AFFAIRS LAW AND BUSINESS Skip all front matter: Jump to Page 16 NATIONAL SECURITY

POPULATION AND AGING

PUBLIC SAFETY Support RAND

SCIENCE AND TECHNOLOGY Purchase this document

TERRORISM AND Browse Reports & Bookstore HOMELAND SECURITY Make a charitable contribution

For More Information Visit RAND at www.rand.org Explore the RAND National Defense Research Institute View document details

Limited Electronic Distribution Rights This document and trademark(s) contained herein are protected by law as indicated in a notice appearing later in this work. This electronic representation of RAND intellectual property is provided for non- commercial use only. Unauthorized posting of RAND electronic documents to a non-RAND website is prohibited. RAND electronic documents are protected under copyright law. Permission is required from RAND to reproduce, or reuse in another form, any of our research documents for commercial use. For information on reprint and linking permissions, please see RAND Permissions. This report is part of the RAND Corporation tool series. RAND tools may include models, databases, calculators, computer code, GIS mapping tools, practitioner guidelines, web applications, and various toolkits. All RAND tools undergo rigorous peer review to ensure both high data standards and appropriate methodology in keeping with RAND’s commitment to quality and objectivity. RAND Suicide Prevention Program Evaluation TOOLKIT

Joie D. Acosta, Rajeev Ramchand, Amariah Becker, Alexandria Felton,

C O R P O R A T I O N Aaron Kofner

NATIONAL DEFENSE RESEARCH INSTITUTE

RAND Suicide Prevention Program Evaluation TOOLKIT

Joie D. Acosta, Rajeev Ramchand, Amariah Becker, Alexandria Felton, Aaron Kofner

Prepared for the Office of the Secretary of Defense Approved for public release; distribution unlimited The research described in this report was prepared for the Office of the Secretary of Defense (OSD). The research was conducted within the RAND National Defense Research Institute, a federally funded research and development center sponsored by OSD, the Joint Staff, the Unified Combatant Commands, the Navy, the Marine Corps, the defense agencies, and the defense Intelligence Community under Contract W74V8H-06-C-0002.

Library of Congress Cataloging-in-Publication Data is available for this publication.

ISBN: 978-0-8330-8072-1

The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND’s publications do not necessarily reflect the opinions of its research clients and sponsors. R® is a registered trademark.

Cover images: iStockphoto/Thinkstock

© Copyright 2013 RAND Corporation Permission is given to duplicate this document for personal use only, as long as it is unaltered and complete. Copies may not be duplicated for commercial purposes. Unauthorized posting of RAND documents to a non-RAND website is prohibited. RAND documents are protected under copyright law. For information on reprint and linking permissions, please visit the RAND permissions page (http://www.rand.org/publications/ permissions.html).

Published 2013 by the RAND Corporation 1776 Main Street, P.O. Box 2138, Santa Monica, CA 90407-2138 1200 South Hayes Street, Arlington, VA 22202-5050 4570 Fifth Avenue, Suite 600, Pittsburgh, PA 15213-2665 RAND URL: http://www.rand.org To order RAND documents or to obtain additional information, contact Distribution Services: Telephone: (310) 451-7002; Fax: (310) 451-6915; Email: [email protected] RAND Suicide Prevention Program Evaluation Toolkit

Preface

Evaluations are critical for assessing the impact of U.S. Department of Defense (DoD) investments in suicide prevention and can be used as the basis for decisions about whether to sustain or scale up existing efforts. The Defense Centers of Excellence for Psychological Health and Traumatic Brain Injury asked the RAND Corporation to draw from the scientific literature and create a toolkit to guide future evaluations of DoD-sponsored suicide prevention programs (SPPs). The overall goal of the toolkit is to help those responsible for SPPs determine whether their programs produce beneficial effects and, ultimately, to guide the responsible allocation of scarce resources. The toolkit is accompanied by a companion document (available at http://www.rand. org/pubs/research_reports/RR283.html) that provides additional background for those who are interested in learning about the toolkit’s development. To develop the toolkit, RAND researchers examined the peer-reviewed literature and other evaluation toolkits and elicited feedback from program staff responsible for implementing DoD-sponsored SPPs. The contents of this toolkit will be of particular interest to DoD SPP managers and suicide prevention coordinators and should also be useful to policymakers in other sectors who sponsor or manage SPPs more generally. This research was sponsored by the Office of the Assistant Secretary of Defense for Health Affairs and the Defense Centers of Excellence for Psychological Health and Traumatic Brain Injury and conducted within the Forces and Resources Policy Center of the National Defense Research Institute, a federally funded research and development center sponsored by the Office of the Secretary of Defense, the Joint Staff, the Unified Combatant Commands, the Navy, the Marine Corps, the defense agencies, and the defense Intelligence Community. For more information on the Forces and Resources Policy Center, see http://www.rand. org/nsrd/ndri/centers/frp.html or contact the director (contact information is on the web page).

iii

RAND Suicide Prevention Program Evaluation Toolkit

Contents

Preface...... iii Tools...... vii Figures...... ix

Chapter One

Introduction and Overview...... 1

Intended Audience...... 1 Toolkit Goals and Specific Aims...... 1 Evaluation Challenges Addressed...... 2 How the Toolkit Was Developed...... 2 User’s Guide...... 4 Start Using the Toolkit...... 5 Summary...... 7

Chapter Two

Identify Your Program’s Core Components and Build a Logic Model...... 9

Specify Core Components...... 9 Use Core Components to Build a Logic Model...... 38 Assess the Quality of Your Program’s Logic Model...... 41 Summary...... 46

Chapter Three

Design an Evaluation for Your Program...... 47

Identify Your Program Type...... 47 Learn How Similar Programs Have Been Evaluated...... 50 Learn the Types of Evaluation Designs...... 51 Identify Issues Pertinent to Your Evaluation Design...... 52 Select an Evaluation Design...... 59 Assess the Quality of the Design...... 61 Summary...... 61

v RAND Suicide Prevention Program Evaluation Toolkit

Chapter Four

Select Evaluation Measures for Your Program...... 63

Select Process Evaluation Measures...... 63 Select Outcome Evaluation Measures...... 68 Assess the Quality of Selected Measures...... 73 Summary...... 74

Chapter Five

Analyze Your Program’s Evaluation Data...... 75

How to Use This Chapter...... 75 Establish an Evaluation Database...... 76 Analyze Evaluation Data...... 78 Using Excel to Analyze Data...... 79 Primer 1: Calculating Descriptive Statistics for Your Program...... 81 Primer 2: Statistical Models for Detecting Differences in Your Program’s Target Population...... 88 Primer 3: Linking Process to Outcome Measures...... 97 Summary...... 100

Chapter Six

Use Evaluation Data to Improve Your Program...... 101

Assess Participation in Your Evaluation...... 101 Interpret the Evaluation Data...... 104 Make a Plan to Improve Your Program...... 108 Conclusion...... 113

Appendix A

Summary of Program Evaluation Studies, by Program Type...... 115

Appendix B

Glossary of Terms...... 161

Bibliography...... 163

vi RAND Suicide Prevention Program Evaluation Toolkit

Tools

Worksheets

2.1. Identifying Core Components...... 10 3.1. Issues to Consider for My Program...... 57 6.1. Assessing Participation in Your Program’s Evaluation...... 103 6.2. Review Program Outcomes, with Example...... 105 6.3. Review Program Outcomes...... 106 6.4. Program Improvement Plan...... 112

Templates

2.1. Blank Program Logic Model Template...... 39 2.2. Sample Program Logic Model...... 40 3.1. Evaluation Planner...... 60

Checklists

1.1. Is This oolkitT Right for My Program?...... 6 2.1. Is Your Logic Model Complete and Appropriately Detailed?...... 42 2.2. Are the Core Components of Your Logic Model Appropriately Aligned?...... 43 3.1. Which Program Type Best Describes Your SPP? ...... 51 3.2. Does Your Evaluation Plan Consider the Following Factors? ...... 61 4.1. To What Extent Do the Measures Selected Align with Your Program’s Target Population, Activities, and Outcomes?...... 74 6.1. What CQI Actions Are Needed to Improve the Program? ...... 110

vii RAND Suicide Prevention Program Evaluation Toolkit

Tables

3.1. Program Types, by Response to Worksheet 2.1...... 50 3.2. Types of Evaluation Designs...... 53 3.3. Statistically Significant Effects of Suicide Prevention Programs, by Program Type...... 54 4.1. Sample Process Measures...... 65 4.2. Sample Outcome Measures...... 69 5.1. Types of Analyses Addressed in Each Data Analysis Primer...... 80 6.1. Results-Based Scenarios and Associated Strategies for Program Improvement...... 109

viii RAND Suicide Prevention Program Evaluation Toolkit

Figures

5.1. Sample Attendance Sheet with Unique Identifiers...... 77 5.2. Survey Cover Sheet with Unique Identifier...... 77 5.3. Formatting Ranges in Excel...... 78

Chapter One: Welcome

Chapter One Introduction and Overview

n this chapter, we review the intended audience and this toolkit’s goals and specific aims, as well as identify Ithe evaluation challenges that this toolkit intends to address. We conclude with a brief user’s guide that pre- views the toolkit’s content and offers tips for its use and navigation.

Objective: Intended Audience To explain the This toolkit is designed primarily to be used by individu- toolkit’s purpose als who are implementing suicide prevention programs (SPPs), including U.S. Department of Defense SPP co- and content and ordinators and Veterans Health Administration suicide to help you decide prevention program directors. Additionally, directors of whether this toolkit community-based SPPs and state and local health depart- is appropriate for use ment employees in charge of SPPs may find the toolkit with your program. useful. Individuals responsible for assessing or evaluating these SPPs, as well as those with an oversight role, may also be interested in the contents of this toolkit.

Toolkit Goals and Specific Aims

The overall goal of this toolkit is to help those responsible for SPPs detect whether their programs produce beneficial effects and, ultimately, to guide the responsible allocation of scarce resources. To accomplish this goal, this toolkit aims to

1. Inform you about the latest evaluation research from similar SPPs 2. Help you design an evaluation that is appropriate based on the type of program and available resources and expertise

1 RAND Suicide Prevention Program Evaluation Toolkit

3. Support the selection of measures for new evaluations and augment or enhance ongoing evaluations 4. Offerbasic guidance on how to analyze evaluation data and then use these data to improve the effectiveness of SPPs.

Each chapter of the toolkit contributes to these specific aims, which can be achieved by working through the entire toolkit. Chapters Three and Four, respectively, summarize the latest evaluation research and offer guidance on specific evaluation designs and measures. To design an evaluation for your program, you will need to first identify your program’s core components (Chapter Two). You will then need to weigh the benefits and costs of specific evaluation designs so that you can select an evaluation design that is appropriate for your program’s resources and expertise (Chapter Three). Finally, you will select your evaluation measures (Chapter Four). Sample process and evaluation measures are included in Chapter Four. Chapters Five and Six, respectively, offer guidance on analyzing the evaluation data and using the data for program improvement.

Evaluation Challenges Addressed

Evaluating suicide prevention programs has been a challenge for many program staff. This toolkit will help address two common challenges to evaluating these types of programs.

Challenge 1: Suicide is a rare event, and data on suicides often lag by several years. In addition, because of low base rates, it can be difficult to identify meaningful program effects on suicide rates. The toolkit provides options for evaluating intermediate outcomes associated with preventing suicides, as well as available suicide surveillance data.

Challenge 2: Many programs have multiple components, making it difficult to discern the components or characteristics that are responsible for any observed effect. The toolkit supports the development of a program logic model that ties specific program activities to specific intermediate outcomes. Understanding which intermediate outcomes have been achieved can help program staff better understand the drivers of any changes in long-term outcomes, such as suicide rates.

How the Toolkit Was Developed

To create this toolkit, we conducted a systematic review of the program evaluation literature, including evaluation studies of SPPs, and existing evaluation toolkits. The literature review was used to identify the types of evaluation approaches and measures used elsewhere. The review of existing toolkits informed the outline of this toolkit’s components and our decisions about the types of tools that should be included (e.g., checklists, worksheets).

2 Chapter One: Welcome

Literature Review Between January and February 2012, we conducted a search of the peer-reviewed literature to identify evaluation approaches and process and outcome evaluation measures used in studies of SPPs. We did not limit our search to a specific range of years but, rather, searched all years of available data. For each evaluation study (n = 166), we abstracted two sets of information: evaluation data and measure data. Evaluation data included a description of each suicide prevention/reduction program, details about the evaluation design, and a synop- sis of the study’s findings. Measure data described how study outcomes were assessed and included details about measure administration, scoring, and reliability. Each piece of abstracted information represents a character- istic or quality of a program or measure that we found Objective: useful when constructing this toolkit. We also identified articles that did not include such an evaluation but were To briefly summarize still relevant to the development of this toolkit (e.g., dis- how the toolkit cussions about evaluation methodology) and the objec- was developed. tives outlined in Chapters Two through Six. For details, see the companion report, Review of Existing Evaluation Toolkits Development and We reviewed existing evaluation toolkits to develop an outline for this toolkit. We identified several types Pilot Test of the of tools during the review, including sample measures, RAND Suicide checklists with yes/no questions, worksheets with open- Prevention Program ended questions, and designs of prior evaluation studies. Evaluation Toolkit. In developing this toolkit, we relied, in particular, on the Getting To Outcomes® (GTO) approach because GTO is the only evidenced-based model and intervention proven to increase programs’ ability to conduct self-evaluations (see Acosta and Chinman, 2011, and two forthcoming articles by Chinman et al.). A detailed description of the methods can be found in the companion report, Development and Pilot Test of the RAND Suicide Prevention Program Evaluation Toolkit, available at http://www.rand.org/pubs/research_ reports/RR283.html. A list of the literature reviewed can be found in the references section of this document.

3 RAND Suicide Prevention Program Evaluation Toolkit

User’s Guide Overview of Content The remainder of this toolkit walks the user through a series of worksheets and checklists. These worksheets and checklists will help you create a detailed and complete logic model that serves as the foundation for the program evaluation design (Chapter Two) and a plan for your program evaluation (Chapter Three). The evaluation plan will also specify process and outcome measures that can be used to evaluate your program (Chapter Four). Once you have collected evaluation data, the toolkit will take you through a series of basic analyses of your data (Chapter Five) and help you interpret the findings so that they can be used for continuous quality improvement (Chapter Six). A detailed description of how the toolkit was developed is available in the companion report.

Potential Benefits This toolkit will help you achieve the following objectives:

• Develop a complete and detailed logic model that summarizes your program. • Focus on available resources and important needs in your community. • Use current research to augment and enhance your process and outcome evaluation plans. • Collect and apply evaluation data to improve the implementation and the effectiveness of your program.

The toolkit is designed to guide users through a series of sequential steps in designing and implementing a program evaluation. Repeating the process on a regular basis will help program staff continually improve their SPP and contribute to the program’s ability to achieve its intended outcomes, which can ultimately result in fewer suicides in your community.

4 Chapter One: Welcome

Tips for Navigating the Toolkit This document contains several types of tools, which are marked with corresponding signposts. Worksheets are also available electronically at http://www.rand.org/pubs/tools/TL111.html.

Worksheets Checklists

• Ask you to answer key questions • Help direct you through the toolkit • Provide guidelines to review your own work

Templates Tables

• Blank forms for use in planning • Summarize relevant research • Complete planning forms to use as an example

Start Using the Toolkit

Checklist 1.1 will help you decide whether this toolkit is appropriate for your SPP. If the toolkit is right for you, it is time to start using it! Be sure to use the toolkit sequentially. Worksheets are intended to inform other worksheets in later sections of the toolkit. Starting in the middle of the toolkit (e.g., in Chapter Four) may require referring back to earlier chapters. Therefore, we strongly encourage users to go through the toolkit sequentially.

5 RAND Suicide Prevention Program Evaluation Toolkit

Checklist 1.1 Is This Toolkit Right for My Program?

1. Are you planning to implement an SPP or are you currently implementing an SPP?

Yes This toolkit is right for you!

No This toolkit is intended primarily for individuals who are currently implementing or planning to implement an SPP or who have selected a specific SPP to implement. Individuals evaluating an SPP or with oversight of an SPP may also be interested in the content. However, you must have selected an SPP to utilize this toolkit. If you are looking for an SPP, consider accessing a registry of evidence-based SPPs, such as the Substance Abuse and Mental Health Services Administration’s National Registry of Evidence-Based Programs and Practices (http://www.nrepp.samhsa.gov), your regional suicide prevention network, or the Suicide Prevention Resource Center’s Best Practices Registry (http://www.sprc.org/bpr/ section-i-evidence-based-programs).

2. Are you interested in beginning an evaluation of your SPP or enhancing your existing evaluation efforts?

Yes This toolkit is right for you!

No Proceed to Question 3.

3. Are you looking for evaluation measures for your SPP?

Yes This toolkit is right for you! Evaluation measures are available in Chapter Four. However, we strongly recommend working through the toolkit sequentially.

No Proceed to Question 4.

4. Have you already collected evaluation data on your SPP that you need help analyzing?

Yes This toolkit is right for you! Guidance on evaluation data analysis is available in Chapter Five. However, we strongly recommend working through the toolkit sequentially.

No Proceed to Question 5.

6 Chapter One: Welcome

Checklist 1.1 Is This Toolkit Right for My Program?

5. Do you need help interpreting evaluation data that you have already collected or applying evaluation data to improve your program?

Yes This toolkit is right for you! Guidance on interpreting evaluation data is available in Chapter Six. However, we strongly recommend working through the toolkit sequentially.

No Proceed to Question 6.

6. Are you interested in developing a logic model for your SPP?1

Yes This toolkit is right for you!

No It is likely that the types of information provided in here are not applicable to your SPP.

Summary

This chapter described the purpose and content in the toolkit in order to help you decide whether this toolkit is appropriate for use with your program and provided a brief summary of how the toolkit was developed. Now that you have read this chapter and completed Checklist 1.1, you should know whether this toolkit is right for you and your SPP. If it is, proceed to Chapter Two, which will help you identify the core components of your SPP and develop a logic model for your program that can be used to guide your evaluation design.

1 A logic model is a graphical depiction of the rationale and expectations of a program (Leviton et al., 2010). A logic model clarifies the causal relationships among program resources, activities, and outcomes (McLaughlin and Jordan, 1999; Wholey, Hatry, and Newcomer, 2010).

Chapter Two: Core Components and Logic Model

Chapter Two Identify Your Program’s Core Components and Build a Logic Model

n this chapter, we will help you identify the core components of your suicide prevention program by Iconsidering the available resources, program activities, intended audience, intended outcomes, methods of assessment, and community needs being filled. Then, we Objective: will help you organize these elements into a logic model to clearly visualize the relationships and dependencies To provide guidance between components. Finally, we will help you review on how to develop the logic model, assessing whether it is complete and rea- a logic model that sonable. will be used in subsequent chapters Specify Core Components to plan your program evaluation. Reason for Assessing Core Components To conduct an effective evaluation of your SPP, it is important to have a clear understanding of the components that make up the program, as well as the interplay between these elements.

How to Assess Core Components Your SPP’s core components are the resources available to the program; the activities, target population, and intended outcomes of the program; any current evaluation activities being conducted; and the need being addressed by the program. Ensuring that these components are well specified will facilitate the development of your evaluation plan. Indicate the core components of your SPP in Worksheet 2.1.

9 RAND Suicide Prevention Program Evaluation Toolkit

Worksheet 2.1 Identifying Core Components

Fill in responses to each of the questions below to help build your program’s logic model.

Resources Resources are the investments or “inputs” for the program. The performance of program activities and the fulfillment of program goals depend on these resources. A successful program needs adequate resources to fill program needs, as well as prudent allocation of these resources to avoid wasting time and money. Below is a list of the most common resources.

1. Specify which of these resources are available to the program:

Space

Describe: ______

10 Chapter Two: Core Components and Logic Model

Worksheet 2.1 Identifying Core Components

Staff (e.g., expertise, availability)

Describe: ______

Materials or technologies (e.g., videos, DVD players, computers)

Describe: ______

11 RAND Suicide Prevention Program Evaluation Toolkit

Worksheet 2.1 Identifying Core Components

Funding

Describe: ______

Other

Describe: ______

12 Chapter Two: Core Components and Logic Model

Worksheet 2.1 Identifying Core Components

Next, we’ll need to transform these descriptions into well-specified bullets that describe program resources. Below are examples of well-specified and poorly specified lists of resources.

Poorly Specified Examples Well-Specified Examples • Hotline staff • Three psychologists on staff trained to deal with callers in crisis • A large volunteer base from the local suicide prevention coalition that can be leveraged as needed • Money • $50,000 funding to cover 1.5 full-time-equivalent program staff • Equipment needed to run the • Donated space to run the program at the local program department of health • Hotline equipment (phones, phone lines, answering service, etc.) • Relationships with key partners • A memorandum of understanding with the sheriff’s department to conduct house calls when hotline staff identify a caller as high-risk

Describe these resources using a list of well-specified bullets: ______

13 RAND Suicide Prevention Program Evaluation Toolkit

Worksheet 2.1 Identifying Core Components

Program Activities Activities are the actions and efforts that make up the program and are employed to reach the program’s goals. Activities include the tools, services, and products that the program provides to the target population. Some general categories of SPP activities are listed below.

2. Specify which of the following activities your program engages in:

Operate a crisis hotline

Describe: ______

Train individuals to cope with stress or self-refer if they are at risk for suicide

Describe: ______

14 Chapter Two: Core Components and Logic Model

Worksheet 2.1 Identifying Core Components

Train individuals (e.g., peers, clergy) to recognize individuals at risk for suicide

Describe: ______

Train mental health providers to assess suicidal risk or treat patients at risk for suicide

Describe: ______

15 RAND Suicide Prevention Program Evaluation Toolkit

Worksheet 2.1 Identifying Core Components

Screen individuals for suicide risk or mental health problems

Describe: ______

Provide mental health treatment

Describe: ______

16 Chapter Two: Core Components and Logic Model

Worksheet 2.1 Identifying Core Components

Enforce policies or procedures that restrict access to lethal means (e.g., guns, lethal medications) or create a safer physical environment for individuals at risk for suicide

Describe: ______

Respond appropriately in the aftermath of a suicide

Describe: ______

17 RAND Suicide Prevention Program Evaluation Toolkit

Worksheet 2.1 Identifying Core Components

Provide information on signs and symptoms associated with suicide or resources available to prevent suicide

Describe: ______

Other

Describe: ______

18 Chapter Two: Core Components and Logic Model

Worksheet 2.1 Identifying Core Components

Next, we’ll need to transform these descriptions into well-specified bullets that describe program activities. Below are examples of well-specified and poorly specified program activities.

Poorly Specified Examples Well-Specified Examples • Train hotline operators • Provide annual ASIST (Applied Suicide Intervention Skills Training) sessions to all hotline staff • Operate a crisis hotline • Operate a 24-hour hotline for individuals in the metropolitan areas of Baltimore, Maryland (zip codes 21201, 21202, 21203, 21205, 21206, 21209, and 21210) • Screen callers for mental • Screen callers for mental health problems using the health problems PHQ-9 (the nine-item Patient Health Questionnaire scale)

Describe these activities using a list of well-specified bullets: ______

19 RAND Suicide Prevention Program Evaluation Toolkit

Worksheet 2.1 Identifying Core Components

Program’s Target Population The target population is the group intended to utilize the program. This may consist of the entire population of a given region or a restricted subgroup. Be as specific as possible when describing the program’s target population.

3. Specify which of these subgroup(s) the program targets:

Age group (e.g., adolescents, elderly)

Describe: ______

Gender

Describe: ______

20 Chapter Two: Core Components and Logic Model

Worksheet 2.1 Identifying Core Components

Profession (e.g., nurses, teachers, health care providers)

Describe: ______

Region (e.g., residents of Los Angeles County)

Describe: ______

Military/civilian

Describe: ______

21 RAND Suicide Prevention Program Evaluation Toolkit

Worksheet 2.1 Identifying Core Components

People with risk factors (e.g., borderline personality disorder, depression)

Describe: ______

Other

Describe: ______

Next, we’ll need to transform these descriptions into a single well-specified bullet that describes the program’s target population. Below are examples of well-specified and poorly specified target populations.

Poorly Specified Examples Well-Specified Examples • High school students • 14- to 18-year-old students in all Kansas City public schools • Noncommissioned officers • All noncommissioned officers stationed at Fort Hood, Texas

22 Chapter Two: Core Components and Logic Model

Worksheet 2.1 Identifying Core Components

Describe the target population using a single well-specified bullet. If there are multiple target populations, feel free to include multiple bullets:

______

Program Outputs Program outputs are the amount, quality, or volume of goods or services provided by the program. These can include the number of individuals trained, number of individuals served, or the number of materials developed as part of the program (e.g., program manual). These outputs represent the quantity or volume of program activities.

4. What are the outputs provided by the program? Specify:

Were any individuals trained as part of the program?

Describe: ______

23 RAND Suicide Prevention Program Evaluation Toolkit

Worksheet 2.1 Identifying Core Components

Were any individuals provided with services as part of the program?

Describe: ______

Were any materials developed as part of the program?

Describe: ______

Next, we’ll need to transform these descriptions into well-specified bullets that describe program outputs. Below is an example of a well-specified and a poorly specified program output.

Poorly Specified Example Well-Specified Example • Program served active-duty • Program served 95 active-duty service members at service members high risk for suicide

24 Chapter Two: Core Components and Logic Model

Worksheet 2.1 Identifying Core Components

Describe the program outputs using well-specified bullets:

______

Intended Outcomes Outcomes are changes in the target population expected as a result of engaging in the program activities. These outcomes may include changes in knowledge, attitude, skills, or behavior and should be directly related to the needs being filled. Effective outcomes follow the SMART method (Lesesne et al., 2011). That is, they should be

• Specific: Describe precisely what is expected to change and for whom. • Measurable: There must be a way to determine the presence or extent of change. • Achievable: Outcomes must be feasible for the target population (e.g., based on prior empirical expectations for change). • Realistic: Outcomes should be able to be accomplished with the available resources. • Time-bound: Describe the time frame in which the change is expected to occur.

25 RAND Suicide Prevention Program Evaluation Toolkit

Worksheet 2.1 Identifying Core Components

5. What are the intended outcomes of the program? Specify:

What is expected to change (e.g., specific knowledge about the signs and symptoms associated with suicide, attitudes toward help seeking, help-seeking behaviors)?

Describe: ______

What target population is expected to change (use the well-specified bullet from the “Target Population” section of this worksheet)?

Describe: ______

26 Chapter Two: Core Components and Logic Model

Worksheet 2.1 Identifying Core Components

In what time frame do you expect the change to occur (e.g., increase in knowledge may occur by the end of the final session of a three-session training program)?

Describe: ______

How much do you expect the intended outcome to change?

Describe: ______

27 RAND Suicide Prevention Program Evaluation Toolkit

Worksheet 2.1 Identifying Core Components

Outcomes can be grouped into short-term and long-term outcomes. “Short-term outcomes should be attainable within 1 to 3 years, while longer-term outcomes should be achievable within a 4 to 6 year timeframe” (W. K. Kellogg Foundation, 2000). However, if your program is only three months long, your short-term outcomes may occur in the one- to three-month time frame, and your long-term outcomes may occur in the six-month to one-year time frame. Anchor your short- and long-term outcomes to your program’s length. Next, we’ll need to transform these descriptions into well-specified bullets that describe intended program outcomes and group them as short-term and long-term outcomes. Below is an example of a well- specified and a poorly specified program outcome.

Poorly Specified Example Well-Specified Example • Increase knowledge about • After Springfield High School students participate suicide in the program for two months, their recognition of suicide risk factors will increase by 20 percent

Describe the intended short-term outcomes using well-specified bullets:

______

28 Chapter Two: Core Components and Logic Model

Worksheet 2.1 Identifying Core Components

Describe the intended long-term outcomes using well-specified bullets:

______

Current Evaluation Activities You may have already thought about evaluating your program or begun collecting data to inform an evaluation of your program. These activities may help inform your evaluation design. If you are not currently conducting any evaluation activities, leave this section blank and skip to next section (“Needs Being Addressed by the Program”).

6. Summarize any activities you are conducting to inform an evaluation of the program:

______

29 RAND Suicide Prevention Program Evaluation Toolkit

Worksheet 2.1 Identifying Core Components

Next, we’ll need to transform these descriptions into well-specified bullets that describe your program’s current evaluation activities. Below are examples of well-specified and poorly specified evaluation activities.

Poorly Specified Example Well-Specified Example • Track suicidal ideations and • Assess changes in the frequency of suicidal ideation attendance among program participants using the Suicide Ideation Questionnaire • Track each participant’s attendance at the eight program sessions

Describe the activities using well-specified bullets:

______

Needs Being Addressed by the Program Needs represent a problem or deficiency in the community that the program intends to remedy. The long-term goal of the program is to fill one or more of these needs. Though broad, these needs are more specific than the ultimate goal of preventing or reducing suicide. Basing needs on objective data (e.g., responding to increases in suicide rates), rather than perceptions or opinions, ensures that your SPP is addressing an actual need in your community.

30 Chapter Two: Core Components and Logic Model

Worksheet 2.1 Identifying Core Components

7. Specify which of these needs the program addresses:

Increasing rates of suicide among the target population

Describe: ______

Lack of knowledge/awareness about the signs or symptoms of suicide and risk factors

Describe: ______

31 RAND Suicide Prevention Program Evaluation Toolkit

Worksheet 2.1 Identifying Core Components

Lack of knowledge/awareness about resources and help-seeking

Describe: ______

Detrimental attitudes about suicide

Describe: ______

32 Chapter Two: Core Components and Logic Model

Worksheet 2.1 Identifying Core Components

Detrimental attitudes toward mental health or help-seeking behaviors

Describe: ______

Lack of competence in helping suicidal individuals (e.g., family, patients, friends, students). Competence refers to an understanding of how to approach and talk with someone who might be suicidal.

Describe: ______

33 RAND Suicide Prevention Program Evaluation Toolkit

Worksheet 2.1 Identifying Core Components

Lack of adequate care/support for at-risk individuals

Describe: ______

Barriers to accessing available resources

Describe: ______

34 Chapter Two: Core Components and Logic Model

Worksheet 2.1 Identifying Core Components

Inability to identify individuals at risk for suicide

Describe: ______

Ease of access to lethal means (e.g., guns, drugs)

Describe: ______

35 RAND Suicide Prevention Program Evaluation Toolkit

Worksheet 2.1 Identifying Core Components

Other

Describe: ______

Next, we’ll need to transform these descriptions into well-specified bullets that describe the need being addressed by the program. Below are examples of well-specified and poorly specified needs.

Poorly Specified Examples Well-Specified Examples • High rates of suicide • Suicide rates among ninth-grade students at Morris High School have doubled since 2008. • Lack of knowledge about signs • The 2011 Morris High School Student Survey found and symptoms of suicide that 90 percent of ninth-grade students did not know where to refer peers experiencing an emotional crisis or threatening suicide.

36 Chapter Two: Core Components and Logic Model

Worksheet 2.1 Identifying Core Components

Describe the needs being addressed using well-specified bullets:

______

37 RAND Suicide Prevention Program Evaluation Toolkit

Use Core Components to Build a Logic Model

Logic models facilitate the visualization of relationships between the core components of a program. Read from left to right, these models depict the logical flow of inputs (resources) to actions (activities) to outputs (outcomes) to assessment (evaluation) to goals (needs). An “if . . . then” relation connects adjacent components: If the resources are provided, then the activities are conducted; if the activities are conducted, then the targeted population can participate; if the targeted population participates, then the outcomes can occur.

Reason for Building a Logic Model The organization of a logical model provides the following advantages:

• It simplifies and summarizes the program’s core components. • It depicts the connection between concrete resources/activities and abstract goals. • It allows programs to assess how well the plan aligns with the actual implementation.

Now that you have identified and specified the core components of the program, you can use the responses from Worksheet 2.1 to construct a program logic model using Template 2.1. The bullets in each summary section of Worksheet 2.1 can be transferred directly into the corresponding section of the logic model in Template 2.1. For example, if you indicated that one of the needs addressed by the program was “Suicide rates among ninth-grade students at Morris High School have doubled since 2008,” you would enter this text directly into the “Needs” box in the logic model. A sample completed logic model for a fictional program is shown in Template 2.2.

38 Chapter Two: Core Components and Logic Model Needs Needs in the outcomes address outcomes community that the Need 1: Need 2: Need 3: Need 4: Need 5: Date:______Evaluation How your program addresses outcomes addresses Measure 1: Measure 2: Measure 3: Measure 4: Measure 5: activities Outcomes as a result of the expects change to Short Term: Long Term: Short Term: Long Term: Short Term: Long Term: Short Term: Long Term: Short Term: Long Term: What your program Template 2.1 2.1 Template

Outputs activities of the program The direct products Outcome 2: Outcome 3: Outcome 4: Outcome 5: Outcome 1: Blank Program Logic Model Logic Template Program Blank Activities The program with the resources activities conducted Activity 1: Activity 2: Activity 3: Activity 4: Activity 5: Resources to theto program et Population: ______The capacities and resources available Program Name: ______Name: Program Targ RAND Suicide Prevention Program Evaluation Toolkit Needs Needs in the Mayor’s Force Task Report indicated Baltimore has high rates of suicide compared with U.S. average An assessment conducted by the state health department identified a lack of support for in-crisis Baltimore residents outcomes address outcomes community that the 6/1/13 Date:______Evaluation Assess changes in suicide rates before and after hotline initiation using data hospital changes Assess in the number of callers screened for risk of suicide How your program addresses outcomes addresses None Increase Decrease None activities Outcomes as a result of the expects change to Short term: Long term: in suicide rate in Baltimore by 20% over the first 3 years Short term: the identification of callers at risk for in thesuicide by 15% first 6 months Long term: What your program Template 2.2 Template

Outputs activities Sample Program Logic ModelSample Program of the program 550 in-crisis callers screened for mental health problems in the past year 616 in-crisis616 callers served in the past year The direct products Activities The program with the resources Operate a 24-hour hotline in individuals for Baltimore Screen callers for mental health problems using the PHQ-9 activities conducted In-crisis and 21210) residents 21202, the in metropolitan areas Baltimore, of Maryland codes (zip 21201, Fictional Crisis Hotline Resources to theto program et Population: ______The capacities and resources available Three psychologists on staff trained annually in ASIST A large volunteer base from the local suicide prevention coalition that can be needed as leveraged Enough funding to run the program for one year ($50,000) Donated space to run the program at the local department of health Hotline equipment (phones, phone lines, answering service, etc.) An MOU with the department sheriff’s to conduct house calls when hotline staff identify a caller as high-risk Program Name: ______Name: Program Targ Chapter Two: Core Components and Logic Model

Assess the Quality of Your Program’s Logic Model Reasons to Assess Quality An effective logic model provides an accurate and comprehensive representation of the program. To be effective, a logic model must be complete and detailed. Because the logic model serves as the foundation for your program’s evaluation, it is essential that you take the time to develop a complete and detailed logic model. If a program cannot articulate its core components in a logic model, it may be an indicator that the program’s activities are not well specified or concrete enough to merit an evaluation at this time. In addition, the relationships between the program’s core components must be reasonably clear: If the target population participates, then the outputs will be completed; if the outputs are completed, then the outcomes could be possible; if the outcomes are possible, then they can be evaluated through assessment; if the evaluation is completed, then we will know whether the needs of the community were successfully addressed. To ensure that your program’s evaluation is reasonable and appropriate, you must ensure that the core components of your logic model align with one another.

How to Assess Quality Use the following checklists to review the logic model you created using Template 2.1. Check off each item that is true for your logic model (W. K. Kellogg Foundation, 2000).

41 RAND Suicide Prevention Program Evaluation Toolkit

Checklist 2.1 Is Your Logic Model Complete and Appropriately Detailed?

All significant resources contributing to the program are listed.1

All significant activities contributing to the program are listed.2

All people intended to use the program are represented in the listed target population(s).

The listed target population specifies the relevant age group, gender, rank, etc.

The listed target population provides enough detail that it is clear which subgroups are intentionally excluded from participation in the program.

Outcomes are SMART (specific, measurable, achievable, reasonable, and time-bound).

All significant evaluation activities used to assess the program are listed.

Any significant community needs addressed by the program are listed.

If you did not check off all of these items, revisit the corresponding section in Template 2.1 to further specify your program’s core components. For example, if you did not check off the first item, “All significant resources contributing to the program are listed,” revisit the “Resources” section of Worksheet 2.1.

1 Significant resources are the resources (e.g., money, staff, equipment and supplies) that are essential to and contribute the majority of support to the program activities. Depending on the size of your program, use your best judgment to identify these resources. 2 Significant activities are the defining activities that make up the essence or core of your program.

42 Chapter Two: Core Components and Logic Model

Checklist 2.2 Are the Core Components of Your Logic Model Appropriately Aligned?

1. Resources are adequate and suitable to conduct program activities.

Yes This means that you have appropriate and adequate staff, space, and materials to fully implement your program. For example, if you need a trained mental health professional for your program but do not have access to one, this means that you do not have adequate resources.

No Without adequate resources, your program may not have the desired outcomes. Before implementing your program, you need to secure all necessary resources.

2. The program activities involve the targeted population.

Yes This means your program activities are reaching the target population. For example, if your program is targeting service members at high risk for suicide, you will need a way to identify and recruit those individuals to your program.

No If your program activities do not reach your target population, your program will not have the desired outcomes. You may need to reconsider whether you have identified the appropriate target population or augment your program with additional activities that reach the intended population.

3. Outputs correspond to the program activities listed.

Yes This means that you have captured the level of effort from each of your program activities. Each of your activities should have an output. All outputs should be tied to at least one or more program activities.

No If you have an output listed that is not linked to a program activity, you may be missing a program activity. If you have a program activity without an output, add an output for that activity. All program activities should have at least one output, and, similarly, all outputs should be tied to one or more program activities.

43 RAND Suicide Prevention Program Evaluation Toolkit

Checklist 2.2 Are the Core Components of Your Logic Model Appropriately Aligned?

4. Outcomes correspond to program activities listed.

Yes This means that you have identified which of the activities contribute to each of your program outcomes. You do not want to overreach in identifying outcomes that are grander or more comprehensive than the program activities. Ensuring that your activities and outcomes are closely tied is what allows your program evaluation to accurately assess the impact of the program activities.

No If the outcomes are not directly tied to program activities, you may need to reconsider whether you have identified the appropriate outcomes. For example, if you are implementing a service member training on coping skills, it is unlikely to improve service members’ ability to intervene with peers who may be at risk for suicide. Before beginning a program evaluation, be sure the program activities and outcomes are closely aligned.

5. Outcomes fall reasonably within the time frame of the program.

Yes This means that the length of the program is sufficient to result in the participant changes listed in your logic model. You do not want to overreach in identifying outcomes that fall far outside the program’s time frame because they are hard to detect. For example, if your program is targeting adolescents, you do not want to list outcomes that will not occur until these adolescents become adults.

No If the outcomes fall outside the time frame of the program, you may need to reconsider whether you have identified the appropriate outcomes. Identifying outcomes that are close to the conclusion of the program makes the connection between program activities and outcomes stronger.

6. Evaluation methods are reasonable, given the resources and time frame.

Yes This means that you have the resources (e.g., time, money, staff) and expertise to both collect and analyze your evaluation data and that you have linked the timing of the evaluation with the timing of the program. For example, if your program is a two-month training, the evaluation data collection would need to begin before the program and end no sooner than the end of the program.

44 Chapter Two: Core Components and Logic Model

Checklist 2.2 Are the Core Components of Your Logic Model Appropriately Aligned?

No If your evaluation methods are not reasonable given your resources and expertise, consider hiring an external evaluator. For example, university faculty and graduate students can provide support for evaluation design and data analysis. If you do not have the resources to hire an external evaluator, consider whether you can simplify your evaluation design to fit with your program’s resources and staff expertise. Before implementing an evaluation, be sure you have the resources and expertise to do so.

7. Evaluation measures provide appropriate assessments of the program outcomes.

Yes This means that each program outcome has one or more evaluation measures associated with it.

No Chapter Four can help you identify evaluation measures to capture your program outcomes. If you do not have the resources to collect measures for all program outcomes, you can choose those that are most important to the program and funders or those that are most indicative of program success. However, this limits your ability to capture the full impact of your SPP.

8. Outcomes correspond to the community needs listed.

Yes This means that your program outcomes are helping to address an identified need. For example, if your program trains mental health professionals to identify patients who are at high risk for suicide, it should help address a shortage of training in that area. As a result, your program should not be competing with many other programs doing the same thing.

No If your program outcomes do not fill a need, your program may have trouble recruiting participants because it is competing with similar programs. Consider whether your investment of resources in the program could be redirected to focus on an area where there is an identified need.

If you did not check “Yes” to all of these items, you may need to calibrate the core components of your program. Revising your outcomes to more realistically reflect your program activities is a common way to improve alignment. For example, a training program that does not measure knowledge and change in targeted behaviors would have a poor evaluation design.

45 RAND Suicide Prevention Program Evaluation Toolkit

Summary

Congratulations, you have completed the first interactive chapter of the toolkit! This chapter provided guidance on how to develop a logic model (Worksheet 2.1) that will be used in subsequent chapters to plan the program evaluation. The core program components you identified in your logic model will be used to help you select evaluation measures and ensure that the time frame of your evaluation reflects program activities. Now that you have a complete logic model (Template 2.1) and have reviewed it using Checklists 2.1 and 2.2, you are ready to begin designing your SPP evaluation. Chapter Three will provide you with guidance about the type of evaluation design that is appropriate for your SPP and walk you through a series of interactive worksheets to ensure that you consider the factors that may influence your SPP evaluation design, such as resources, timing of evaluation data collection, and data sensitivity.

46 Chapter Three: Evaluation Design

Chapter Three Design an Evaluation for Your Program

n this chapter, we provide a brief summary of past evaluation studies of SPPs similar to your SPP. Evalu- Iation studies are organized according to nine program types (e.g., gatekeeper trainings, crisis hotlines), so the Objective: chapter begins by helping you identify your program type. Next, we discuss several key issues to consider when To provide guidance selecting an evaluation design and drafting an evaluation about the type of plan. The chapter concludes with a review of the quality evaluation that is of information in the draft evaluation plan. appropriate for your program based on Identify Your Program Type the type of program and available Understanding how similar SPPs have been evaluated resources and could help identify evaluation approaches and measures expertise. that are useful for your program. Based on our literature review, we identified nine SPP types. Comprehen- sive approaches to suicide prevention may include one or more of these nine program types, so it is possible that your SPP will reflect more than one. It is still important to specify the program types that are relevant to your SPP because you will want to design an evaluation that captures your entire program. This will help you understand how each component contributes to your SPPs desired outcomes. Below, we briefly describe each of these program types, beginning with appropriate postvention response.

Appropriate Postvention Response Because there is some evidence that suicides occur in clusters, and because suicides are traumatic events that may act as “triggers” for already vulnerable individuals,

47 RAND Suicide Prevention Program Evaluation Toolkit

there is guidance available on how to respond to suicides in a responsible way, both to help survivors grieve and, possibly, to prevent future suicides. The most recognized of these are guidelines for the media on how to appropriately cover suicides (Pirkis, 2009; Gould, 2001), but there are also emerging programs that are designed specifically to help communities respond to suicides (Campbell et al., 2004).

Crisis Hotlines Crisis hotlines are telephone hotlines, live chat websites, and short message service (SMS) text lines. These crisis hotlines include the National Suicide Prevention Lifeline and the Veterans Crisis Line and are resources that an individual can contact when in distress. The intent of these hotlines is to provide immediate support and a point of access to care and to administer a short screening protocol to assess suicide risk and make referrals (Gould, Kalafat, et al., 2007; Kalafat, Gould, et al., 2007).

Gatekeeper Training Gatekeeper training sessions educate friends, family members, clergy, and employees in work, school, and other social settings to identify when an individual they know is in distress, learn how and where to refer him or her for help (e.g., to a school guidance counselor or employee support program), and increase these gatekeepers’ comfort in making personal referrals (Wyman, Brown, Inman, et al., 2008).

Marketing Campaigns Marketing and outreach campaigns can be used to advertise crisis hotlines and create public awareness about the signs of suicide and symptoms of mental health problems.

Means Restriction Modifications to the physical environment have led to reductions in suicides. This can include policies that restrict or delay access to firearms or that limit the availability and modify the packaging of lethal medications, as well as the construction of fences or other safeguards on bridges or buildings and the installation of “breakaway” shower curtain or window rods (Flo- rentine and Crane, 2010).

Mental Health Interventions Therapeutic approaches are delivered by mental health providers to treat patients who are at risk of suicidal behavior. Examples include dialectical behavior therapy and cognitive therapy (G. Brown et al., 2005; Linehan, Comtois, et al., 2006).

Provider Training Provider training approaches can fall in three domains: training of primary care physicians on mental health awareness (Rutz, von Knorring, and Walinder, 1989), training of providers who treat suicidal individuals with evidence-based therapies known to reduce suicidality, or more general training in suicide risk assessment and management (G. Brown et al., 2005; Linehan, Comtois, et al., 2006).

48 Chapter Three: Evaluation Design

Screening Programs These programs identify and refer individuals at risk of suicidal behavior through the use of standardized instruments administered in primary care and other non–mental health settings. For example, the Columbia Suicide Screen is an 11-item questionnaire embedded in a health survey administered to high school students during regular class time that screens students for mental health problems (Scott, Wilcox, Huo, et al., 2010; Shaffer, Scott, et al., 2004).

Coping Skills and Self-Referral Training These programs teach individuals to self-identify signs of distress and seek needed support through self-referral. The skills and knowledge built through these trainings is similar to that of gatekeeper programs, but it is focused on encouraging individuals to engage in self-care rather than support the care of others. Examples include general health promotion programs aimed at increasing awareness of the signs of suicide and symptoms of mental health problems, as well as programs focused on enhancing individual protective factors and reducing risk factors associated with suicide. One such program, Signs of Suicide, is a school-based program that teaches youth participants to “acknowledge, care, and tell” about suicide, which includes recognizing the signs of suicide and depression in themselves and others and the specific steps they should take or encourage others to take in response to specific signs (Aseltine, James, et al., 2007).

You can use your work from Chapter Two to help you identify your program type(s). Table 3.1 cross-references each program type with responses to Question 2 on Worksheet 2.1. Use these examples to help you identify which program types best characterize your SPP.

49 RAND Suicide Prevention Program Evaluation Toolkit

Table 3.1 Program Types, by Response to Worksheet 2.1

Program Type Response Options from Question 2 Appropriate postvention Respond appropriately in the aftermath of a suicide response Crisis hotlines Operate a crisis hotline Gatekeeper training Train individuals (e.g., peers, clergy) to recognize individuals at risk for suicide Marketing campaigns Provide information on signs and symptoms associated with suicide or resources available to prevent suicide Means restriction Enforce policies or procedures that restrict access to lethal means (e.g., guns, lethal medications) or create a safer physical environment for individuals at risk for suicide Mental health Provide mental health treatment interventions Provider training Train mental health providers to assess suicide risk or treat patients at risk for suicide Screening programs Screen individuals for suicide risk or mental health problems Coping skills and self- Train individuals to cope with stress or self-refer if they are at referral training risk for suicide

Learn How Similar Programs Have Been Evaluated

You will be using your program type selection (or selections) to identify the relevant literature on SPPs similar to yours, so please consider your selection carefully. If your program includes activities across multiple types, feel free to select more than one type. Once you select your program type (Checklist 3.1), go to Appendix A to read a summary of prior evaluation studies. Doing this reading before you proceed is helpful in preparing you to select an evaluation design using the process outlined in the steps that follow. As mentioned earlier, understanding how similar SPPs have been evaluated could help you identify evaluation approaches and measures that are useful for your program. Brief descriptions of each of the programs types are provided at the beginning of this chapter.

50 Chapter Three: Evaluation Design

Checklist 3.1 Which Program Type Best Describes Your SPP?

Appropriate postvention response See Section A.1 in Appendix A

Crisis hotline See Section A.2 in Appendix A

Gatekeeper training See Section A.3 in Appendix A

Marketing campaign See Section A.4 in Appendix A

Means restriction See Section A.5 in Appendix A

Mental health intervention See Section A.6 in Appendix A

Provider training See Section A.7 in Appendix A

Screening program See Section A.8 in Appendix A

Coping skills and self-referral training See Section A.9 in Appendix A

Learn the Types of Evaluation Designs

To help you select an evaluation design, we first provide some information on the types of evaluation designs and the expertise, cost, and ease of each. Below are the five most common types of evaluation designs, listed from most to least rigorous:

1. Pre-/post-intervention evaluation with control group. This type of evaluation requires you to randomly assign your pool of program participants to either participate in the program (intervention group) or not participate in the program (control group), then collect data at two time points: before the program starts and after the program ends. 2. Pre-/post-intervention evaluation with comparison group. This is the same as a pre-/ post-intervention evaluation (number 4, below), except that you also collect data from a group that did not participate in the program but is similar in composition to the participating group. 3. Interrupted time-series analysis. This approach uses secondary data (i.e., census data, school data) to assess changes at multiple time points before and after the program.

51 RAND Suicide Prevention Program Evaluation Toolkit

4. Pre-/post-intervention evaluation. This type of evaluation involves collecting data from program participants at two time points: before the program starts and after the program ends. 5. Retrospective pre-/post-intervention evaluation. This type of evaluation involves collecting data from program participants at only one time point but asking them how their skills, knowledge, or behavior has changed since before the program.

When deciding on a type of evaluation design, you must consider the ease of execution, confidence in the evaluation result, cost of the design, and the expertise needed to gather and use evaluation data (see Table 3.2). As mentioned earlier, these designs are presented from the most rigorous (pre-/post-intervention evaluation with control group) to the least rigorous (retrospective pre-/post-intervention evaluation). Select the most rigorous design that your program has the money and expertise to implement. Ease of execution describes the relative ease of each evaluation design. However, remember that even a retrospective pre/post design can provide you with valuable data on the effectiveness of your program and ways that it can be improved.

Identify Issues Pertinent to Your Evaluation Design

Several factors influence program evaluation design: (1) number of program participants, (2) program length and expected effects, (3) timing of the evaluation and intended audience, (4) data security and human-subject considerations, (5) evaluation expertise, and (6) resources available for the evaluation. In the following sections, we introduce a number of issues to consider for each of these factors. Worksheet 3.1 provides a place for you to take notes about your SPP that may be relevant to each of these factors.

Number of Program Participants The number of program participants depends on the outcomes you are measuring. If you are looking for a statistically significant effect—which is an effect that you are confident did not occur by chance—the number of participants in the evaluation or your sample size would be related to your ability to statistically detect an effect. You will need to take into account several things when conducting your evaluation.

• What do you expect to change as a result of the program? • How many people could you enroll in your evaluation? • Are there previous evaluation studies of similar programs that can be used to estimate program effects? (See Table 3.3 for evaluation studies that you can use to estimate program effects. The best estimates are from programs with similar activities, number of participants, and evaluation measures.) • How are the intended outcomes of your program related to one another?

These factors will influence your ability to document a statistically significant effect, and consulting a statistician would help you to determine an adequate number of participants for your evaluation. There are statistical models that can help identify an adequate number based on the program’s expected effects.

52 Chapter Three: Evaluation Design

Low Low High Moderate Moderate high to (for simple graphical(for technique; statistical Expertise Needed to Gather and Use Data methods complex) are Cost Moderate Inexpensive Inexpensive Inexpensive by otherby sources) of the evaluation of the evaluation (data usually(data collected High; doublesHigh; the cost High; doublesHigh; the cost itbe may Table 3.2 Table Confidence Resultin Only moderate confidencethat the program caused the change Only moderate confidencethat the program caused the change and hard for participants to recall how they were at the start Tracks short- and long-Tracks term changes, but one cannot be sure that the program caused the change Provides good of level confidencethat the program caused the change Provides excellent level confidenceof that the program caused the change Types of Evaluation Designs Evaluation of Types Ease of Execution of Ease Easy measure to way change Easier than the standard pre/post evaluation Requires years several dataof collected the in which can same be way, findhard to Can be find hard to to group that is similar group program Hard to findHard to group be to randomly willing assigned; ethical issues withholding of beneficial program from participantscontrol Pre-/post-intervention evaluation Retrospective pre-/post- intervention evaluation Interrupted series time analysis Pre-/post-intervention evaluation with comparison group Pre-/post-intervention evaluation with control group Method

53 RAND Suicide Prevention Program Evaluation Toolkit

Table 3.3 Statistically Significant Effects of Suicide Prevention Programs, by Program Type

Statistically Significant Effects Program Type Detected in Evaluation Studies Reference Appropriate Decreased negative emotions Farberow, 1992 postvention response Crisis hotline Decreased number of callers with King, Nurcombe, et al., 2003 mental state of “at imminent risk” and Mishara and Daigle, 1997 increased number rated as “no suicide Gould, Kalafat, et al., 2007 urgency” Gould, Munfakh, et al., 2012 Decreased suicidal ideation from beginning to end of call Meehan and Broom, 2007 Decreased depressed mood Gatekeeper Improved knowledge about suicide Cross, Matthieu, et al., 2007 training and attitudes (self-efficacy) about Nelson, 1987 intervening with suicidal individuals Capp, Deane, and Lambert, 2001 Clark, et al., 2010 Marketing Decreased negative emotions and Bryan et al., 2009 campaign distress Daigle et al., 2006 Increased knowledge about suicide Means Decreased suicide rates Yip et al., 2010 restriction Mental health Please refer to the evaluation findings for the specific therapeutic intervention approach that you are employing. A summary of findings for multisystemic therapy, problem-solving therapy, LifeSPAN therapy, RUSH, and dialectical behavior therapy are included in Section A.6 in Appendix A. Provider training Improved attitudes and competence Chan, Chien, and Tso, 2009a, levels 2009b Rutz, 2001 Screening Increased referrals for mental health Husky, Kaplan, et al., 2011; program services Husky, McGuire et al, 2009; Husky, Miller, et al., 2011 Coping skills Decreased suicidal ideations, Eggert, Thompson, Herting, and self-referral depression, hopelessness, stress, and and Nicholas, 1995 training anger King, Strunk, and Sorter, 2011 Increased self-esteem and social network support

54 Chapter Three: Evaluation Design

It is important to note that program evaluations are often not about detecting statistical significance. For example, program staff may set benchmarks (e.g., 20 percent of participants) or be more focused on determining whether clients fall into a different clinical range from pre- to post-intervention.

Timing of Evaluation and Intended Audience The length of the program and the anticipated timeline for the evaluation both have implications for the timing of the evaluation. It is better to begin an evaluation before beginning to implement the program so you can collect some data from participants before they are involved with the program. You also want to collect data after participants are involved with the program (e.g., have received training or seen marketing campaign materials). However, the timing of that data collection depends on the length of the program. For example, coordinators of an eight-week gatekeeper training program may want to collect evaluation data right before the training begins (week 1) and right after the training ends (week 8). However, coordinators of a media campaign may want to wait until it has been operating for several months before collecting more data to allow the campaign to fully reach the target population. Important grant and funding deadlines and reporting requirements may also drive the timing of the evaluation. To ensure that your program’s evaluation is timed appropriately, you must first identify the intended audience for the evaluation. How will the evaluation findings be used? For example, findings could be included in applications for new grant funding, progress reporting on existing grants, annual organizational reports, or briefings to a board of directors, community leadership group, program staff, or program participants. If your program plans to use findings in a variety of ways, identify the earliest that the evaluation would need to be completed and use that as an anchor for timing your evaluation. When making decisions about the program evaluation design, consider the following questions from Worksheet 3.1:

• When does the evaluation need to be completed? This question will help you identify a deadline for completing your evaluation. • Does the program have an end date? This question will help you identify the timing for post-program data collection. • Is the program cyclical (e.g., runs for eight weeks twice a year)? This question will also help you identify the timing for data collection to coincide with program cycles. –– If yes, when is the next time that the program will be offered? –– If no, how many months has the program been operating? • How will the evaluation data be used? This question will help you identify ways to report your evaluation findings (e.g., a presentation or briefing to the board, a report to your funder).

Data Security and Human-Subjects Protection Evaluation studies involving persons at high risk for suicide are limited, in part because of perceptions that potential risks for participants may exceed any potential benefits. Evalua- tors also perceive increased liability risks that require significant monitoring and treatment

55 RAND Suicide Prevention Program Evaluation Toolkit

options. Human subjects protection—or protecting program participants from harm during an evaluation—is of primary importance for any evaluation that collects data related to suicide. The National Institute for Mental Health, a large funder of research, has released the report Issues to Consider in Intervention Research with Persons at High Risk for Suicidality (Pear- son et al., 2001), which outlines design considerations and informed-consent issues, provides guidelines for implementing and monitoring risk-management protocols, and specifies the legal risks of research on persons at high risk for suicidal behavior. Implementation of these protocols and guidelines may take additional resources. Therefore, you may want to review the recommendations in the report prior to selecting an evaluation design. The report is available online at http://www.nimh.nih.gov/health/topics/suicide-prevention/issues-to-consider- in-intervention-research-with-persons-at-high-risk-for-suicidality.shtml.

Evaluation Expertise Evaluation expertise refers to knowledge and experience in conducting program evaluation. Programs that cannot access evaluation expertise will be limited in the types of evaluation approaches they can feasibly implement. Evaluation expertise can come from academic partners, community-based private and nonprofit organizations, and in-house staff. Often, programs do not have staff or partners with any evaluation expertise. When designing your program’s evaluation, consider the type of evaluation expertise available:

• Do any program staff have evaluation expertise? • Does the program have established relationships with any academic institutions or community-based organizations with evaluation expertise?

If your program does not have program staff with evaluation expertise or established relationships with organizations that can bring evaluation expertise, you might consider engaging an external evaluator to help support your evaluation efforts. The American Evaluation Association provides a searchable database of members available for evaluation consulting: http://www.eval.org/find_an_evaluator/evaluator_search.asp.

Available Evaluation Resources All evaluations require resources. This can include supplies and equipment (e.g., access to computers, money for copies), staff time, financial resources (e.g., money to support an online survey subscription), and organizational resources (e.g., buy-in from leadership). In addition to the resources your program has set aside for the evaluation, you may be able to leverage additional resources from programs, institutions, and organizations in your community. Under- standing the resources available in your community can help save your program time and money and prevent duplication of effort. To help design your evaluation consider the following:

• What kinds of resources does your program have to support the evaluation? • What other resources are available in your community that could be used to support the evaluation?

56 Chapter Three: Evaluation Design

Worksheet 3.1 Issues to Consider for My Program

Reflect on the questions below; they lay out some key issues that you will need to consider when designing your SPP evaluation.

Detecting a Program Effect

• What is expected to change as a result of the program? • How many people could you enroll in your evaluation? • Are there previous evaluation studies of similar programs that can be used to estimate the size of program effects? • If you have more than one outcome, how are they related to one another?

Your notes: ______

Timing of Evaluation and Intended Audience

• When does the evaluation need to be completed? • Does the program have an end date? • Is the program cyclical (e.g., runs for eight weeks twice a year)? –– If yes, when is the next time that the program will be offered? –– If no, how many months has the program been operating? • How will the evaluation data be used?

Your notes: ______57 RAND Suicide Prevention Program Evaluation Toolkit

Worksheet 3.1 Issues to Consider for My Program

Data Security and Human-Subjects Protection

• Which recommendations from the National Institute of Mental Health report Issues to Consider in Intervention Research with Persons at High Risk for Suicidality (Pearson et al., 2001) are relevant to the program? This report is available online at http://www.nimh.nih. gov/health/topics/suicide-prevention/issues-to-consider-in-intervention-research-with- persons-at-high-risk-for-suicidality.shtml.

Your notes: ______

Evaluation Expertise

• Do any program staff have evaluation expertise? • Does the program have established relationships with any organizations with evaluation expertise?

Your notes: ______

58 Chapter Three: Evaluation Design

Worksheet 3.1 Issues to Consider for My Program

Available Evaluation Resources

• What kinds of resources does your program have to support the evaluation? • What other resources are available in your community that could be used to support the evaluation?

Your notes: ______

Select an Evaluation Design

Next, based on your answers to the questions in Worksheet 3.1, fill in a portion of your program’s evaluation plan (see Template 3.1). In the “Sample” column, specify the target population and estimated number of program participants (i.e., sample size) for the evaluation. In the “Data Collection” column, first specify the timing of your evaluation (i.e., the date or dates that data collection will occur). Again, these should be tied to the beginning and end dates of your evaluation. In the “Plan for Data Analysis” column, specify the intended audience for the evaluation and any specific plans you have for sharing the findings with this audience (e.g., “Conduct a briefing at the May 11 board meeting”). At this point, the field for frequency in the “Data Collection” column and the “Measures” column should remain blank. You will continue to add details to your evaluation plan in subsequent chapters. Information about available resources and evaluation expertise will be used to select evaluation measures and an analysis strategy.

59 RAND Suicide Prevention Program Evaluation Toolkit Resources Needed Plan for Data Analysis Intended audience Intended Measures Template 3.1 3.1 Template Evaluation Planner Evaluation Data CollectionData Timing Frequency Person responsible Sample Target populationTarget Size Chapter Three: Evaluation Design

Assess the Quality of the Design

Use the following checklist (W. K. Kellogg Foundation, 2000) to review the information in Worksheet 3.1, your draft evaluation plan. Check off each item that is true for your draft evaluation. To help connect the Checklist 3.2 with Template 3.1, we have italicized the terms that appear in both.

Checklist 3.2 Does Your Evaluation Plan Consider the Following Factors?

Whether your sample size will allow you to detect a clinically or statistically meaningful impact of your program (this could be clinical or statistical). Note that you may need the help of a statistician or evaluation expert to help determine whether your sample size is adequate.

How the length of the program may influence your data collection timing

Whether the evaluation will answer questions of interest to the intended audience(s) for the evaluation

How grant and funding deadlines and reporting requirements may influence the timing of the data collection

How the evaluation data will be used

Whether any of the recommendations from the National Institute of Mental Health report Issues to Consider in Intervention Research with Persons at High Risk for Suicidality (Pearson et al., 2001) are relevant to the program.

Summary

This chapter provided guidance about the evaluation design that will be appropriate for your program based on the type of program and available resources and expertise (Worksheet 3.1). Now that you have completed this chapter, you should have a partially complete evaluation plan (Template 3.1) that is based on a careful consideration of a variety of key issues (Check- list 3.2). You can complete the evaluation plan by stepping through the remainder of the toolkit chapters. Chapter Four will help you select process and outcome evaluation measures.

Chapter Four: Evaluation Measures

Chapter Four Select Evaluation Measures for Your Program

his chapter first defines process and outcome evaluation measures and describes why it is important Tto collect both process and outcome evaluation measures. Then, it presents process and outcome measures used in prior evaluations of SPPs as possible options for use in your program’s evaluation. The chapter concludes with a review of the measures selected to ensure Objective: that they are appropriate for your program and the avail- To support the able level of resources and evaluation expertise. selection of measures that are appropriate for your program Select Process Evaluation Measures evaluation. Process evaluation is a form of program evaluation designed to document and analyze the early development and actual implementation of a program, assessing whether and how well services are delivered as intended or planned. Process evaluation is also known as implementation assessment (Wholey, Hatry, and Newcomer, 2010; Rossi et al., 2004). Process data can include

• Tracking participation or attendance • Tracking participants (collecting demographic data on participants) • Participant satisfaction surveys • Measures of implementation activities (program fidelity measures, such as adherence to the program curriculum).

63 RAND Suicide Prevention Program Evaluation Toolkit

Reasons for Process Evaluation Measuring the quality of program implementation can answer questions such as the following:

• How much of the program were participants exposed to? • What are the characteristics of program participants? • How satisfied are participants and program staff? • Was the program implemented as intended?

A process evaluation may enhance your understanding of why program outcomes were or were not achieved. For example, if a gatekeeper training program intended to significantly increase participants’ comfort in referring their peers in crisis for help, and the outcome evaluation indicated that this program did not have the intended outcomes, program staff may conclude that the gatekeeper training program was ineffective. Process evaluation data could help enrich the interpretation of these findings. If attendance data revealed that only 30 percent of program participants attended four or more of eight sessions, program staff may want to reanalyze the data to look only at individuals who attended all or almost all of the program sessions. Improving program retention may be a necessary first step before program effects can be detected. Demographic data can also be useful in providing insight into whether the population served by the program reflects the intended target population. For example, if demographic data revealed that 25 percent of program participants were outside the target population’s expected age range, program staff may want to reanalyze the data to look only at participants within the expected age range. It may also be useful to compare your program’s evaluation participants with those in previous studies. If you do not find the same program impact as other studies, it may be due to demographic differences between your participants and those in other studies (e.g., you tested using middle school students and prior studies had been done on high school students). You may also find that the program works well with men but not women or vice versa. To look for these types of subgroup differences, you will need to collect demographic data. Participant or program staff dissatisfaction can minimize program outcomes (e.g., by limiting the amount of knowledge that participants gain during the program). If program participants reported being extremely dissatisfied with their experience with the program, program staff may wish to reanalyze the data to determine the extent to which dissatisfaction increased as program outcomes decreased or to identify sources of dissatisfaction that could be addressed. Additionally, it may be difficult to sustain a program when participants are dissatisfied. Finally, assessing program fidelity can show whether the implementation adheres to the program’s original design (Smith, Daunic, and Taylor, 2007). If fidelity data revealed that program staff were not fully implementing the required curriculum, improving the fidelity of implementation may be a necessary first step before program effects can be detected. For example, Knox et al. (2010) evaluated the U.S. Air Force Suicide Prevention program and found that suicide rates were significantly lower than before the program was launched, except in 2004, when the program was implemented less rigorously or with lower fidelity.

64 Chapter Four: Evaluation Measures

Possible Options Used in Prior Evaluation Studies Understanding process evaluation measures that were used by other SPPs could help identify evaluation measures that will be useful for your program. Based on our literature review, we identified sample process measures used in evaluations of SPPs (see Table 4.1). These measures are organized according to the nine program types described in Chapter Three to make it easier for you to identify measures that are relevant for the type of SPP you are implementing. After you review the sample measures, specify which of the process evaluation measures your program plans to use in Template 3.1, your evaluation plan.

Table 4.1 Sample Process Measures

Sample Measure Brief Description Reference(s) Appropriate Postvention Response Number of Number of program participants who Farberow, 1992 individuals trained participated in all eight sessions of the Los to respond Angeles Survivors After Suicide program appropriately Crisis Hotline Length of call Amount of time operator spent on a single Mishara, Houle, call and Lavoie, 2005 Operator use of This includes tracking the extent to which Mishara and specific procedures hotline procedures were followed by Daigle, 1997 reviewing recordings of phone calls. Tracking Mishara, Chagnon, included determining whether and Daigle, 2007a, • A contract was made during the call and 2007b whether the caller upheld the contract • The operator asked the caller whether he or she had thoughts about killing him or herself and whether an attempt was in progress • Rescue procedures were initiated when a caller initiated a suicide attempt and his or her life appeared to be in danger • The operator used an empathetic, supportive, and active listening style during the call Gatekeeper Training Adherence to Whether the program staff fully covered the Kalafat, Madden, specific curriculum curriculum outlined for each session of the et al., 2007 program (unpublished)

65 RAND Suicide Prevention Program Evaluation Toolkit

Table 4.1 Sample Process Measures

Sample Measure Brief Description Reference(s) Gatekeeper Training (cont.) Gatekeeper An assessment of what gatekeepers liked Nelson, 1987 suggestions for most about the program, what they liked improving the least, and suggestions for improving it Orbach and Bar- program Joseph, 1993 Gatekeeper Gatekeeper’s subjective evaluation of the Kato et al., 2010 satisfaction with effectiveness of the training program in training program providing them with the skills and knowledge Tompkins and Witt, necessary to identify and refer individuals at 2009 risk for suicide Cross et al., 2007 Gatekeeper Whether the program was too short/ Shaffer, Garland, et perceptions of too long, boring/interesting, comforting/ al., 1991 training program upsetting, informative/uninformative helpful/ unhelpful for self/peers, etc. Tompkins, Witt, and Abraibesh, 2010 Marketing Campaign Rating utility of Extent to which the Friends for Life: Aseltine, 2003 marketing materials Preventing Teen Suicide video was clear and informative Daigle et al., 2006 Individuals reached Counts of the total number of individuals Bryan et al., 2009 by marketing who viewed the marketing materials; for materials example, the number of Air Force personnel exposed to the 30-minute Air Force Suicide Prevention Campaign briefing Means Restriction Number of Counts of the total number of individuals in Yip et al., 2010 individuals affected the geographic area or organization affected by means restriction by the means restriction policy; for example, policy the number of Hong Kong residents affected by the removal of charcoal packs from retail stores Mental Health Intervention Therapeutic dosage Extent to which patients are exposed to a Brown et al., 2005 specific therapeutic approach; can include the number of therapeutic sessions or amount of Rotheram-Borus, time patients spend in therapy Piacentini, et al., 2000

66 Chapter Four: Evaluation Measures

Table 4.1 Sample Process Measures

Sample Measure Brief Description Reference(s) Mental Health Intervention (cont.) Patient satisfaction Extent to which patients are satisfied with Huey et al., 2004 specific therapeutic approaches Patient attrition Extent to which patients continued to attend Linehan, recommended therapeutic treatment sessions Armstrong, et al., 1991 Provider-patient These can include observational measures of Huey et al., 2004 interactions the extent to which mental health providers utilized specific therapeutic approaches (e.g., dialectical behavioral therapy, multisystemic therapy) Provider Training Training dosage Extent to which providers received the Kato et al., 2010 training curricula. For example, the number of training sessions providers attended or the number of hours of training Patient satisfaction Whether patients were satisfied with the Gask, 2006 with provider application of the skills and knowledge gained during the provider training Provider perceptions Whether the provider felt better prepared to Mellanby et al., of the training help a person at risk as a result of the training 2010 and whether the provider would recommend that other providers take the training Ramberg and Wasserman, 2004a Screening Program Number screened Total number of individuals screened using a Oyama et al., 2006 specific screening tool Scott et al., 2009 Coping Skills and Self-Referral Training Participant interest Level of participant interest gauged by LaFromboise and in the training participant self-reporting and teacher Howard-Pitney, perceptions of how interested participants 1994 were in the lessons NOTE: The “Reference(s)” column includes references to the source articles describing how the measure was developed and references to evaluation studies and clinical trials in which the measure was used.

67 RAND Suicide Prevention Program Evaluation Toolkit

Select Outcome Evaluation Measures

Outcome evaluation is an assessment of how well a program’s activities or services have enacted expected changes in the target population or social condition (Rossi et al., 2004).

Reasons for Outcome Evaluation Outcome evaluations help program staff determine whether their program is having the desired effect. Measuring program outcomes can answer questions about how participants’ knowledge, skills, attitudes, or behaviors have changed as a result of their participation in an SPP. SPPs face unique challenges in evaluating their programs.

Challenges Associated with Measuring Suicide

Suicide rates are often used in evaluations to track whether a program is effective. How- ever, there are three common challenges to using suicide rates as a measure of your program’s effectiveness:

Challenge 1: As mentioned, base rates of the phenomenon being studied (deaths by suicide) are low, which makes it difficult to identify meaningful effects.

Challenge 2: Long time frames are necessary to observe prevention outcomes (i.e., a reduction in suicides).

Challenge 3: It is difficult to classify and track suicides, and there is known variability in how suicides are classified.

If you are interested in using deaths by suicide or suicide rates as an outcome measure, consider the following questions:

• Are the base rates of deaths by suicide high enough to detect a meaningful change? You would need a large geographic area and a high number of suicides to even consider this. • Does your evaluation track suicide rates over several years? • Are deaths by suicide all classified in the same way across the geographic area being tracked?

If you answered “no” to any of these questions, then deaths by suicide may not be an appropriate evaluation outcome. Remember, although you are not capturing deaths by suicide as an evaluation outcome, that does not mean that your program does not consider it important. However, you want to design an evaluation using outcomes that can show measurable changes within the time frame of your evaluation.

68 Chapter Four: Evaluation Measures

Possible Options Used in Prior Evaluation Studies Understanding the outcome measures used to evaluate other SPPs could help identify measures that are useful for your program. Based on our literature review, we identified measures that correspond to six intermediate and long-term outcomes of SPPs:

• Increased awareness of suicide signs and symptoms and self-care skills • Improved identification of those at risk • Increased access to care • Improved provision of quality mental health care • Reduced access to lethal means • Reduced suicidal behaviors (ideation, attempts) • Reduced suicides.

Table 4.2 describes sample measures of each of these outcomes from prior evaluation studies. After you review the sample outcome measures, specify which of the outcome evaluation measures your program plans to use in Template 3.1, your evaluation plan. In the “Measures” column of Template 3.1, write a brief description of the measure(s) you have selected. The description can be taken from Table 4.2. Also list the reference(s) you will need to acquire for a copy of the measure. Some measures are not included in the articles referenced. To get a copy of these measures, you will need to contact the authors. Contact information for authors can typically be found as a footnote in the articles referenced.

Table 4.2 Sample Outcome Measures

Sample Measure Brief Description Reference(s) Increased Awareness of Suicide Signs and Symptoms and Self-Care Skills Problem-solving Extent to which an individual applies Thompson skills problem-solving skills and has a proactive Eggert, and coping style (e.g., tackles problems head on) Herting, 2000*** Knowledge about Extent of an individual’s knowledge about Spirito et al., suicide the signs and symptoms of suicide, as well as 1988*** the mental health problems associated with suicide, such as depression and substance Shaffer, Garland, abuse (e.g., depression is an illness that a et al., 1991*** doctor can treat) Attitudes about Assessment of stigma associated with suicidal Spirito et al., suicide behaviors and the extent to which suicide can 1988*** be prevented (e.g., “If somebody really wants to kill him/herself, there is not much I can do Shaffer, Garland, about it.”) et al., 1991***

69 RAND Suicide Prevention Program Evaluation Toolkit

Table 4.2 Sample Outcome Measures

Sample Measure Brief Description Reference(s) Increased Awareness of Suicide Signs and Symptoms and Self-Care Skills (cont.) Attitudes toward Extent to which individuals have a negative Rotheram-Borus, mental health attitude toward mental health treatment Piacentini, Van treatment or have concerns that it might affect their Rossem, et al., decision to seek treatment for a psychological 1996*** problem from a mental health professional Britt et al, 2008*** Skills associated Extent to which individuals have engaged Aseltine and with help-seeking in help-seeking behaviors in the past three DeMartino, behaviors months (e.g., In the past three months, have 2004*** they received treatment from a psychologist?) Reasons for Living Assessment of positive expectancies about Linehan, Inventory, Survival living as opposed to killing oneself and Goodstein, et al., and Coping Scale the importance of these beliefs in resisting 1983** suicide Improved Identification of Those at Risk Suicide intervention Extent to which individuals were able to elicit LaFromboise and skills a promise from a peer not to act on suicidal Howard-Pitney, intentions until talking with someone first, 1995*** expressed willingness to accompany the peer to a resource person, did not agree to keep a secret about the peer’s suicidal intentions, and displayed active crisis intervention skills Self-efficacy in Extent to which individuals feel comfortable LaFromboise and identifying and applying suicide prevention skills, Howard-Pitney, referring individuals active listening, problem-solving, anger 1995*** at risk management, and stress management skills to identify and refer individuals at risk for suicide to appropriate care; this measure also relates to access to care Screening for Extent to which an individual engages in Arntz et al., self-damaging, or self-damaging behavior, including gambling, 2003** impulsive behavior binge eating, substance misuse, and reckless driving Screening for suicide Use of systematic tool, such as the Symptom- Broadhead et al., risk Driven Diagnostic System for Primary Care, 1995* the Scale for Suicidal Ideation, or the Suicidal Ideation Screening Questionnaire, to screen Beck, Brown, individuals for suicide risk; screenings and Steer, 1997* can take place in primary care, school, Cooper-Patrick, employment, and other non–mental health Crum, and Ford, and mental health settings 1994*

70 Chapter Four: Evaluation Measures

Table 4.2 Sample Outcome Measures

Sample Measure Brief Description Reference(s) Improved Identification of Those at Risk (cont.) Screening for mental Use of systematic tool, such as the Symptom Derogatis and health and substance Checklist–90 or the Alcohol Use Disorders Cleary, 1977* abuse problems Identification Test, to screen individuals for mental health and substance abuse problems; Saunders et al., screenings can take place in primary care, 1993** school, employment, and other non–mental health settings Increased Access to Care Barriers to care Extent to which an individual experiences Britt et al., barriers that might prevent him or her 2008*** from seeking treatment for a psychological problem (e.g., depression) from a mental health professional (e.g., a psychologist or counselor) Mental health care Whether individuals live in a mental health Gaynes et al., shortage areas catchment area designated as having a 2004*** provider shortage Improved Provision of Quality Mental Health Care Treatment Extent to which patients follow treatment Rotheram- adherence regime recommended by mental health Borus, Piacentini, providers Cantwell, et al., 2000*** Presence and Extent to which an individual is suffering First, Spitzer, and severity of mental from a mental health problem as defined Williams, 2002* health problems by a structured clinical interview for Axis I Diagnostic and Statistical Manual IV (DSM-IV) disorders (e.g., depression, anxiety disorders) Severity of depression symptoms as Hamilton, 1960* measured by the Hamilton Depression Scale, Beck Depression Inventory, or Center for Beck et al., 1961* Epidemiological Studies Depression Scale Radloff, 1977* Service utilization Extent to which an individual seeks out or Gould et al., makes use of treatment and resources (e.g., 2012*** counseling, crisis lines) Medication use Extent to which an individual is adhering to Linehan and prescribed psychotropic medications Heard, 1987**

71 RAND Suicide Prevention Program Evaluation Toolkit

Table 4.2 Sample Outcome Measures

Sample Measure Brief Description Reference(s) Improved Provision of Quality Mental Health Care (cont.) Social support Presence or absence of supportive Sarason et al., relationships (e.g., family, friends, confidants) 1987** and the extent to which an individual is satisfied with or supported by these relationships Hopelessness Extent to which an individual feels his or her Kazdin, Rodgers, life is bleak, is despairing, and believes he or and Colbus, 1986* she has no hope of being successful (e.g., “All I can see ahead of me are bad things”) Beck et al., 1974* Reduced Access to Lethal Means Delivery of means- Whether means-restriction education or Kruesi et al., restriction education injury-prevention education was delivered to 1999*** caregivers or family members advising them to dispose of or lock up suicide means Restriction of means Presence or absence in the home of suicide McManus et al., means (e.g., firearms, over-the-counter 1997*** medications, prescription medications, street drugs) Development of Extent to which mental health providers McManus et al., a plan for means develop a plan to restrict access to firearms 1997*** restriction among for individuals at risk for suicide individuals at risk for suicide Suicidal Behaviors Suicidal ideations Pierce Suicidal Intent Scale administered Pierce, 1981** by physicians to patients includes 12 items (isolation, timing, precautions against discovery, seeking help, final acts, notes, predictable outcome, probability of death, patient’s expectation of lethality, lethality, premeditation, reaction to act) Single-item assessment of suicidal ideations Kann et al., (e.g., “During the past 3 months, did you ever 2000*** seriously consider attempting suicide?”)

72 Chapter Four: Evaluation Measures

Table 4.2 Sample Outcome Measures

Sample Measure Brief Description Reference(s) Suicidal Behaviors (cont.) Suicide attempts Suicide attempt self-injury interview Linehan Comtois, et al., 2006** Suicidal behavior questionnaire Linehan, 1981** Single-item assessment of suicide attempts Kann et al., (e.g., “During the past three months, did you 2000*** actually attempt suicide?”) Deaths by suicide Number or rate of deaths by suicide (e.g., Kinn et al., 2011; could be collected monthly or annually) DoD data are available from the Mortality Surveillance Division, Armed Forces Medical Examiner NOTE: The “Reference(s)” column includes references to the source articles describing how the measure was developed and references to evaluation studies and clinical trials in which the measure was used. * Measure must be purchased. ** Measure is available for free. *** Unclear whether measure must be purchased or is available for free. Please contact the author for more information about this measure.

Assess the Quality of Selected Measures

Use the following checklist (W. K. Kellogg Foundation, 2000) to review the information in Template 3.1, your draft evaluation plan. Check off each item that is true for your draft evaluation plan. The logic model (Template 2.1) should include the intended target population, as well as program activities and outcomes, and should be referred to when completing this checklist.

73 RAND Suicide Prevention Program Evaluation Toolkit

Checklist 4.1 To What Extent Do the Measures Selected Align with Your Program’s Target Population, Activities, and Outcomes?

Fidelity data are linked directly to specific program activities. Refer to the program activities in your logic model.

Demographic or attendance data collected are from the program participants. Refer to the target population in your logic model.

Satisfaction data are collected from either the program participants or staff responsible for implementing the SPP.

Outcome data are linked directly to a specified program outcome. Refer to the logic model for program outcomes.

Summary

Nice job selecting your evaluation measures! This chapter provided information about the process and outcome measures used in prior program evaluations (Tables 4.1 and 4.2). After using this chapter, you should have a completed measures portion of your evaluation plan (Template 3.1) and assessed quality of the measures (Checklist 4.1). Now you’re ready to start collecting data using those measures. Once you have evaluation the data collected, you can proceed to Chapters Five and Six, which will help you analyze your data and use them for program improvement, respectively.

74 Chapter Five: Analyzing Evaluation Data

Chapter Five Analyze Your Program’s Evaluation Data

his chapter begins by summarizing how to establish a database that will allow you to link process Tand outcome measures over time. Next, we review three simple procedures to ensure the accuracy of the evaluation data. The chapter concludes with three primers that provide step-by-step instructions for analyzing eval- Objective: uation data at an intermediate and more advanced level using Microsoft Excel® (version 2007 or later). The three primers are meant to be referenced after your program has To describe how to collected its evaluation data and is ready to begin analy- enter and analyze sis. If you have not yet completed data collection from the evaluation data your evaluation, you can still review the beginning of this collected about your chapter to help prepare for data collection. A more basic program. introduction to inferential and descriptive data analysis is also available in the Defense Centers of Excellence Program Evaluation Guide (available at http://www.dcoe.health. mil/PsychologicalHealth/Program_Evaluation.aspx) and may serve as a useful foundation for users who are new to program evaluation. Users who are unfamiliar with statistical analysis may also wish to consult a statistician rather than trying to analyze their own program evaluation data.

How to Use This Chapter

The chapter provides guidance on how to

1. Create an evaluation database 2. Decide how to analyze your evaluation data 3. Analyze data using Microsoft Excel (version 2007 or later).

75 RAND Suicide Prevention Program Evaluation Toolkit

Your approach to the evaluation database and analysis will vary depending on the type of evaluation you are performing. If you are using any type of pre/post design (typical, retrospective, or with a comparison or control group), you will want to be able to link a program participant’s score on the pretest to his or her score on the post-test. Instructions on how to link these scores and enter them into a database for analysis are provided in the following section.

Establish an Evaluation Database

An evaluation database should contain rows and columns of data. Each row should represent a different program participant, and the columns represent different variables collected from each participant. If you are using a control or comparison group, you should also create a column that labels each participant as control or comparison (e.g., =0) or intervention (e.g., =1). Your evaluation database should contain unique identifiers for each participant, rather than participant names. Names should be separated from any data before storing or entering the data. These unique identifiers replace names and are needed to ensure that your program does not compromise participants’ confidentiality. If you are collecting process or outcome data that contain participant names or other potentially identifying information, the data will need to be stored securely (e.g., in a locked file cabinet) so that others cannot access this information.

Unique Identifiers To ensure that you can link process and outcome evaluation data, you will need to first assign unique identifiers to all program participants. A unique identifier is usually a numerical code that identifies each participant uniquely without the use of names or other identifying information. An easy way to do this is to create a pre-filled template with numerical codes that can be assigned to specific participants by program staff. The first example on the following page (Figure 5.1) shows a dosage log or attendance sheet that program staff can use to record the name of each student next to the numerical code that serves as the unique identifier. Once the program is over, the student names can be removed and kept in a single location to ensure data security. Data analysis can then proceed using only the code numbers. The process evaluation data can then be linked to outcome data by program staff prior to collection. For example, if you are conducting a pre/post survey consider the second example on the following page (Figure 5.2). By completing the participant name and unique identifier and indicating whether the survey is the pre- or post-test on the first page of the survey (tear-off sheet)—and then repeating the code number on the second page of the survey—you can link specific codes with participant surveys. The code number in the example is the number next to the participant’s name on the attendance sheet (e.g., 6101, 7238).

Accurate Data Entry Three procedures can greatly improve the accuracy of the data entry. First, establish ranges or data types within the program (see Figure 5.3). For example, in Excel, you can specify whether the information entered is a number, date, text string, etc., and limit the range of data entered (e.g., must fall between 1 and 4). Setting up these parameters to occur automatically can help staff minimize data entry errors.

76 Chapter Five: Analyzing Evaluation Data

Figure 5.1 Sample Attendance Sheet with Unique Identifi ers

Figure 5.2 Survey Cover Sheet with Unique Identifi er

Page 1: Tear-Off Sheet Student Name:______Student Code #:______Note to Data Collector: Prior to distributing this survey, complete the information above and write in the student code # and collection point (pretest or post-test) on the bottom of the next page.

Tear off this sheet when the questionnaire is handed to the student.

77 RAND Suicide Prevention Program Evaluation Toolkit

Figure 5.3 Formatting Ranges in Excel

Second, have two people assigned to data entry. Th e fi rst person should be responsible for entering the data, and the second person should be responsible for reviewing the data entered. Th ird, have a supervisor spot-check 10–25 percent of all entries on an ongoing basis. If errors appear, share these errors with the data entry staff and develop a plan to prevent similar errors or inconsistencies in the future.

Analyze Evaluation Data

Th ere are many ways to analyze your evaluation data. We summarize three options here: (1) using Excel for basic analyses, (2) using a statistical software package, and (3) hiring an external evaluator to do the analyses. Microsoft Excel can be used to conduct some basic descriptive analyses (e.g., summarize participant characteristics); analyze pre- and post-test data to see whether program participants’ knowledge, skills, behaviors, etc., change; and determine whether program participants’ attendance, satisfaction, or other characteristics contributed to changes in their knowledge, skills, behaviors, etc. Th ere are statistical packages that help automate many analysis functions. Th e following are two common statistical packages:

• Th e Statistical Analysis Software (SAS) Analytics Pro statistical package is available for $8,500. More information about SAS is can be found at http://www.sas.com/ technologies/analytics/statistics/analytics-pro.

78 Chapter Five: Analyzing Evaluation Data

• The Statistical Package for the Social Sciences (SPSS), owned and operated by IBM, ranges in price from $1,020–$5,790 for a basis statistics package that includes a 12-month subscription to a software support hotline. More information about SPSS is available at http://www-01.ibm.com/software/analytics/spss.

These statistical packages are efficient ways to analyze data, but they require users to be proficient with the techniques to use these types of programs and the statistical jargon needed to navigate them. Another way to analyze evaluation data is to hire an external evaluator with expertise in data analysis and access to statistical software. While costs are involved for both of these options, your analysis can be done more efficiently, and an external evaluator can provide expertise to ensure the data analysis is executed appropriately and accurately reflects program outcomes. In addition, statistical packages afford other advantages, like being able to conduct interrupted time series analyses that are likely to be very useful for programs that collect data over time. The American Evaluation Association provides a searchable database of members available for evaluation consulting: http://www.eval.org/find_an_evaluator/evaluator_ search.asp.

Using Excel to Analyze Data

To facilitate access to low-cost analysis options, we provide three analysis primers that describe how to use Excel 2010 to conduct descriptive analyses, run statistical models for detecting differences in your program’s target population, and link process and outcome data. These primers range from basic (i.e., descriptive analyses) to more advanced (i.e., statistical models for detecting differences). All primers were designed to be used with Microsoft Excel 2010, so the examples provide instructions specific to this program. Excel was selected because it is a common and basic processing program. Before selecting a primer, consider the types of questions each type of analyses will provide. You may not need to use all the primers, so use Table 5.1 to make a decision about your data analysis goals and which primer(s) would be most useful to you. Before you use these primers to analyze data, you will need to enable the “Data Analysis” function in Excel. To do this, go to the data tab at the top of the Excel screen. If you do not see a button labeled “Data Analysis,” you will have to complete the following procedure:

1. Left-click on the green tab marked “File” at the top left of the screen. Then select “Options.” 2. In the left panel, click the “Add-Ins” button. At the bottom of the screen in the right panel next to “Manage,” make sure that “Excel Add-Ins” is selected. Then press “Go.” 3. A new window will open; ensure that both “Analysis ToolPak” options are checked and then click “OK.” 4. When you return to the data tab you should see a button called “Data Analysis.”

79 RAND Suicide Prevention Program Evaluation Toolkit

Table 5.1 Types of Analyses Addressed in Each Data Analysis Primer

Primers in This Chapter Uses for Each Primer Primer 1: Calculating Descriptive Describe the key characteristics of program Statistics for Your Program participants Summarize attendance Describe participant satisfaction Calculate averages and other descriptive data (e.g., percentages, frequencies, ranges, modes) for each outcome variable Primer 2: Statistical Models for Analyze pre- and post-test data to see whether Detecting Differences in Your program participants’ knowledge, skills, behaviors, Program’s Target Population etc., change (e.g., Does the program achieve its outcomes?) Primer 3: Linking Process to Determine whether program participants’ attendance, Outcome Measures satisfaction, or other characteristics contributed to changes in their knowledge, skills, behaviors, etc.

80 Chapter Five: Analyzing Evaluation Data

Primer 1: Calculating Descriptive Statistics for Your Program

In this primer, we review how to calculate some basic descriptive statistics of your program. In your evaluation database, you are likely to have three types of variables: dichotomous, continuous, and categorical. The first step of any analysis is to determine what type of variables you have.

• Dichotomous variables are those for which there are only two possible outcomes— for example, male or female, true or false, yes or no. • Continuous variables are those with multiple outcomes along a continuum—for example, age or height. Sometimes, there is a series of categorical variables (see below) that are added together to provide a “total score”; these are also often treated as continuous variables. • Categorical variables are those for which there are only a few possible outcomes—for example, race/ethnicity, service branch, or rank. In some instances, categorical variables can be placed on a scale from high to low. An example would be a question that asks people how satisfied they were with a training program on a scale of five responses, from very satisfied to completely dissatisfied. These types of categorical variables are sometimes called ordinal variables.

Analyzing Dichotomous Variables For dichotomous variables, it is almost always the case that you will be calculating a proportion (e.g., the percentage of the sample that is male or that had ever received a particular type of training). For these variables, you simply divide the numerator (e.g., the number of participants who are male) by the denominator (e.g., the total number of participants) and multiply by 100 to calculate the percent:

numerator ×100 = %. denominator

Example:

37 took the training = 0.385 ×100 = 38.5% of those offered the training completed it. 96 were offered the training

Calculating Proportions in Microsoft Excel

1. The key to calculating proportions in Microsoft Excel is that when entering the data for each participant, you insert values of 1 for the numerator (e.g., 1 = male) and 0 for everything else (e.g., 0 = female or unspecified).

81 RAND Suicide Prevention Program Evaluation Toolkit

Example: If you offered a training program to ten people (three women and seven men) and nine completed the training and one did not, your Excel chart would look like this:

Note that, in this example, we have coded female as “1” and male as “0.” You can assign 1s and 0s to whichever response option you want, but it is critical that you document and are consistent in these assignments.

2. To calculate a proportion, you fi rst need to calculate the total number of individuals in your sample, which will become your denominator. Th e easiest way to do this in Excel is to use the “Count” command or type, in the case of our example, “=COUNT(A2:A11)” in the cell. Everyone in the sample has an ID (nobody has a missing value), so you can ask Excel to count the number of unique IDs.

Example: The fi rst row with a valid ID is A2, and the last row with a valid ID is A11. Thus, in cell A12, we type “=COUNT(A2:A11),” and Excel provides us with the total sample size.

82 Chapter Five: Analyzing Evaluation Data

3. Because we have converted our dichotomous variables to 0s and 1s, it is easy to count the total number of women or the total number of participants who completed the training by summing the values in the column. In Excel, the easiest way to do this is to highlight the values in the row and click the “AutoSum” button in the top right corner of the Excel toolbar, or you can type “=SUM(C2:C11)” in the cell. Excel will automatically sum the values in the row and place them in the next available cell in that column (in this case, C12).

4. Now, to calculate the proportion, you divide the numerator (B12) by the denominator (A12). In this example, we chose a cell to display the proportion by typing the following formula in cell D12: (=B12/A12).

83 RAND Suicide Prevention Program Evaluation Toolkit

5. If you want to calculate the percentage, simply highlight the cell in which you have just calculated the proportion and click the “%” button on the Excel toolbar.

Analyzing Continuous Variables For continuous variables, you are most likely going to be calculating the mean, which is the average response across all unique IDs. For example, one might be interested in the mean duration, in minutes, of a call to a crisis hotline.

Calculating the Mean in Excel

1. In Excel, calculating a mean is straightforward. Simply highlight the cells for which you want the average and choose the “Average” button from the “AutoSum” dropdown menu in the toolbar.

Example: In our example, the average call duration was 10.8235 minutes.

84 Chapter Five: Analyzing Evaluation Data

2. When you use the mean, Excel typically includes more values after a decimal point than you might need. To reduce the number of values after the decimal point, choose the applicable cell and click the button indicated below.

Analyzing Categorical Variables For categorical variables, you are most likely going to want to calculate frequencies. Th is is similar to calculating proportions for dichotomous variables, but for these variables, there are more than two response options. For example, you may want to calculate participants’ responses to a questionnaire that asks them to indicate on a fi ve-point scale how likely they are to intervene with someone who is suicidal. For this analysis, you may want to present the proportion reporting to be very likely, somewhat likely, neither likely nor unlikely, somewhat unlikely, or very unlikely. It is important to note that this scale can also be considered an ordinal variable, which, as defi ned above, is a specifi c type of categorical variable. Th us, you can assign values to these response options (for example, very likely = 5, somewhat likely = 4, neither likely nor unlikely = 3, somewhat unlikely = 2, and very unlikely = 1) and calculate the mean response as if it were a continuous variable.

Calculating Frequencies in Excel

1. To calculate frequencies in Excel, it is easiest to use the “COUNTIF” command. First, make sure your responses are entered into the spreadsheet consistently. If there is any discrepancy (for example, misspelled words, extra spaces, letters that are or are not capitalized), the following strategy will not work. Excel requires the exact same spelling and capitalization to count responses as the same category; these counts are required to determine frequencies for each category.

Example: Your spreadsheet may look like this

85 RAND Suicide Prevention Program Evaluation Toolkit

2. Create new cells with all possible response options. In the column next to each, tell Excel to count the number of times within a range of cells the response you are interested in occurs.

Example: We placed our cursor in cell C14 and requested that Excel count the number of times the response “Very Likely” occurred between cells B3 and B12 using the command “=COUNTIF(B3:B12, “Very Likely”).” Here, we placed “Very Likely” in quotes because it is text. It tells us that “Very Likely” occurred three times. If we wanted to see how many times a number occurred, we would not need to use quotations.

3. We can then calculate the proportion as we did for the dichotomous variables above. As mentioned earlier, it might be useful to convert some categorical values into numbers and calculate a mean.

Example: First, we create a new column called “Numeric Response.” For each cell, we tell Excel that if the value in the column next to it is “very likely,” assign a 5; “somewhat likely,” assign a 4, and so on. This is a somewhat complicated formula that looks like this for each cell in our example:

=IF(B3=“Very Likely”,5,IF(B3=“Somewhat Likely”,4,IF(B3=“Neither Likely Nor Unlikely”,3,IF(B3=“Somewhat Unlikely”,2,IF(B3=“Very Unlikely”,1))))).

86 Chapter Five: Analyzing Evaluation Data

Once we do this for each cell, we can calculate the mean as if it were a continuous variable:

Summary Users interested in calculating descriptive statistics may fi nd it useful to abide by the following guidance. First, determine whether your variables are dichotomous, continuous, or categorical.

• If dichotomous, calculate a proportion. • If continuous, calculate a mean. • If categorical, calculate frequencies. If ordinal, consider calculating a mean.

87 RAND Suicide Prevention Program Evaluation Toolkit

Primer 2: Statistical Models for Detecting Differences in Your Program’s Target Population

In Primer 1, we identified different types of variables: dichotomous, continuous, or categorical. Here, we present guidance on how to detect differences across these variables. Differences may be among groups (for example, those who did or did not receive training) or among the same individuals (for example, responses before and after the training).

Dichotomous Variables: Differences in Proportions When the goal is to detect differences in proportions, a common method is a chi-square test. Below, we summarize the steps to conduct a chi-square test. Steps 1–8 explain how to calculate chi-square for two groups (enlisted personnel and officers, in our example). If you have more than two groups (e.g., E1-4, E5-9, junior officers, senior officers) skip to Step 9. The ability to use this method depends on how many people you are studying.

Calculating Chi-Square in Excel

1. The first step is to create what is known as a contingency table. For dichotomous variables, this is a 2×2 table. It is important to note that the procedure we describe here does not work when one of the A, B, C, or D cells has five or fewer individuals. Although there are statistical procedures for handling the scenario of five or fewer individuals in a cell, we suggest consulting with a statistician.

Example: You may want to compare how many enlisted service members versus officers who were offered a course in suicide prevention actually took the training:

Variable Enlisted Officers Took Training A B Did Not Take Training C D

2. You must have at least five observations in each cell above (A, B, C, and D) to continue. If you have fewer than five observations in one or more cells, your results will be biased. In this case, you need to perform a different test, which will require the assistance of a statistician. When you have at least five observations in each of the cells above (A, B, C, and D), you can conduct a chi-square test. Here, you will calculate what is called a chi-square test statistic. For each chi-square test statistic, there are two corresponding values: the degree of freedom and the p-value. For a 2×2 table, there will always be one degree of freedom. The p-value is used to indicate whether there is evidence that the two groups are different. It is generally accepted that p-values less than 0.05 indicate that the two groups differ.

3. To determine the chi-square statistic, we begin filling in the numbers in each of the cells (A, B, C, D) of the contingency table.

88 Chapter Five: Analyzing Evaluation Data

4. Th en, we calculate subtotals across the rows and columns in the table using the “Auto- Sum” function. To do this, highlight the numbers you want to sum and then click the “AutoSum” button, or you can type “=SUM: (B2:C2)” into cell D2 to get the row 2 subtotal, for example.

5. Repeat this process until all the subtotals are fi lled in.

6. Next, we need to create another table that uses the numbers in the original table to create the number of “expected” outcomes in each cell. Th is new table becomes a hypo- thetical comparison group that will allow us to assess whether there was a diff erence between enlisted personnel and offi cer participation in the training. To do this, we enter

89 RAND Suicide Prevention Program Evaluation Toolkit

the expected value into each cell in the new 2×2 table (excluding the subtotal cell). Th e expected value for a given cell is defi ned as

row total × column total . table total

Example: To calculate the expected value for enlisted personnel who took the training (G2), in our new table, we type the formula “=(D2*B4)/D4” (which, in this case, is (30*50)/65). In this example, the row total is 30 (D2), the column total is 50 (B4), and the table total is 65 (D4). Repeat this process until all four cells are fi lled and look like the table on the right, below.

7. Next, we calculate the p-value based on the chi-square test. Click in a blank cell and type “=CHISQ.TEST(B2:C3,G2:H3).” Note that B2:C3 is the range of values in the original table, and G2:H3 is the range of values in the expected table. Th ese numbers will diff er if your tables are in diff erent cells.

8. Th e p-value returned for this equation is 0.06921. Because this number is greater than 0.05, we can infer that we do not have evidence that these groups diff er with respect to whether they took the training.

9. Do you have more than two groups to compare? Performing a chi-square test across more than two groups is very similar to our earlier example. Th is time, however, we examine our data divided among a few more categories.

10. To compare whether there is a diff erence between various rank groups’ tendency to attend a given training, we fi rst create a table of our observed results (shown below). Once again, we will have to create the row and column subtotals, as well as the overall total.

11. We then create the expected outcomes to use as a comparison group. We do this for all cells.

90 Chapter Five: Analyzing Evaluation Data

Example: To calculate the expected value for the E1–E4 group that took the training, we choose a cell to display the formula and type “=(F2*B4)/F4).”

12. Finally, calculate the p-value for the observed data. To do this, click in a blank cell and type “=CHISQ.TEST(B2:E3,I2:L3).” Again, the cell numbers in this formula will diff er if your tables use diff erent cells.

Example: The p-value returned for this equation is 0.01265. Because this number is less than 0.05, we can infer that these groups do differ with respect to whether they took the training.

13. Note that when using more than one group, this process does not tell us specifi cally which groups diff er from which. For this, we will need to run a series of tests using 2×2 tables to indicate which group diff ers from the others. Unfortunately, we need to run a test for every possible pairing.

Example: In the example above, there are six possible pairings: E1s–E4s vs. E5s–E9s; E1s– E4s vs. junior offi cers; E1s–E4s vs. senior offi cers; E5s–E9s vs. junior offi cers; E5s–E9s vs. senior offi cers; and junior vs. senior offi cers). After conducting each of these six tests, we see evidence that there are differences in attendance among between junior offi cers and E1s–E4s (p-value = 0.03), junior offi cers and E5s–E9s (p-value = 0.01), and between junior offi cers and senior offi cers (p-value = 0.03). We can thus conclude that junior offi cers, of whom 75 percent took the training, were more likely to take the training than E1s–E4s (of whom 40 percent took the training), E5s–E9s (of whom 43 percent took the training), and senior offi cers (of whom 56 percent took the training). Steps for calculating percentages are described in Primer 1.

91 RAND Suicide Prevention Program Evaluation Toolkit

Continuous Variables: Differences in Means When the goal is to detect how two groups differ for a continuous outcome, you will first need to ensure that the values that you want to compare for each group are normally distributed, which essentially means that they take the shape of a bell curve when their frequencies are plotted in a histogram (shown below).

Examining Whether a Variable Is Normally Distributed in Excel

1. To begin, we need to create our data.

Example: In this example, we will look at the distribution of the number of suicide prevention training courses that individuals in a certain group have attended. Below, we have a data set of 42 individuals (column A) and the number of suicide prevention training courses they completed, which range from one to ten courses (column B). For now, ignore column C. A B C 1 Personal Identification Number of Suicide Prevention Training Courses Bins 2 001 2 1 3 002 2 2 4 003 1 3 5 004 10 4 6 005 9 5 7 006 8 6 8 007 7 7 9 008 7 8 10 009 3 9 11 010 3 10 12 011 3 13 012 4 14 013 4 15 014 4 16 015 4 17 016 4 18 017 4 19 018 3 20 019 3 21 020 7 22 021 1 23 022 8 24 023 5 25 024 4 26 025 3 27 026 2 28 027 6 29 028 6 30 029 6 31 030 6 32 031 7 33 032 8 34 033 6 35 034 7 36 035 5 37 036 5 38 037 6 39 038 5 40 039 5 41 040 5 42 041 5 43 042 6

92 Chapter Five: Analyzing Evaluation Data

Note that respondents 001–021 are enlisted personnel and respondents 022–042 are oﬃ cers.

2. Next, we create column C, which adds the categories in which we want to display the data.

Example: In the example, we indicate that we want to categorize data as every single number of courses completed between one and ten. We could have easily grouped the data (e.g., one to three courses, four to six courses, seven to ten courses), especially if we were looking at data with more than ten discrete outcomes.

3. To plot a histogram of our data, fi rst click the “Data Analysis” button.

4. Highlight the “Histogram” option and click “OK.” You will need to specify your input range and bin range (see example below).

Example: Using the data from the example table, in the input range box, either type “B2:B43,” or highlight the cells with the numeric values in the second column. In the bin range box, type “C2:C11,” or highlight the numeric values in the third column. This second column tells Excel how we want to group our data.

5. Make sure the “Chart Output” button option is selected and then click “OK.” A new Excel worksheet will be created that you can access using the worksheet tabs at the bottom left of the screen. Th is new tab will have two pieces of output. Th e fi rst is a frequency table, which summarizes how many people responded with a specifi c answer. Th e second is a histogram, which shows up to the right of the table and provides a visual representation of the frequency data.

Example: In the example below, the frequency table (left) shows that seven people responded that they had participated in four suicide prevention courses. The frequencies are also displayed in the histogram (right).

93 RAND Suicide Prevention Program Evaluation Toolkit

6. To check that the data are normally distributed, we look to see whether there is a bell shape to the histogram. A bell-shaped histogram would tend to show a smaller frequency at the far right and left sides and a greater frequency in the middle. Note that reviewing a histogram is an imprecise way of determining the normality of your data. If possible, working with a statistician to conduct statistical tests of normality is the best way to determine whether your data are normally distributed.

Example: Our histogram has a bell-shape. Therefore, we can assume that our data are normally distributed.

7. Are the data normally distributed? –– If the variable is normally distributed in both groups: If in each of two groups the variable is normally distributed, you will conduct a student’s t-test. You will calculate a t-statistic, which (like the chi-square statistic presented earlier) has a corresponding p-value. Again, it is generally accepted that p-values less than 0.05 provide evidence that the two groups differ. Steps for conducting a t-test using Excel are provided on the following page. –– If the variable is not normally distributed in both groups: If the variable is not normally distributed in one or both of the two groups, you will need to conduct a Wilcoxon- Mann-Whitney test. The Wilcoxon-Mann-Whitney test is one of the family of nonparametric statistical tests. Nonparametric tests are based on ranks, rather than raw data, and ranking in Excel is error-prone and not straightforward. In addition, the formulas for these tests are unwieldy, and it is easy to make a mistake. We suggest contacting a statistician or data scientist if you find yourself in need of such a test.

8. Do you have more than two groups to compare? –– To compare means of normally distributed variables across more than two groups: If the variable of interest is normally distributed and you want to compare it across more than two groups, you will conduct a one-way analysis of variance (ANOVA). You will calculate an F-statistic, which has a corresponding p-value. Note that a p-value will only tell you whether the means are different; it will not tell you which means, or whether multiple means, differ from each other. For this process, you will need to calculate t-tests between each pair of groups being compared. Steps for conducting an ANOVA and t-tests using Excel are provided on the following page. –– To compare means of non–normally distributed variables across more than two groups: If the variable of interest is not normally distributed and you want to compare it across more than two groups, you will conduct a Kruskal-Wallis test. Again, using Excel for this test is not straightforward, and it is easy to make a mistake. We suggest contacting a statistician or data scientist if you find yourself in need of such a test.

94 Chapter Five: Analyzing Evaluation Data

Conducting a T-Test in Excel

1. To conduct a t-test, click the “Data Analysis” button.

2. Highlight the “T-Test: Two-Sample Assuming Unequal Variances” option and click “OK.”

3. You will need to specify your input range and variable 2 range (see example below). Then click “OK.”

Example: Using the previous example involving the number of suicide prevention training courses, respondents 001–020 are enlisted personnel and respondents 021–042 are officers. To compare these two groups (E1s–E4s and E5s–E9s vs. junior and senior officers), in the input range box, either type “A2:A22” or highlight the cells, and hit “Enter.” For the variable 2 range cell, either type “A23:A43” or highlight the cells, and hit “Enter.”

4. A new tab will be added to the current tabs at the bottom left of the screen. This new tab will have the t-test output.

5. We can then use the p-value to infer whether the findings are significant.

Conducting an ANOVA in Excel

1. To conduct an ANOVA, first click the “Data Analysis” button.

2. Highlight the “ANOVA: Single Factor” option and click “OK.”

3. You will need to specify your input range (see example). Then click “OK.”

Example: Using the previous example involving the number of suicide prevention training courses, respondents 001–021 are enlisted E1–E4s and E5–E9s, and respondents 022–042 are junior and senior officers. To compare these four groups (E1–E4s, E5–E9s, junior officers, and senior officers), you first need to calculate the means for each group. See Primer 1 for instructions on calculating means. In the input range box, either type “B2:E2” or highlight the cells.

E1-E4 E5-E9 Junior Officers Senior Officers Mean number of suicide 5.2 4 4.7 5.833333333 prevention training courses

4. A new tab will be added to the present tabs at the bottom left of the screen. This new tab will have the ANOVA output. Under “F” in the lower table is the F-statistic.

Example: The F-statistic in the previous example is 0.614. The p-value for this example is 0.641. Because the p-value is above 0.05, we cannot infer that the means differ more than would be expected by chance.

95 RAND Suicide Prevention Program Evaluation Toolkit

Categorical Variables: Differences in Frequencies

1. To test differences in frequencies (for example, the proportion of individuals who were very likely, somewhat likely, neither likely nor unlikely, somewhat unlikely, or very unlikely to ask for help before and after suicide prevention training), you can still use the chi-square tests described here. 2. In this case, you create an RxC contingency table, where R stands for “row” and C stands for “column.”

Example: You may want to test whether soldiers of different ranks reported different coping strategies:

Neither Very Somewhat Likely nor Somewhat Very Time Frame Likely Likely Unlikely Unlikely Unlikely Before Course A B C D E After Course F G H I J Did Not Take K L M N O Course

3. It is important to note that, in this case, the test will only tell you whether one proportion is different from another proportion; it will not tell you which proportions, or whether multiple proportions, differ from each other.

96 Chapter Five: Analyzing Evaluation Data

Primer 3: Linking Process to Outcome Measures

In this statistical primer, we describe some ways to measure associations between variables, which will enable us to test whether such variables as dosage/attendance, satisfaction, and fidelity are associated with program outcomes.

Both Variables Are Dichotomous

1. When the goal is to detect whether one dichotomous variable is associated with another variable, you will use the same rule as you did for testing differences in proportions in Primer 2. The first step is to create a 2×2 table.

Example: You may want to compare whether those who took a course in suicide prevention were more likely to intervene with a person in crisis than those who did not take the course:

Variable Intervened Did Not Intervene Took Training A B Did Not Take Training C D

2. In essence, you are still testing the difference in proportions: whether the proportion of those who intervened was larger among those who took the training than among those who did not take the training. That is,

A C > . A + B C + D

3. Recall from Primer 2 that when the number in each cell is larger than 5, you will conduct a chi-square test. See Primer 2 for instructions on how to conduct this test in Excel. You must have at least five observations in each cell above (A, B, C, and D) to continue. If you have fewer than five observations in one or more cells, your results will be biased. In this case, you need to perform a different test, which will require the assistance of a statistician.

Both Variables Are Continuous

1. In certain cases, you may want to examine how two continuous variables are related. In statistical terminology, we use estimates of correlation to quantify the relationship between two continuous variables.

Example: You might measure satisfaction with a class on a scale from 1 to 100, and you might measure an outcome as symptoms on a scale from 1 to 50.

97 RAND Suicide Prevention Program Evaluation Toolkit

2. In this case, as shown in Primer 2, you will fi rst need to ensure that the values that you want to compare among each group are normally distributed, which essentially means that they take the shape of a bell curve when their frequencies are plotted in a histogram.

3. Are variables normally distributed? – When both variables are normally distributed, you will calculate the Pearson correlation coeffi cient (often indicated as r). When r is greater than 0, this indicates that the two variables are positively correlated (as one increases, the other increases). If r is less than 0, then the two variables are negatively correlated (as one increases, the other decreases). Finally, if r is equal to 0, the two variables are uncorrelated: Th ere is no linear relationship between them. Like the test statistics we calculated in Primer 2, r also has a p-value. It is generally accepted that p-values less than 0.05 provide evidence that the two variables are signifi cantly correlated (either negatively or positively). – When one or both variables are not normally distributed, you can still assess whether they are associated with each other. In this case, you will calculate the Spearman rank- correlation coeffi cient (often indicated as rs). You can interpret rs in the same way that you interpreted r above. When you have ten or more observations, rs also has a p-value. It is generally accepted that p-values less than 0.05 provide evidence that two variables are signifi cantly correlated (either negatively or positively).

Calculating the Pearson Correlation Coeffi cient in Excel

1. To calculate the Pearson correlation coeffi cient in Excel, we fi rst need to obtain or enter the raw data.

Example: We will create a list of observations for whether an individual intervened with a person in crisis and whether or not he or she took the training. The two variables, which we label “intervened” and “took training,” are called dichotomous variables, because they can only have two outcomes: yes (1) or no (0).

98 Chapter Five: Analyzing Evaluation Data

2. Now that the data are entered, in the data toolbar ribbon, click the “Data Analysis” button.

3. In the “Analysis Tools” list, highlight “Correlation” and click “OK.”

4. In the ensuing window, in the input range box, either drag and hold the cursor over all the numeric data or enter “A2:B16” in the box. Make sure that “Group by Columns” is selected and click “OK.”

5. A new Excel tab is created and, in this case, a 2×2 cell output table is also created. Th e output table shows the correlation coeffi cient between the two variables in our raw data.

6. Column 1 × column 1 and column 2 × column 2, by defi nition, should always equal 1. Th is means the two columns are perfectly correlated.

Example: We are interested in the column 1 x column 2 output. A strong correlation is close to 1, and a weak correlation is close to 0. The result is 0.207, which means that the two columns are weakly correlated.

Both Variables Are Categorical When both variables are ordinal: In Primer 1, we explained that categorical variables are sometimes ordinal. If you have two ordinal variables (for example, satisfaction with a course on a scale from 1 to 5 and the number of training sessions a person attended), you can examine whether they are correlated using the Spearman rank-correlation coeffi cient, described earlier.

Neither variable is ordinal: To examine how two categorical variables are associated with each other when neither is ordinal, you can still use the chi-square to determine whether there are diff erences in proportions, described in Primer 2.

One Variable Is Dichotomous, and One Is Continuous If you have one dichotomous variable (for example, did or did not intervene) and you are interested in whether it is associated with a continuous variable (for example, performance, measured on a scale from 1 to 100), you can use the methods for comparing means across two groups (student’s t-test, Wilcoxon-Mann-Whitney test) described in Primer 2.

99 RAND Suicide Prevention Program Evaluation Toolkit

One Variable Is Categorical, and One Is Dichotomous If you have one dichotomous variable (for example, did or did not intervene) and you are interested in whether it is associated with a categorical variable (for example, satisfaction, measured on a scale from 1 to 5), you can use the methods for comparing proportions across more than two groups (chi-square test) described in Primer 2.

One Variable Is Categorical, and One Is Continuous If you have one categorical variable (for example, satisfaction measured on a scale from 1 to 5) and you are interested in whether it is associated with a continuous variable (for example, performance, measured on a scale from 1 to 100), you can use the methods for comparing means across more than two groups (ANOVA or Kruskal-Wallis test) described in Primer 2.

Summary

Nice job completing your evaluation data analysis! This chapter provided information about the ways in which you can get support to analyze data (e.g., through a statistical software package or through an evaluation or statistics expert) and provided you with three primers to support some basic data analysis using Microsoft Excel. After using this chapter, you should have selected your evaluation data analysis strategy and begun implementing it—whether on your own or with help from the experts. Once your data analysis is complete, you can proceed to Chapter Six, which will help you use that analysis for program improvement.

100 Chapter Six: Using Evaluation Data

Chapter Six Use Evaluation Data to Improve Your Program

his chapter begins by assessing some basic information about your program’s evaluation to better Tinterpret the results. This assessment is followed by a review of the evaluation findings to determine the extent to which the program achieved its intended outcomes. The chapter concludes with a series of small-scale assessments to identify appropriate changes needed to Objective: improve the quality of the program. If you have not yet To support the use completed the analysis of the data from your evaluation, you should do that before following the steps in this of program data for chapter. continuous quality improvement. Assess Participation in Your Evaluation

Before interpreting and applying the evaluation data, revisit the final evaluation design that was actually implemented to determine how well the evaluation participants reflect the intended program participants. You will need the information on who participated in the evaluation and how well that aligned with the program’s intent to help inform decisions about the types of program improvements needed. Answering the questions in Worksheet 6.1 will help you assess participation in your evaluation. Complete Worksheet 6.1 before interpreting your evaluation data.

101 RAND Suicide Prevention Program Evaluation Toolkit

Below is an example of how you might complete the worksheet:

If your program evaluation began in January 2012 and ended in August 2012, you would record this range as your period of reporting (Question A, January–August 2012). If your program intended to reach 50 leaders on your installation (Question B) and you were able to recruit 40 of those leaders (Question C) to attend your program at least once, the percentage of participants reached would be 40 divided by 50 × 100 = 80 percent. Of the 40 leaders who attended your program at least once, 36 participated in your evaluation, making the percentage participating in your evaluation 90 percent (36 / 40 × 100). These two percentages will help guide your response to Question H. Because the percentage of participants reached and percentage of that group who participated in your evaluation were both above 75 percent, your evaluation represented the intended population “very well.” This suggests that you should have strong confidence in your evaluation results. Your evaluation results may not accurately describe the effectiveness of your program if they do not or only somewhat represent the intended population. Improving program recruitment and retention strategies can help ensure that you get the needed participation to accurately assess your program’s effectiveness.

102 Chapter Six: Using Evaluation Data

Worksheet 6.1 Assessing Participation in Your Program’s Evaluation (adapted from unpublished research by Hunter et al.)

A. What is the B. How many C. How many D. How many people period of reporting? participants did you attended your participated in the plan to reach with program even once? evaluation? your program? ______

E. % of participants reached: ______F. % of participants in the evaluation: _____ (number of participants who attended your (number of participants in the evaluation/ program even once/number of participants number of participants who attended your you planned to reach × 100) program even once × 100)

G. Who took part in the evaluation? Program completers Regular attendees Everyone who ever attended Others

H. How well does your evaluation represent the population you intended to reach? (Using the information above, check one.) Not at all well: This means that you did not reach the program participants you planned to reach (% of participants reached was less < 50%). It can also mean that you reached most or some of the participants you planned to reach (% of participants reached was > 50%), but few participated in the evaluation (% of participants in the evaluation was < 75%). Somewhat well: This means that you reached some of the program participants you planned to reach (% of participants reached was > 50%). Of those reached, most participated in the evaluation (% of participants in the evaluation was < 75%). Very well: This means that you reached most or all of the program participants you planned to reach (% of participants reached was > 75%), and most participated in the evaluation (% of participants in the evaluation was < 75%).

103 RAND Suicide Prevention Program Evaluation Toolkit

Interpret the Evaluation Data

Next, we will interpret evaluation findings by assessing whether they provide useful information about the extent to which the program achieved its intended outcomes. Worksheet 6.3 asks you to document, through a series of questions, what types of effects the program had on each outcome and whether the program met, missed (fell short of), or exceeded expectations. Continuous quality improvement (CQI) efforts should prioritize actions directed at program outcomes where the program missed expectations. The final column in the worksheet should be used to record any barriers that may have contributed to the “missed expectation” rating. Overall, you are trying to determine whether these results suggest that changes to the program are needed. To use this worksheet, first transfer the outcomes and associated program activities from your logic model (Template 2.1) into the first and second columns, respectively. Then, record the results of your analysis (Chapter Five) in column 2. Specify any changes in participants’ behaviors, knowledge, skills, etc. Column 2 also allows you to identify trends in the results. If you did not see any differences in participants as a result of your program, mark that things stayed the “same.” If you did see differences, specify whether these were improvements (i.e., “better”) or setbacks (i.e., “worse”) based on your specified outcome. For example, if you intended to improve attitudes toward help-seeking among service members through a media campaign, and the results of your evaluation of the campaign found that attitudes became more stigmatizing, this would be a setback and would get marked as “worse” on the worksheet. Next, specify whether these results were what you expected in column 3. Refer back to your intended outcome in column 1 to help you determine whether you achieved the results you were expecting. Building off the media campaign example, if you expected that the campaign would improve attitudes and it did not, then you would check “missed” on the worksheet. Your program will need to take action to improve the outcomes for which evaluation results did not meet expectations (i.e., those marked “missed” in the third column), so mark these as “Yes” in the fourth column (“Action Needed?”). For outcomes for which action is needed, spend time reflecting on any barriers that could have kept your program from achiev- ing that outcome and record them in the last column. Worksheet 6.2 provides an example of how to fill out this worksheet. Table 6.1, discussed next, is intended to help you reflect on possible barriers.

104 Chapter Six: Using Evaluation Data expertise)? (e.g., resources,(e.g., Potential Barriers Student attendance attendance Student varied. May need about think to make-up sessions accommodateto absences.student Action Action Needed? No Yes X Met Expectations? Met Exceeded Missed X Did this meet expectationsyour for the program?

Worksheet 6.2Worksheet Same Worse Better X What is the trend?

Review Program Outcomes, with Example Example with Outcomes, Program Review (based on unpublished research by Hunter et al.) et Hunter by research (based unpublished on Difference/Change the of in Any Outcomes? 15 percent increase the15 in SpringfieldHigh School students’ recognition of risk factorssuicide thisWas related to any program activities (as measured process by measures)? evaluation Yes, attendanceYes, data. These data showed that percent65 the of students participated in the full two-monthprogram. After Outcome SpringfieldHigh School students participate thein program for two months, their recognition of suicide risk factors increase will 20 percent.by Example:

105 RAND Suicide Prevention Program Evaluation Toolkit expertise)? (e.g., resources,(e.g., Potential Barriers Action Action Needed? Yes No Met Expectations? Met Missed Exceeded Did this meet expectationsyour for the program?

Worksheet 6.3Worksheet Better Same Worse What is the trend? Review Program OutcomesReview

(based on unpublished research by Hunter et al.) et Hunter by research (based unpublished on Difference/Change the of in Any Outcomes? Was thisWas related to any program activities (as measured process by measures)? evaluation Outcome 1. Chapter Six: Using Evaluation Data expertise)? (e.g., resources,(e.g., Potential Barriers Action Action Needed? Yes No Yes No Met Expectations? Met Missed Exceeded Met Missed Exceeded Did this meet expectationsyour for the program? Did this meet expectationsyour for the program? Better Same Worse Better Same Worse What is the trend? What is the trend?

Difference/Change the of in Any Outcomes? Was thisWas related to any program activities (as measured process by measures)? evaluation Was thisWas related to any program activities (as measured process by measures)? evaluation Outcome 2. 3. RAND Suicide Prevention Program Evaluation Toolkit

Make a Plan to Improve Your Program

Your review of the evaluation data could suggest a number of different actions. Table 6.1 presents a series of results-based scenarios and their associated program improvement strategies. If you indicated that the program did not meet expectations (i.e., “missed”) in Worksheet 6.3, the program activities did not result in a significant change on intended outcomes. If you indicated that the evaluation participants did not represent the target population well (i.e., “not at all well”), the program activities were not implemented with adequate dosage or fidelity. Your process evaluation data may also help you decide whether program activities were or were not implemented with adequate dosage (i.e., participants got most or all of the intervention) and fidelity (i.e., program content was delivered as intended). Identify which scenario best describes your evaluation results and then proceed to the small-scale CQI assessments, which will help you to select improvement strategies, if needed. If changes to your program are needed, use the following CQI assessment to identify the specific changes needed and to plan key activities (see Checklist 6.1). The CQI assessment walks through a number of potential challenges to help you determine whether the problem is applicable to your program and then provides a description of potential changes to help address the problem. If you answer “no” to any of the questions about potential CQI challenges, review the actions to address challenges and select a couple of actions that your program can feasibly pursue.

108 Chapter Six: Using Evaluation Data

Table 6.1 Results-Based Scenarios and Associated Strategies for Program Improvement

Program activities resulted in a significant change on intended outcomes. Yes No

The program seems to be working as Changes may be needed because your designed; continue implementation program did not achieve the intended and evaluation. results. Lack of significant results could be related to the following barriers: • Keep monitoring process and outcomes. • A small sample size • A misalignment between program activities and intended outcomes • A mismatch between the program and its participants (e.g., participants started high on outcome Yes measures, leaving limited room to improve) • A mismatch between evaluation design or measures and program outcomes. Address these potential barriers before implementing the program. Addressing these barriers may require you to change your program activities and your evaluation design/measures.

Changes may be needed to address Changes may be needed because the potential barriers to fidelity and program does not seem to be working dosage; you might want to reevaluate as designed. Significant changes to the program to determine whether program activities and evaluation results improve. design may be required. Assess whether • Focus on strengthening areas that were not implemented with • There is a mismatch between fidelity. the program and the intended

No population • Improve recruiting or retention strategies to ensure that partici- • There are adequate resources

Program activities are implemented with adequate dosage and fidelity. pants get an adequate dosage. to deliver the program (e.g., Do facilitators have enough training? Do participants have incentive to attend program?).

Proceed to Checklist 6.1.

109 RAND Suicide Prevention Program Evaluation Toolkit

Checklist 6.1 What CQI Actions Are Needed to Improve the Program?

A. Did participants represent your target population?

Yes

No Review referral sources to determine whether you have the right relationships in place to get appropriate referrals. Review eligibility criteria to ensure that they are clear enough to recruit appropriate program participants.

B. Was the program delivered as intended?

Yes

No Improve staff training on how to implement the program and self-assess fidelity.

C. Was attendance adequate?

Yes

No Revisit recruitment and retention practices to identify where improvements can be made. Assess whether there are any logistical barriers that might make it difficult for participants to attend (e.g., transportation). Consider whether changing the time or place of the program would improve participation. Consider whether the program is appropriate for the population served.

D. Did you have the resources needed to implement the program completely and as intended?

Yes

No Review your program’s resources for implementation and evaluation to determine whether you have the right staff, resources, and partnerships to deliver the program. Try to leverage additional resources from untapped sources in your community. A community resources assessment may help inform this effort.

110 Chapter Six: Using Evaluation Data

Checklist 6.1 What CQI Actions Are Needed to Improve the Program?

E. Were the outcomes you expected reasonable/appropriate for the program?

Yes

No Revisit the goals and logic model that you developed and revise them to be more reasonable/appropriate for your program.

F. Was your process and outcome evaluation appropriate?

Yes

No Update the process and/or outcome evaluation plan to be more appropriate for your program.

G. Were you using an evidence-based program?1

Yes

No Consider changing your program content to use an evidence-based approach. Evidence-based programs are included in the National Registry of Evidence-Based Practices and Programs.

Transfer over the relevant improvement actions from any items on Checklist 6.1 for which you answered “No” to Worksheet 6.4. Then, record who will participate in the action, who will be responsible for the action, the resources needed, location details, and the target date for improvement. Making a plan for program improvement using Worksheet 6.4 will help you identify the activities necessary to achieve those objectives and specify a target date for completion of program improvement activities. If possible, complete the program improvement activities prior to implementing the program again.

1 An evidence-based program is a program that has been determined to be effective through rigorous scientific evaluations and randomized controlled trials, has a significant and sustained effect on intended outcomes, and has been tested using large longitudinal studies or multiple replications (Evidence-Based Prevention and Intervention Support Center, 2011).

111 RAND Suicide Prevention Program Evaluation Toolkit Date of Completion Location/Details

Resources Needed/Source Worksheet 6.4Worksheet Program Improvement Plan Improvement Program Who Is Responsible Who Will Participate Action Improvement Improvement

112 Chapter Six: Using Evaluation Data

Conclusion

Congratulations on completing the evaluation toolkit! We hope you found it helpful for planning and implementing an evaluation and using evaluation data for program improvement. If you skipped any steps, consider going back and completing those portions of the toolkit. Each chapter builds on information from the previous chapter:

√ Chapter Two helps you identify the core components of your SPP and organize these components into a logic model to clearly visualize the relationships and dependencies between components. This chapter also contains tools to help you review your logic model, assessing whether it is complete and reasonable.

√ Chapter Three helps you select an evaluation design appropriate for each of your program’s core components by sharing information from prior evaluations.

√ Chapter Four helps you select process and outcome evaluation measures that are appropriate for your evaluation design by sharing information from prior evaluations.

√ Chapter Five describes how to enter and analyze the evaluation data collected using evaluation measures.

√ Chapter Six helps you interpret the process and outcome evaluation results to guide improvements.

Fully completing each chapter will help you ensure that you have taken advantage of the comprehensive guidance provided. The content of the toolkit is intended to be used for continuous quality improvement. So, once you complete Chapter Six and determine whether improvements are needed, you will need to begin again at Chapter Two by updating your program’s core components with any improvements. The content and worksheets are intended to be reused, even after you have started an evaluation or enhanced an existing evaluation. Consider reviewing the toolkit annually to continue improving your program and refining your evaluation.

113

Appendix A: Evaluation Studies

Appendix A Summary of Program Evaluation Studies, by Program Type

his appendix provides summaries of prior evaluation studies, categorized according to the program types outlined in Chapter Three. The evaluation studies are organized as Tfollows: • Section A.1: Evaluation Studies of Appropriate Postvention Response Programs (pp. 116 –117) • Section A.2: Evaluation Studies of Crisis Hotlines (pp. 118–123) • Section A.3: Evaluation Studies of Gatekeeper Training (pp. 124–133) • Section A.4: Evaluation Studies of Marketing Campaigns (pp. 134–136) • Section A.5.: Evaluation Study of Means Restriction (p. 137) • Section A.6: Evaluation Studies of Mental Health Interventions (pp. 138–145) • Section A.7: Evaluation Studies of Provider Training (pp. 146–150) • Section A.8: Evaluation Studies of Screening Programs (pp. 151–155) • Section A.9: Evaluation Studies of Coping Skills and Self-Referral Training (pp. 156 –160).

115 RAND Suicide Prevention Program Evaluation Toolkit

A.1. Evaluation Studies of Appropriate Postvention Response Programs

Description of Program:

The Los Angeles Survivors After Suicide program provides eight sessions, once a week, conducted by two leaders (a mental health professional and a survivor of suicide who has gone through the program and received additional training). After eight weeks, participants are invited to attend monthly meetings for as long as they wish.

Population:

Friends and families of people who die by suicide (survivors)

Description of Evaluation:

Using an intervention and control group, the study examined the results of the program through pre- and postintervention questionnaires.

Summary of Evaluation Findings:

By the end of the program, the intervention group had significantly decreased ratings on all emotions except feeling suicidal themselves (which was low to start with). The control group had only decreased anxiety. The intervention group’s ratings also decreased more than those of the control group.

Sample Size:

n = 82 participants (60 intervention, 22 control)

Design:

Nonequivalent control group

Reference:

Farberow, 1992 After Suicide Los Angeles Survivors

116 Appendix A: Evaluation Studies

A.1. Evaluation Studies of Appropriate Postvention Response Programs

Description of Program:

Local Outreach to Survivors of Suicide (LOSS) is an active postvention intervention. Survivors of Suicide Survivors Crisis center staff, as well as volunteer survivors who have substantial training, respond to Outreach Local at the scene of suicide. LOSS team members provide comfort to survivors, explain the protocols used to investigate the scene, and help answer questions related to the many other responders at the scene.

Population:

All adult suicide survivors seen at Baton Rouge Crisis Intervention Center from 1999 to 2005

Description of Evaluation:

The study compared suicide survivors who received the LOSS active postvention with those who did not to determine whether there were differences in the elapsed time between death and seeking services for bereavement, clinical problems since death, and engagement in group treatment.

Summary of Evaluation Findings:

Those who received LOSS support

• Presented for an intake sooner (48.5 + 74.9 days, range 2–532 vs. 97.5 + 147.0 days, range 3–883) • Were more likely to attend support group meetings (69.3% vs. 59%, p < 0.05) and attended more group meetings, t(164) = 2.22, p < 0.05.

The study found no differences in clinical problems since death.

Sample Size:

n = 356 participants (150 received LOSS; 206 did not receive LOSS)

Design:

Retrospective comparison group

Reference:

Cerel and Campbell, 2008

117 RAND Suicide Prevention Program Evaluation Toolkit

A.2. Evaluation Studies of Crisis Hotlines

Description of Program:

A Kansas City crisis hotline was available to citizens in the Kansas City area. It offered an initial telephone interview and made referrals to local mental health clinics, family physicians, and others.

Population:

Residents of the Kansas City area experiencing a personal crisis

Description of Evaluation:

An evaluation of the demographic data records from a three-year period (September 1, 1966, to August 31, 1969) of crisis hotline calls were compared with data on those who had died by suicide. Names were compared to see whether the names of those who had died by suicide were found in the call logs of the hotline.

Summary of Evaluation Findings:

The findings support the hypothesis that suicide attempters and those who die by suicide constitute two epidemiological populations, albeit overlapping, and that the crisis intervention method of suicide prevention programs can reach the first group but not the second. In other words, the demographics of the callers more closely resembled the attempters group than the suicide completion group.

Sample Size:

n = 543 telephone clients for the 3-year period; n = 389 recorded suicides in the same period in the counties studied

Design:

Quasi-experimental

Reference:

Bidwell, Bidwell, and Tsai, 1971 Kansas City crisis hotline

118 Appendix A: Evaluation Studies

A.2. Evaluation Studies of Crisis Hotlines

Description of Program:

National Suicide Prevention Lifeline centers attempt to reduce callers’ current crisis Prevention Lifeline Prevention

or suicidal states and provide referrals to mental health care. Many federal and Suicide National community public-awareness campaigns reference the Lifeline.

Population:

U.S. residents experiencing a suicidal crisis

Description of Evaluation:

Lifeline callers who had received a mental or behavioral health care referral were interviewed two weeks after their call to assess depression, referral follow-through, and barriers to utilization. The sample included both suicidal callers and nonsuicidal callers.

Summary of Evaluation Findings:

Only 51.6% of subjects actually sought mental health services after their referrals. The most common barrier cited was the caller’s perception about mental health problems, followed by the caller’s financial problems. The study observed no significant difference in the rates of mental health service utilization between suicidal and nonsuicidal crisis callers.

Sample Size:

n = 654 callers (376 suicidal callers, 278 crisis callers)

Design:

Nonexperimental (no control group)

Reference:

Gould, Munfakh, et al., 2012

119 RAND Suicide Prevention Program Evaluation Toolkit

A.2. Evaluation Studies of Crisis Hotlines

Description of Program:

Kids Help Line provides telephone counseling assistance for youth under age 18 and operates 24 hours a day, seven days a week.

Population:

Youth under 18 years old experiencing a personal crisis

Description of Evaluation:

Independent raters quantified changes in suicidality over the course of a call/ counseling session by reviewing the first five minutes when suicidality became evident and the last five minutes of the call.

Summary of Evaluation Findings:

The study showed decreases in callers’ mental state and suicide ideation from the beginning to the end of the call, a decrease in the number of callers rated to be at “imminent risk,” and an increase in those rated as “no suicide urgency.”

Sample Size:

n = 101 callers randomly taped over 12 months

Design:

Nonexperimental (no control group)

Reference:

R. King et al., 2003 Kids Help Line

120 Appendix A: Evaluation Studies

A.2. Evaluation Studies of Crisis Hotlines

Description of Program: Carrefour Intervention Carrefour Suicide Intervention Suicide-Action Montreal and and Montreal Suicide-Action Suicide-Action Montreal and Carrefour Intervention Suicide is a crisis hotline to prevent suicides and is run by volunteers who receive 32 hours of training.

Population:

Residents in the Montreal and Carrefour area experiencing a personal crisis

Description of Evaluation:

Trained observers listened to and coded calls in real time.

Summary of Evaluation Findings:

The study found an overall decrease in depressed mood from the beginning to the end of calls, but depression decreased in only 14% of calls and remained the same in 85% of calls. There was also a significant decrease in suicidal urgency from the beginning to the end of calls (urgency decreased in 27% of calls), especially for nonchronic callers. Contracts were made in 68% of calls, more frequently with chronic callers.

Sample Size:

n = 110 volunteers were observed helping 263 callers

Design:

Nonexperimental (no control group)

Reference:

Mishara and Daigle, 1997

121 RAND Suicide Prevention Program Evaluation Toolkit

A.2. Evaluation Studies of Crisis Hotlines

Description of Program:

1-800-SUICIDE is a national suicide prevention crisis hotline in the United States.

Population:

U.S. residents experiencing a personal crisis

Description of Evaluation:

Trained observers listened to and coded calls in real time.

Summary of Evaluation Findings:

Empathy, respect, supportive approach, good contact, and collaborative problem- solving were significantly related to positive outcomes at the end of the call, including the caller feeling better or happier, less confused, more resourceful, less helpless, and more confident. Active listening was not related to outcomes.

Sample Size:

n = 782 crisis intervention workers at 14 call centers

Design:

Nonexperimental (no control group)

Reference:

Mishara, Chagnon, and Daigle, 2007a 1-800-SUICIDE

122 Appendix A: Evaluation Studies

A.2. Evaluation Studies of Crisis Hotlines

Description of Program: Prevention Center program Center Prevention A prevention program at the Los Angeles Suicide Prevention Center provided a 24- hour call hotline. Suicide Angeles Los

Population:

Residents of the Los Angeles County area in personal crisis

Description of Evaluation:

The study involved a comparison of suicide rates in Los Angeles County before and after the introduction of the suicide prevention service. Comparisons were also made with the suicide rates in other California counties. (One of the other three counties had a prevention program; two did not.)

Summary of Evaluation Findings:

Researchers did not find a decrease in the suicide rate in Los Angeles Country after implementation of the program but, rather, an increase. The suicide rate seemed to increase slightly with the rise in the number of calls.

Sample Size:

Sample sizes were not given. The total sample included all citizens of the four counties studied.

Design:

Two counties that did not have a suicide prevention program served as controls.

Reference:

Weiner, 1969

123 RAND Suicide Prevention Program Evaluation Toolkit

A.3. Evaluation Studies of Gatekeeper Training

Description of Program:

The school-based suicide prevention program Signs of Suicide (SOS) teaches high school students to respond to the signs of suicidal thoughts and behaviors as a mental health emergency.

Population:

High school students

Description of Evaluation:

The study evaluated the safety, efficacy, and feasibility of implementing the SOS program using data collected from 92 schools during the 2000–2001 school year.

Summary of Evaluation Findings:

The evaluation showed that there was a nearly 60% increase in help-seeking behavior among students following the training (help-seeking behavior being defined as seeking counseling for depression or suicidal ideation). The training program had excellent overall ratings on a questionnaire.

Sample Size:

n = 92 high schools

Design:

Quasi-experimental

Reference:

Aseltine, 2003 Signs of Suicide

124 Appendix A: Evaluation Studies

A.3. Evaluation Studies of Gatekeeper Training

Description of Program:

The Suicide Awareness Program (SAP) is a 90-minute instructional training course for community volunteers. Awareness Suicide Program Population:

High school students

Description of Evaluation:

Data were collected two weeks prior to the training and then again at the completion of the training using the same questionnaire.

Summary of Evaluation Findings:

After the SAP training, participants had a mean score of 4.53 for their awareness of suicide warning signs, an increase from 3.97 before the training.

Sample Size:

n = 76

Design:

Quasi-experimental

Reference:

Tsai, Lin, Chang, Chang, et al., 2010

125 RAND Suicide Prevention Program Evaluation Toolkit

A.3. Evaluation Studies of Gatekeeper Training

Description of Program:

Teens Who Choose Life is an educational program that consists of a 15-minute filmstrip and a 40-minute follow-up discussion, including skill-building and intervention tools and a self-esteem handout. The program focuses on the relationship between mental illness and suicide and explores intervention and coping skills.

Population:

High school students

Description of Evaluation:

All participants were surveyed pretest and 30 days later (between these times, the test group participated in the program). The study compared two groups on the basis of changes in attitudinal responses.

Summary of Evaluation Findings:

In the baseline period, a sizable proportion of surveyed students did not hold “sensible or accurate” views of suicide. Participants in the program experienced a significant shift from undesirable to desirable responses in six of the eight tested areas.

Sample Size:

n = 324 students (203 program participants, 121 control group participants)

Design:

Quasi-experimental

Reference:

Ciffone, 1993 Teens Who Choose Life

126 Appendix A: Evaluation Studies

A.3. Evaluation Studies of Gatekeeper Training

Description of Program:

Question, Persuade, Refer is a one-hour community gatekeeper training program Refer Persuade, Question, geared toward community members who may serve in gatekeeper roles for citizens in the community. The training is focused on suicide education and awareness, active listening, and role-play.

Population:

Community residents who may serve in gatekeeper roles (e.g., nurses, college resident advisors, school personnel)

Description of Evaluation:

The goals of the pilot study were to (1) study training outcomes, including skill development, from a brief gatekeeper training; (2) assess the feasibility of incorporating active learning principles (i.e., role-play practice) into standardized gatekeeper training; and (3) examine employee satisfaction with, and diffusion of information from gatekeeper training conducted in the workplace (e.g., sharing the training information with family/friends).

Summary of Evaluation Findings:

Pre- and postintervention analyses showed positive changes in participants’ knowledge about suicide and attitudes (self-efficacy) about intervening with suicidal individuals. A subset of participants engaged in role-play practice of gatekeeper skills after the training and rated the experience positively.

Sample Size:

n = 76 (n = 26 role-play participants; n = 30 agreed to a six-week follow-up)

Design:

Quasi-experimental

Reference:

Cross, Matthieu, et al., 2007

127 RAND Suicide Prevention Program Evaluation Toolkit

A.3. Evaluation Studies of Gatekeeper Training

Description of Program:

The Youth Suicide Prevention school program is a three-year pilot program that trains high school students (four-hour training course), school personnel, and parents (90-minute seminar) to recognize suicidal behavior among young people and to respond in a manner that is most likely to prevent a fatal outcome.

Population:

High school students

Description of Evaluation:

The evaluation involved a questionnaire administered to a sample of 189 students in eight schools prior to exposure to the four-hour training course. A post-test using the same instrument was administered following exposure to the course to a comparable sample of 181 additional students from the same schools. The researchers obtained feedback evaluations from an additional sample of students, school staff, and parents who completed the course.

Summary of Evaluation Findings:

Significant gains in the first two years of the program in understanding of youth suicide prevention techniques among high school students who took the course. School staff and parents taking the seminars indicated that they most appreciated receiving practical advice and resources that they could use in responding to a potentially suicidal young person.

Sample Size:

n = 189 students (181 at post-test); 390 program evaluators (218 students, 132 staff, 40 parents)

Design:

Nonexperimental (no control group)

Reference:

Nelson, 1987 school program Youth SuicideYouth Prevention

128 Appendix A: Evaluation Studies

A.3. Evaluation Studies of Gatekeeper Training

Description of Program: Teenagers for Program Prevention

The Curriculum-Based Suicide Prevention Program for Teenagers was a 1.5-hour Suicide Curriculum-Based program provided in a regular classroom by teachers who had previously received approximately six hours of training. It targeted a predominantly white suburban or rural student body and emphasized support networks in alleviating stress, confronting one’s peers, and taking advantage of community resources.

Population:

High school students

Description of Evaluation:

Students were assessed through a pretest and then again at a four- to six-week follow- up. The authors of this evaluation felt that four to six weeks was not sufficient to appropriately measure whether survey respondents were able to apply the knowledge learned.

Summary of Evaluation Findings:

The study failed to find convincing evidence of any program effect. There was also no evidence that the program had an effect on suicide attempt rates among exposed teens. Of the exposed group, 2.5% reported having made a (first) suicide attempt during the 18 months of follow-up, compared with 2.7% of the control group.

Sample Size:

Total n = 381; n = 174 received training and n = 207 in the control group who were available/willing after 18 months

Design:

Experimental

Reference:

Vieland et al., 1991

129 RAND Suicide Prevention Program Evaluation Toolkit

A.3. Evaluation Studies of Gatekeeper Training

Description of Program:

The Aboriginal Community Gatekeeper Training provided eight free, one-day gatekeeper training workshops to increase the ability of the local community to identify individuals at risk of suicide, mobilize local informal helping networks, and, where necessary, facilitate help-seeking behavior.

Population:

Members of the aboriginal community, i.e., health workers, youth workers, and education assistants

Description of Evaluation:

Paired sample t-tests were performed to assess whether there were changes in knowledge, intention to help, intention to refer to the mental health service, and confidence in identifying a suicidal person.

Summary of Evaluation Findings:

Evaluation of the workshops demonstrated an increase in participants’ knowledge about suicide, greater confidence in identifying people who are suicidal, and high levels of intention to provide help.

Sample Size:

n = 44 workshop attendees agreed to participate in the voluntary evaluation.

Design:

Quasi-experimental

Reference:

Capp et al., 2001 Gatekeeper Training Aboriginal Community

130 Appendix A: Evaluation Studies

A.3. Evaluation Studies of Gatekeeper Training

Description of Program:

The Samaritans of New York public education suicide awareness and prevention York New of Samaritans program is designed to train lay and professional staff in effective suicide prevention practices and how to “befriend” a person in crisis. Two three-hour training sessions were conducted in 2005.

Population:

Community- and school-based staff

Description of Evaluation:

The study aimed to examine baseline and post-training differences among participants who attended the program regarding their knowledge about suicide and suicide prevention and their ability to intervene with individuals at risk for suicide. The study also explored the influence of previous exposure to suicidal individuals, as well as other predictors of gains subsequent to training.

Summary of Evaluation Findings:

Results indicated increased self-efficacy after suicide prevention training. Trainees with higher levels of education and previous contact with suicidal individuals were significantly more likely to indicate gains in self-efficacy after training.

Sample Size:

n = 365

Design:

Quasi-experimental

Reference:

Clark et al., 2010

131 RAND Suicide Prevention Program Evaluation Toolkit

A.3. Evaluation Studies of Gatekeeper Training

Description of Program:

Raising Awareness of Personal Power (RAPP) is a curriculum-based suicide education program designed for high school and middle/junior high school students. RAPP educates students about depression and bipolar disorder, suicidal warning signs, a three-step process (i.e., Listen, Ask, and Take Action) for responding, and local resources. The program, generally presented by trained peer counselors and adult RAPP volunteers, includes a lecture and activities, such as interactive games, role- playing, and the analysis of several stories. The duration was one class period, which ranged from 50 to 75 minutes.

Population:

High school and middle/junior high students

Description of Evaluation:

The study used three methodological approaches: a rolling group design (RGD), an internal referencing strategy (IRS), and a minimum-competency approach (MC).

Summary of Evaluation Findings:

Findings for each approach were as follows:

1. The RGD approach found improvements in the knowledge, attitude, and efficacy variables that were significant for both the treatment and the control groups. 2. The IRS approach showed improvement on training materials, but there was little improvement on the untrained-yet-relevant materials. 3. The MC approach results showed that the RAPP program exceeded the minimum standards.

Sample Size:

n = 33

Design:

Quasi-experimental

Personal Power Reference: Raising Awareness of Cigularov et al., 2008

132 Appendix A: Evaluation Studies

A.3. Evaluation Studies of Gatekeeper Training

Description of Program:

Question, Persuade, Refer (QPR) is a universal gatekeeper training program intended Refer Persuade, Question, to improve school staff’s knowledge, appraisals (including willingness to assume a gatekeeper role), and self-reported suicide identification behaviors by students.

Population:

Staff in secondary schools

Description of Evaluation:

Staff from 20 middle schools and 12 high schools were randomized to either receive QPR or be assigned to the control group.

Summary of Evaluation Findings:

Intent-to-treat analyses showed that training increased school staff’s self-reported knowledge (effect size [ES] 0.41), appraisals of efficacy (ES 1.22), and service access (ES 1.07).

Sample Size:

102,000 students in the Cobb County (Marietta, Georgia) school district (10 middle and 6 high schools randomized to receive QPR and 10 middle and 6 high schools randomized to the control group)

Design:

Experimental (randomized control trial)

Reference:

Wyman, Brown, Inman, et al., 2008

133 RAND Suicide Prevention Program Evaluation Toolkit

A.4. Evaluation Studies of Marketing Campaigns

Description of Program:

The Air Force Suicide Prevention Campaign is a 30-minute briefing regularly required of all U.S. Air Force personnel that consists of 16 slides and four videos. It focuses on identifying warning signs, responding to those at risk, the impact on suicide survivors, and barriers to help-seeking behavior.

Population:

U.S. Air Force personnel

Description of Evaluation:

The study compared participants’ pre- and post-briefing surveys to assess the emotional impact of the briefing. Changes in the emotional status of suicidal and nonsuicidal subjects, as well as survivors and nonsurvivors, were also compared.

Summary of Evaluation Findings:

Suicidal female subjects showed a decrease in negative emotions. In general, there was a decrease in distress and an increase in calmness between pre- and post-briefing responses. There was no observed difference in the emotional changes between survivors and nonsurvivors.

Sample Size:

n = 286 subjects (173 nonsurvivors, 113 survivors; 249 nonsuicidal, 32 suicidal)

Design:

Quasi-experimental

Reference:

Bryan et al., 2009 Air Force Suicide Prevention Campaign

134 Appendix A: Evaluation Studies

A.4. Evaluation Studies of Marketing Campaigns

Description of Program:

Suicide Prevention Week is a yearly one-week media campaign that uses newspapers, Week Prevention Suicide radio, and television to change the behavior and attitudes of suicidal individuals and influence the public.

Population:

Men aged 20–40 in Quebec

Description of Evaluation:

Researchers surveyed a random sample of male Quebec residents regarding attitudes, knowledge, intentions, behaviors, and exposure to the campaign. Those who were not exposed to the campaign formed the control group. Data were also collected daily on the number of suicides and suicide attempts and the use of various resources.

Summary of Evaluation Findings:

Respondents who had exposure to Suicide Prevention Week demonstrated significantly more knowledge about suicide than those who had not, but no differences were observed in attitudes or help-seeking intentions. There was also no significant difference in observed behaviors, with the exception of suicide-related websites visited. No unintended negative effects of Suicide Prevention Week were observed.

Sample Size:

n = 1,020 respondents (190 test group, 830 control group)

Design:

Quasi-experimental

Reference:

Daigle et al., 2006

135 RAND Suicide Prevention Program Evaluation Toolkit

A.4. Evaluation Studies of Marketing Campaigns

Description of Program:

Youth Suicide: Recognizing the Signs is a video campaign intended to teach the signs of suicide and how to respond.

Population:

Parents of youth aged 10 or older

Description of Evaluation:

The evaluation included a pre- and post-test of knowledge of suicide, response to suicidality, perceptions of suicide, and intention to respond to suicidal individuals.

Summary of Evaluation Findings:

After the video participants’ knowledge, response, and intention to help improved.

Sample Size:

n = 112 parents

Design:

Nonexperimental (no control group)

Reference:

Maine, Shute, and Martin, 2001 Youth Suicide: Suicide: Youth Recognizing the Signs

136 Appendix A: Evaluation Studies

A.5. Evaluation Study of Means Restriction

Description of Program:

Means restriction intervention removed all charcoal packs from open shelves of major intervention restriction retail outlets in the intervention region for 12 months. means Kong Hong

Population:

Hong Kong residents

Description of Evaluation:

The evaluation compared charcoal-burning deaths between the intervention region and a nearby region where the means restriction intervention was not instituted. The evaluation was a double-blind trial (the removal of charcoal packs was not publicly announced, and public and frontline staff at supermarkets were not aware of the intervention). The control group was a region where charcoal packs were displayed as usual.

Summary of Evaluation Findings:

Suicide rates from charcoal burning decreased in the intervention region (p < 0.05) but not in the control region.

Sample Size:

Tuen Mun (intervention region) had 502,000 inhabitants. Yuen Long (control region) had 534,000 inhabitants.

Design:

Experimental, double-blind trial

Reference:

Yip et al., 2010

137 RAND Suicide Prevention Program Evaluation Toolkit

A.6. Evaluation Studies of Mental Health Interventions

Description of Program:

A 10-session cognitive therapy intervention program designed to prevent repeat suicide attempts among adults who recently attempted suicide.

Population:

Individuals who attempted suicide and who received a medical or psychiatric evaluation within 48 hours of the attempt

Description of Evaluation:

The evaluation used a randomized control trial design to assess the impact of the cognitive therapy as compared with enhanced usual care with tracking and referral services.

Summary of Evaluation Findings:

Cognitive therapy was effective in preventing suicide attempts among adults who recently attempted suicide. Participants in the therapy group had a significantly lower 2 reattempt rate (Wald X 1 = 3.9; p = 0.049) and were 50% less likely to reattempt suicide than participants in the usual care group (hazard ratio, 0.51; 95% confidence interval, 0.26–0.997). The severity of self-reported depression was significantly lower for the therapy group than for the usual care group at six months (p = 0.02), 12 months (p = 0.009), and 18 months (p = 0.046). The therapy group reported significantly less hopelessness than the usual care group at 6 months (p = 0.045).

Sample Size:

n = 120 individuals (60 to receive usual care; 60 to receive cognitive therapy and usual care)

Design:

Experimental (randomized control trial)

Reference:

G. Brown et al., 2005 intervention program 10-day cognitive therapy

138 Appendix A: Evaluation Studies

A.6. Evaluation Studies of Mental Health Interventions

Description of Program:

Multisystemic therapy (MST) is a home-based model of service provision/treatment therapy Multisystemic delivery grounded in social-ecological and systems theories and used to treat youth with mental health problems.

Population:

Youth referred for emergency psychiatric hospitalization (suicidal, homicidal, psychosis, or threat to self or others due to mental illness)

Description of Evaluation:

Multiple evaluation studies have been conducted on MST comparing it to hospitalization, by randomizing youth to receive one or the other.

Summary of Evaluation Findings:

The evaluations found that those who received MST exhibited

• Reductions in self-reported suicide attempts over the course of 16 months following recruitment (Huey et al., 2004) • Higher levels of satisfaction with treatment (Huey et al., 2004) • Reductions in externalizing symptoms and improvements in family functioning and school attendance (Henggeler et al., 1999; Rowland et al., 2005)

Also, 25% of youth randomized to MST were subsequently hospitalized during the two weeks following referral to MST (Schoenwald et al., 2000).

Sample Size:

Henggeler et al., 1999; Huey et al., 2004; and Schoenwald et al., 2000: n = 113 patients (57 MST, 56 hospitalization) Rowland et al., 2005: n = 31 patients (15 MST, 16 usual services)

Design:

Henggeler et al., 1999; Huey et al., 2004; and Schoenwald et al., 2000: Control group received hospitalization. Rowland et al., 2005: Control group received usual services.

Reference: Henggeler et al., 1999; Huey et al., 2004; Rowland et al., 2005; Schoenwald et al., 2000

139 RAND Suicide Prevention Program Evaluation Toolkit

A.6. Evaluation Studies of Mental Health Interventions

Description of Program:

RUSH is a ten-week treatment program based on dialectical behavior therapy that facilitates and encourages self-help and life enhancement skills through twice-weekly group skills training.

Population:

Offenders exhibiting borderline personality disorder characteristics

Description of Evaluation:

A pre-/post-test design evaluated participants’ level of stress, anxiety, and depression. Observational evaluations were made throughout the program.

Summary of Evaluation Findings:

The majority of participants experienced a decline in symptomology between the beginning and end of treatment—especially on the stress subscale. High levels of group cohesion, motivation, and commitment were observed.

Sample Size:

n = 29 patients

Design:

Nonexperimental (no control group)

Reference:

Eccleston and Sorbello, 2002 RUSH

140 Appendix A: Evaluation Studies

A.6. Evaluation Studies of Mental Health Interventions

Description of Program:

Problem-solving therapy is a short-term, secondary prevention approach designed therapy Problem-solving to decrease the intensity of suicidal ideation and associated psychopathology by bolstering problem-solving abilities and coping skills.

Population:

Participants were selected from the Psychology 100 Research Experience Program at Ohio State University and local campus and community service providers.

Description of Evaluation:

The evaluation assessed whether the intervention increased the use of positive coping mechanisms and decreased suicidal ideation, depression, and hopelessness.

Summary of Evaluation Findings:

Using a results classification system (maintained, improved, or stayed the same), the pattern of results indicated little substantial shifting at post-treatment (83% of control vs. 81% of treatment maintained; 13% vs. 13%, improved; and 4% vs. 6% worsened).

Sample Size:

n = 110

Design:

Those in the treatment group watched a 35-minute video focused on problem-solving and coping styles.

Reference:

Fitzpatrick, Witte, and Schmidt, 2005

141 RAND Suicide Prevention Program Evaluation Toolkit

A.6. Evaluation Studies of Mental Health Interventions

Description of Program:

The Suicide Prevention Program is a “mandated assessment” response to suicide threats and attempts that requires students at the University of Illinois to receive four sessions of professional assessment immediately following a suicide incident. The first of these weekly sessions is held within a week of the incident.

Population:

College students

Description of Evaluation:

The study compared the suicide rate among students at the University of Illinois during the program years to the rate before the program’s initiation. It also compared this trend to contemporary data from peer institutions that did not use the program.

Summary of Evaluation Findings:

The university observed a 45.3% reduction in the rate of suicide between the pre- intervention and intervention years. Furthermore, rates at the university were declining in the same years when the rates at peer institutions were steady or increasing. However, the program was not effective in reducing the rate of suicide among graduate students.

Sample Size:

One-group comparison: 1,041,951 students (274,942 control; 767,009 intervention); two-group comparison: 2,057,780 students (249,812 intervention; 1,807,968 control)

Design:

Quasi-experimental

Reference:

Joffe, 2008 the University of Illinois Suicide Prevention Program at

142 Appendix A: Evaluation Studies

A.6. Evaluation Studies of Mental Health Interventions

Description of Program:

LifeSPAN therapy is a cognitive therapy approach provided through the Early Psychosis

Prevention and Intervention Centre and includes eight to ten individual sessions. therapy LifeSPAN

Population:

Young people (aged 15–29)

Description of Evaluation:

The study evaluated the effectiveness of the LifeSPAN therapy through a randomized controlled trial.

Summary of Evaluation Findings:

The evaluation found that standard care reduces suicidality, and adding LifeSPAN therapy leads to an improvement in cognitive factors linked to suicide risk (e.g., hopelessness).

Sample Size:

n = 56 were randomized to either the

treatment group (n = 31) or a control group (n = 25). Thirteen of the 56 dropped out during the next ten weeks, and one dropped out before the six-month follow-up. This left 21 patients in each group (treatment and control) at the end of ten weeks.

Design:

Treatment group (n = 21) (LifeSPAN therapy plus standard clinical care) and control group (n = 21) (standard clinical care)

Reference:

Power et al., 2003

143 RAND Suicide Prevention Program Evaluation Toolkit

A.6. Evaluation Studies of Mental Health Interventions

Description of Program:

Dialectical behavior therapy (DBT) is a one-year behaviorally oriented outpatient psychotherapy approach for bipolar disorder patients with the goals of reducing self- injurious and life-threatening behaviors, reducing therapy-interfering behaviors, and reducing quality-of-life–interfering behaviors. It consists of weekly individual and group therapy sessions that include skill training and contingency management.

Population:

Suicide attempters meeting the criteria for borderline personality disorder

Description of Evaluation:

Multiple evaluation studies have been conducted comparing DBT to “treatment as usual” for parasuicidal patients with bipolar disorder and personality disorder over the course of a year. Measures from these studies include the number of parasuicidal acts, maintenance in therapy, amount of psychiatric inpatient treatment, depression, hopelessness, reasons for living, anger, social adjustment, work performance, and suicide ideation. Turner (2000) examined the comparative effectiveness of DBT and client-centered therapy (CCT).

Summary of Evaluation Findings:

The evaluations found that those who received DBT exhibited

• Reductions in the frequency and medical risk of parasuicidal behavior (Linehan, Armstrong, et al. 1991; Linehan, Heard, and Armstrong, 1993; Bohus et al., 2004) • Lower attrition rates compared with the control group (Linehan, Armstrong, et al., 1991) • Fewer days of inpatient hospitalization (Linehan, Armstrong, et al., 1991; Linehan, Heard, and Armstrong, 1993; Linehan, Comtois, et al., 2006; Bohus et al., 2004) compared with the control group • Less anger than the control group (Linehan, Heard, and Armstrong, 1993; Linehan, Tutek, et al., 1994); however, this was not replicated in Bohus et al., 2004) • Better self-reported social adjustment (Linehan, Heard, et al., 1993; Linehan, Tutek, et al., 1994)

therapy • Higher scores on the Global Assessment Scale (Linehan, Heard, et al., 1993; Linehan, Tutek, et al., 1994) Dialectical behavior • Better work performance than the control group (Linehan, Heard, et al., 1993)

144 Appendix A: Evaluation Studies

A.6. Evaluation Studies of Mental Health Interventions

Summary of Evaluation Findings (cont.):

• Fewer self-mutilating behaviors and self-damaging, impulsive behavior than the Dialectical behavior behavior Dialectical

control group (Bohus et al., 2004); however, Verheul et al. (2003) found that DBT is (cont.) therapy more effective for patients with high severity of self-mutilating acts than for low- severity patients

• Higher retention in therapy (Verheul et al., 2003).

There was mixed evidence as to whether DBT reduced depression, hopelessness, suicide ideation, and inability to find reasons for living. A 1991 study by Linehan, Armstrong, et al., found that DBT did not have any significant effects on these outcomes. However, a later study by Linehan, Comtois, et al. (2006) found that DBT significantly reduced suicide ideation and depression and improved reasons for living among patients. Turner (2000) found that DBT subjects showed more improvement than client-centered therapy subjects at both six and 12 months on their parasuicide rating, Brief Symptom Inventory, number of suicide attempts, and hospitalization days. At 12 months, DBT subjects showed more improvement in impulsiveness, anger, and depression. There was no difference in anxiety between treatments.

Sample Size:

Linehan, Armstrong, et al., 1991: n = 44 subjects (22 DBT, 22 control) Linehan, Heard, et al., 1993: n = 39 patients (19 DBT, 20 control) for PHI; 20 of these (9 DBT, 11 control) for all other assessments Linehan, Tutek, et al., 1994: n = 39 patients (19 DBT, 20 control) for the Parasuicide History Interview; 20 of these (nine DBT, 11 control) for all other assessments Linehan, Comtois, et al., 2006: n = 101 patients (52 DBT and 49 community treatment by experts) Turner, 2000: n = 24 patients (12 DBT, 12 CCT) Verheul et al., 2003: n = 58 patients (27 DBT, 31 treatment as usual) Bohus et al., 2004: n = 50 patients (31 DBT, 19 waiting list/usual treatment)

Design:

Bohus et al., 2004: Quasi-experimental All other studies: Experimental

Reference:

Linehan, Armstrong, et al., 1991; Linehan, Heard, et al., 1993; Linehan, Tutek, et al., 1994; Linehan, Comtois, et al., 2006; Turner, 2000; Verheul et al., 2003; Bohus et al., 2004

145 RAND Suicide Prevention Program Evaluation Toolkit

A.7. Evaluation Studies of Provider Training

Description of Program:

A nurse education program for nurses working in general hospitals aimed to increase nurses’ knowledge of suicide prevention, promote the adoption of a positive attitude by nurses in caring for patients at risk of suicide and their families, and enhance nurses’ competence in suicide prevention.

Population:

Registered general nurses

Description of Evaluation:

The evaluation used a mixed-methods design relying on a single group (convenience sample) for pretest and post-test analysis and focus group interviews.

Summary of Evaluation Findings:

Statistically significant positive changes were found between the pre- and post-test measures of participants’ attitudes and competence levels. Qualitative data showed that participants had applied the new knowledge in clinical practice.

Sample Size:

n = 54 nurses (18 in focus groups)

Design:

Nonexperimental (no control group)

Reference:

Chan, Chien, and Tso, 2009a, 2009b program Nurse education

146 Appendix A: Evaluation Studies

A.7. Evaluation Studies of Provider Training

Description of Program:

STORM is a one- to two-day training program conducted by mental health nurses that includes lectures, group discussions, role-playing and video feedback in the areas of assessment, crisis management, problem-solving, and crisis intervention. STORM

Population:

Mental health care providers

Description of Evaluation:

A pre-/post-test design evaluated participants acting as their own controls using questionnaires and interviews to measure changes in attitude, confidence, skills, competence, and satisfaction.

Summary of Evaluation Findings:

The training resulted in significant positive changes in attitudes and confidence, but the study found no statistically significant changes in skill acquisition.

Sample Size:

n = 458 (baseline), 458 (immediately post-training), 143 (four-month post-training)

Design:

Nonexperimental (one group pre-/post-test)

Reference:

Gask et al., 2006

147 RAND Suicide Prevention Program Evaluation Toolkit

A.7. Evaluation Studies of Provider Training

Description of Program:

CALM is a two-hour workshop for mental health care providers to learn how reducing access to lethal means can prevent suicide. The training includes presentations, a video screening, instruction, discussion, and role-playing in addition to written materials.

Population:

Mental health care providers

Description of Evaluation:

Participants attended CALM training and immediately answered a questionnaire comparing their pre-CALM and post-CALM attitudes and beliefs. This post-test was compared to the follow-up responses of the same group six to eight weeks later.

Summary of Evaluation Findings:

Participating in CALM led to a significant increase in commitment and skills for counseling using means reduction. These beliefs and attitudes were sustained in the weeks following the program. However, there was a significant decrease in perceptions of the effectiveness of means reduction between post-test and follow-up.

Sample Size:

168 (post-test); 111 (follow-up)

Design:

Nonexperimental (one group pre-/post-test)

Reference:

Johnson et al., 2011 CALM

148 Appendix A: Evaluation Studies

A.7. Evaluation Studies of Provider Training

Description of Program:

The Gotland Educational Program was directed at all general practitioners on the Gotland Educational Educational Gotland island of Gotland, Sweden, and consisted of two two-day sessions of oral and written

information, group work, case reports, and discussions focused on suicide prevention. Program

Population:

General practitioners

Description of Evaluation:

The study compared baseline measures, prior to the program, with trends in suicide rates, prescription rates, sick leave, inpatient care, and general practitioner attitudes over the next five years on Gotland.

Summary of Evaluation Findings:

General practitioners reported an increase in competency, knowledge, and support of comprehensive treatment. Additionally, the suicide rate, the number of psychiatric referrals, and depression-related sick leave, and inpatient care decreased on Gotland following the program. Lithium and antidepressant prescriptions also increased.

Sample Size:

Not reported

Design:

Nonexperimental (one group pre-/post-test)

Reference:

Rutz, 2001

149 RAND Suicide Prevention Program Evaluation Toolkit

A.7. Evaluation Studies of Provider Training

Description of Program:

The suicide prevention education program in Slovenia is a four-hour educational program for primary care physicians focused on recognition and treatment of depression and suicide prevention.

Population:

Primary care doctors who had written at least five prescriptions of antidepressants at the baseline period

Description of Evaluation:

The study was conducted in three neighboring regions in Slovenia (Celje, Ravne na Koroškem, and Podravska). The study compared prescription rates in the regions for antidepressants and anxiolytics before and after the intervention. The study also compared suicide rates three years before and after the intervention.

Summary of Evaluation Findings:

The training program was beneficial for primary care physicians’ ability to recognize and manage depression. However, there was no significant decrease in local suicide rates.

Sample Size:

Physicians from Celje and Ravne na Koroškem (n = 82; educational group) attended the program, and the other half (n = 108; control group 1) and physicians from the Podravska region (n = 164; control group 2) did not attend the program.

Design:

The intervention group received the training; the control group did not.

Reference:

Roskar et al., 2010 education program Slovenian suicide prevention prevention suicide Slovenian

150 Appendix A: Evaluation Studies

A.8. Evaluation Studies of Screening Programs

Description of Program: program for the elderly the for program A participatory screening program included (1) public health education from 1991 to screening Participatory 2000 and (2) screening for depression with follow-up from 1991 to 1997, using public health and primary care resources in the town with the assistance of the psychiatric care resources in the neighboring county.

Population:

Elderly residents (65 years and older)

Description of Evaluation:

Quasi-experimental design with intervention and reference municipalities were used to estimate changes in the risk of dying by suicide before and after the ten-year implementation.

Summary of Evaluation Findings:

The risk for women in the intervention area was reduced by 64%, whereas there was no significant change in the risk for men in the intervention area or for either men or women in the reference area. The risk reduction for women in the intervention area was greater than the secular trend.

Sample Size:

Oyama et al., 2006: approx. 1,400 (426 residents in the intervention town, 1,000 in the reference town) Scott et al., 2009: n = 1,729 students completed a questionnaire; 356 who screened positive and 285 who screened negative completed the Diagnostic Interview Schedule for Children

Design:

Oyama et al., 2006: Quasi-experimental Scott et al., 2009: A sample of those who screened negative also completed the Diagnostic Interview Schedule for Children

Reference:

Oyama et al., 2006; Scott et al., 2009

151 RAND Suicide Prevention Program Evaluation Toolkit

A.8. Evaluation Studies of Screening Programs

Description of Program:

The Columbia Health Screen (CHS) and Post-Screening Structured Interview is a 14-item questionnaire that includes questions on reporting needing help with an emotional problem. Youth who screened positive on the CHS were referred for further mental health screening.

Population:

High school students

Description of Evaluation:

The study randomized classrooms to receive the screening or not and then assessed differences in referrals to mental health services among those identified via screening versus those identified via usual process (e.g., parents, teachers).

Summary of Evaluation Findings:

Screened students were 21 times more likely to receive a referral for mental health services.

Sample Size:

Husky, McGuire, et al., 2009: n = 483 students offered screening Husky, Miller, et al., 2011: n = 890 (467 received screening, 423 in control group

Design:

Husky, McGuire, et al., 2009: Nonexperimental (no control group) Husky, Miller, et al., 2011: Classrooms were randomized to screening condition or control

Reference:

Husky, McGuire, et al., 2009; Husky, Miller, et al., 2011 Structured Interview CHS and Post-Screening

152 Appendix A: Evaluation Studies

A.8. Evaluation Studies of Screening Programs

Description of Program:

The Columbia Suicide Screen (CSS) is an 11-item questionnaire embedded in a health Screen Suicide Columbia survey administered during regular class time.

Population: High school students

Description of Evaluation: Several evaluation studies have assessed the CSS. In all studies, youth who screened positive and a control group were reassessed with the Diagnostic Interview Schedule for Children, and school staff were also asked to assess all youths who completed the instrument for emotional problems. Scott et al. (2010) and Shaffer, Scott, et al. (2004) also tested optimal positive-screen algorithm using sensitivity, specificity, positive- predictive value.

Summary of Evaluation Findings: 34% of those with mental health problems were identified by screening only, 13% were identified by school officials, and 35% were identified by both. Of those, 18% with mental health problems were not identified by either (Scott et al., 2009). Three screening algorithms were also tested and identified 96%, 92%, and 89% of youth with mental health problems (corresponding positive screens are 35%, 24%, 17%; Scott et al., 2010). The best algorithm relied on suicide ideation or previous attempt and a score greater than or equal to 3 for the screening questions about whether they were unhappy, withdrawing, irritable, and anxious (Shaffer, Scott, et al., 2004).

Sample Size: Scott et al., 2009, 2010: n = 1,729 students completed the questionnaire; 356 who screened positive and 285 who screened negative completed the DISC Shaffer, Scott, et al., 2004: n = 1,729 students completed questionnaire; 489 who screened positive and 356 who screened negative completed the DISC

Design: Scott et al., 2009, 2010: A sample of those who screened negative also completed the Diagnostic Interview Schedule for Children Shaffer, Scott, et al., 2004: A matched sample of those who screened negative also completed the DISC

Reference: Scott et al., 2009, 2010; Shaffer, Scott, et al., 2004

153 RAND Suicide Prevention Program Evaluation Toolkit

A.8. Evaluation Studies of Screening Programs

Description of Program:

Diagnostic Predictive Scales–8 (DPS-8) is an 84-item computerized, voluntary, self- report assessment offered to youth at no cost. The DPS-8 screens for mental health problems, including suicide.

Population:

Adolescents seen in a pediatric health center

Description of Evaluation:

Clinicians reviewed the results of the screening at a visit, and the study assessed whether youth who screened positive received pediatric mental health care or were referred to specialty mental health care.

Summary of Evaluation Findings:

Fourteen percent of youth who completed the scale screened positive for one or more mental disorders. Screening was associated with an increased likelihood of receiving pediatric mental health care or referral for specialty mental health care

Sample Size:

n = 483 patients offered screening

Design:

Nonexperimental (no control group)

Reference:

Husky, Kaplan, et al., 2011 (DPS-8) Diagnostic Predictive Scales–8

154 Appendix A: Evaluation Studies

A.8. Evaluation Studies of Screening Programs

Description of Program:

The Universal Youth Suicide Screening program screens high school students for Youth Suicide Universal mental health problems and suicide risk. program Screening

Population:

High school students

Description of Evaluation:

Classes at six high schools were randomized to either the experimental or control group. The experimental group was surveyed on mood, depression, drug use, suicide ideation, suicide history, while the control group was surveyed on just mood, depression, and drug use (no suicide questions). Two days later, both groups were reassessed on mood, depression, suicide history, and ideation.

Summary of Evaluation Findings:

The two groups did not differ significantly in distress levels or suicidal ideation. Furthermore, there was no evidence of an iatrogenic effect of asking about suicide, even among high-risk subjects.

Sample Size:

n = 2,342 students (1,172 experimental, 1,170 control)

Design:

Experimental

Reference:

Gould, Marrocco, et al., 2005

155 RAND Suicide Prevention Program Evaluation Toolkit

A.9. Evaluation Studies of Coping Skills and Self-Referral Training

Description of Program:

The Personal Growth Class is a semester-long class involving a small-group work component that focuses on social support, weekly monitoring of activities targeting changes in mood management, school performance and attendance, drug involvement, and life skills training in self-esteem enhancement, decisionmaking, personal control, and interpersonal communication.

Population:

High school students

Description of Evaluation:

The evaluation compared youth in the class to a random sample of youth not at risk for school failure over time.

Summary of Evaluation Findings:

The intervention resulted in improvements in outcomes (suicide risk behaviors, depression, hopelessness, stress, anger, self-esteem, and social network support) among the intervention group as compared with the control group. However, no differences in personal control were detected.

Sample Size:

Not reported

Design:

There was both an assessment-only condition and a random sample of youth not at risk for school failure to serve as a normative comparison group.

Reference:

Eggert et al., 1995 Personal Growth Class

156 Appendix A: Evaluation Studies

A.9. Evaluation Studies of Coping Skills and Self-Referral Training

Description of Program:

Surviving the Teens Suicide Prevention and Depression Awareness Program is a four- session classroom intervention designed to increase knowledge, decrease the stigma of Teens the Surviving depression and other mental health disorders, improve coping, increase help-seeking, increase the likelihood of depressed teens seeking help, increase family and school connectedness, and decrease suicidal and other risk-taking behaviors.

Population:

High school students

Description of Evaluation:

A pre- and post-tests were used to examine the effect of the intervention on suicide ideation, suicidal behavior, self-efficacy, and behavioral intent regarding help-seeking.

Summary of Evaluation Findings:

The evaluation found that youth participants exhibited reductions in considering suicide, making a suicide plan, and attempting suicide, as well as increases in self- efficacy and behavioral intentions toward help-seeking.

Sample Size:

n = 1,030 students

Design:

Nonexperimental (no control group)

Reference:

K. King, Strunk, and Sorter, 2011

157 RAND Suicide Prevention Program Evaluation Toolkit

A.9. Evaluation Studies of Coping Skills and Self-Referral Training

Description of Program:

A problem-solving intervention was designed to improve coping and problem-solving to decrease suicidal ideation and behavior. The program included an experiential- affective group, psycho-educational classes, and an extended problem-solving group. Treatment was group-based and outpatient but administered in a partial or day hospital venue.

Population:

Suicidal adolescents and young adults

Description of Evaluation:

Suicidal patients were randomly assigned to the experimental group vs. treatment as usual. Treatment as usual involved a combination of inpatient and outpatient care.

Summary of Evaluation Findings:

The evaluation found no group differences at one, six, or 12 months. Both groups improved over time in terms of symptoms, problem-solving, suicidality, and other outcomes. However, the experimental group had less attrition than control group.

Sample Size:

n = 264 (143 experimental, 121 control)

Design:

Experimental with random assignment

Reference:

Rudd, Rajab, and Orman, 1996 Problem-solvingintervention

158 Appendix A: Evaluation Studies

A.9. Evaluation Studies of Coping Skills and Self-Referral Training

Description of Program:

Reaching Young Europe was based on the Reaching Young People program and Young Europe Reaching consisted of 25 weekly one-hour sessions focused on increasing coping skills and resilience.

Population:

Young children (kindergarten through second grade)

Description of Evaluation:

The evaluation assessed the program’s implementation through weekly reports from program coordinators and program facilitators’ reports of participation and appreciation after each session. The evaluation of program effects assessed changes in social skills, including assertion and empathy, as well as changes in coping skills.

Summary of Evaluation Findings:

According to the authors, the results showed that the program could be successfully implemented, and some activities were appreciated by certain age groups more than others. They also reported that the evaluation showed significant improvement in overall social skills (teacher-reported) among first graders, assertion among second graders, and empathy among kindergarteners and first graders. The program had no significant effects on coping skills.

Sample Size:

Evaluation of program implementation: two kindergarten groups, one after-school club, and 12 grade school classes. Evaluation of program effects: 214 participating children and 109 control children.

Design:

Implementation evaluation: no control group. Evaluation of effects: control group, not randomized

Reference:

Mishara and Ystgaard, 2000

159 RAND Suicide Prevention Program Evaluation Toolkit

A.9. Evaluation Studies of Coping Skills and Self-Referral Training

Description of Program:

The program involved sending letters to patients who refused continued interaction with the health care system. Letters were sent monthly for four months, then every two months for eight months, and finally every three months for four years—a total of five years and 24 contacts. Content of the letters was personalized but always contained the sentiment that “It has been some time since you were here at the hospital. . . . If you wish to drop us a note we would be glad to hear from you.”

Population:

Patients who had refused ongoing care after being hospitalized because of a depressive or suicidal state

Description of Evaluation:

The study assessed differences in suicide rates among 3,005 patients five and 15 years after hospital discharge. The study assessed differences by whether patients accepted or declined ongoing treatment and, among those who refused treatment, whether they were periodically contacted by letter.

Summary of Evaluation Findings:

Five years after initial discharge from the hospital, patients in the group that refused treatment and then were contacted by letter had a lower suicide rate (3.9%) than patients who chose continued treatment (6.2%), and patients who refused treatment and were not contacted by letter (4.6%). Fifteen years after initial discharge from the hospital, patients who refused treatment and were not contacted by letter had the lowest suicide rates (5.7%), followed by patients who refused treatment and were contacted by letter (6.4%), and patients who chose to continue treatment (8.2%).

Sample Size:

3,005 individuals were divided into three groups: those who continued to seek treatment (n = 1,939); those who refused treatment and were randomized to receive letters (n = 389); and those who refused treatment and were randomized into the control group (i.e., did not receive letters; n = 454). An additional 223 individuals were not included in the study because it was unclear whether they were seeking treatment.

Design:

Follow-up letters Experimental (randomized control trial)

Reference:

Motto and Bostrom, 2001

160 Appendix B: Glossary

Appendix B Glossary of Terms

Evidence-based program A program that has been determined to be effective through rigorous scientific evaluations and randomized controlled trials, has a significant and sustained effect on intended outcomes, and has been tested using large longitudinal studies or multiple replications (Evidence-Based Pre- vention and Intervention Support Center, 2011).

Fidelity Adherence of implementation to a program’s original design (Smith, Daunic, and Taylor, 2007).

Logic model A graphical depiction of the rationale and expectations of a program (Leviton et al., 2010). A logic model clarifies the causal relationships among program resources, activities, and outcomes (McLaughlin and Jordan, 1999; Wholey, Hatry, and Newcomer, 2010).

Outcomes Changes or benefits resulting from activities and outputs. Typically, programs have short-, intermediate-, and long-term outcomes (Leviton, et al., 2010; Wholey, Hatry, and Newcomer, 2010).

Outcome evaluation An assessment of how well the program’s activities or services have brought about expected changes in the target population or social condition (Rossi, Lipsey, and Freeman, 2004).

Outputs The products, goods, and services provided to the program’s participants (Wholey, Hatry, and Newcomer, 2010).

Process evaluation A form of program evaluation designed to document and analyze the early development and actual implementation of a program, assessing whether and how well services are delivered as intended or planned. Also known as implementation assessment (Wholey, Hatry, and Newcomer, 2010; Rossi, Lipsey, and Freeman, 2004).

161 RAND Suicide Prevention Program Evaluation Toolkit

Program A set of activities, tied together through shared resources (e.g., staff, funding, space, materials), meant to influence a targeted population’s knowledge, attitudes, or behavior to accomplish a specific goal or goals.

Theory of change A model that describes the mechanisms through which the initiative’s inputs and activities are thought to lead to desired outcomes (Leviton et al., 2010).

162 Bibliography

Bibliography

Abbey, Karen J., Charles H. Madsen, Jr., and Roland Polland, “Short-Term Suicide Awareness Curriculum,” Suicide and Life-Threatening Behavior, Vol. 19, No. 2, Summer 1989, pp. 216–227. Acosta, Joie, and Matthew Chinman, “Building Capacity for Outcomes Measurement in Human Service Organizations: The Getting To Outcomes Method,” in Jennifer L. Magnabosco and Ronald W. Manderscheid, eds., Outcomes Measurement in the Human Services: Cross-Cutting Issues and Methods in the Era of Health Reform, Washington, D.C.: NASW Press, 2011. Alexopoulos, George S., Ira R. Katz, Martha L. Bruce, Moonseong Heo, Thomas Ten Have, Patrick Raue, Hillary R. Bogner, Herbert C. Schulberg, Benoit H. Mulsant, and Charles F. Reynolds III, “Remission in Depressed Geriatric Primary Care Patients: A Report from the PROSPECT Study,” American Journal of Psychiatry, Vol. 162, No. 4, April 2005, pp. 718–724. Alexopoulos, George S., Charles F. Reynolds III, Martha L. Bruce, Ira R. Katz, Patrick J. Raue, Benoit H. Mulsant, David W. Oslin, and Thomas Ten Have, “Reducing Suicidal Ideation and Depression in Older Primary Care Patients: 24-Month Outcomes of the PROSPECT Study,” American Journal of Psychiatry, Vol. 166, No. 8, August 2009, pp. 882–890. Althaus, David, and Ulrich Hegerl, “The Evaluation of Suicide Prevention Activities: State of the Art,” World Journal of Biological Psychiatry, Vol. 4, No. 4, October 2003, pp. 156–165. Angerstein, Gearld, Suzanne Linfield-Spindler, and Laura Payne, “Evaluation of an Urban School Adolescent Suicide Program,” School Psychology International, Vol. 12, Nos. 1–2, 1991, pp. 25–48. Aoun, S., and L. Johnson, “A Consumer’s Perspective of a Suicide Intervention Programme,” Australian and New Zealand Journal of Mental Health Nursing, Vol. 10, No. 2, June 2001, pp. 97–104. Appleby, L., R. Morriss, L. Gask, M. Roland, B. Perry, A. Lewis, L. Battersby, N. Colbert, G. Green, T. Amos, L. Davies, and B. Faragher, “An Educational Intervention for Front-Line Health Professionals in the Assessment and Management of Suicidal Patients (The STORM Project),” Psychological Medicine, Vol. 30, No. 4, July 2000, pp. 805–812. Arntz, Arnoud, Marije van den Hoorn, Jurgen Cornelis, Roel Verheul, Weis M. C. van den Bosch, and Arthur J. H. T. de Bie, “Reliability and Validity of the Borderline Personality Disorder Severity Index,” Journal of Personality Disorders, Vol. 17, No. 1, 2003, pp. 45–59. Aseltine, Robert H., Jr., “An Evaluation of a School Based Suicide Prevention Program,” Adolescent and Family Health, Vol. 3, No. 2, Summer 2003, pp. 81–88. Aseltine, Robert H., Jr., and Robert DeMartino, “An Outcome Evaluation of the SOS Suicide Prevention Program,” American Journal of Public Health, Vol. 94, No. 3, March 2004, pp. 446–451. Aseltine, Robert H., Jr., Amy James, Elizabeth A. Schilling, and Jaime Glanovsky, “Evaluating the SOS Suicide Prevention Program: A Replication and Extension,” BMC Public Health, Vol. 7, No. 1, July 2007. Auerbach, Stephen M., and Peter R. Kilmann, “Crisis Intervention: A Review of Outcome Research,” Psychological Bulletin, Vol. 84, No. 6, 1977, pp. 1189–1217. Bean, Gretchen, and Kristine M. Baber, “Connect: An Effective Community-Based Youth Suicide Prevention Program,” Suicide and Life-Threatening Behavior, Vol. 41, No. 1, February 2011, pp. 87–97.

163 RAND Suicide Prevention Program Evaluation Toolkit

Beautrais, Annette L., “National Strategies for the Reduction and Prevention of Suicide,” Crisis: The Journal of Crisis Intervention and Suicide Prevention, Vol. 26, No. 1, 2005, pp. 1–3. Beautrais, Annette L., Sheree J. Gibb, Alan Faulkner, David M. Fergusson, and Roger T. Mulder, “Postcard Intervention for Repeat Self-Harm: Randomised Controlled Trial,” British Journal of Psychiatry, Vol. 197, No. 1, July 2010, pp. 55–60. Beck, Aaron T., Gregory K. Brown, and Robert A. Steer, “Psychometric Characteristics of the Scale for Suicide Ideation with Psychiatric Outpatients,” Behaviour Research and Therapy, Vol. 35, No. 11, November 1997, pp. 1039–1046. Beck, A. T., C. H. Ward, M. Mendelson, J. Mock, and J. Erbaugh, “An Inventory for Measuring Depression,” Archives of General Psychiatry, Vol. 4, June 1961, pp. 561–571. Beck, A. T., A. Weissman, D. Lester, and L. Trexler, “The Measurement of Pessimism: The Hopelessness Scale,” Journal of Consulting and Clinical Psychology, Vol. 42, No. 6, 1974, p. 861. Bidwell, R. E., J. J. Bidwell, and S. Y. Tsai, “Suicide Prevention: A Study of the Community Program in Kansas City,” Journal of the Kansas Medical Society, Vol. 72, No. 4, April 1971, pp. 167–173 passim. Bohus, Martin, Brigitte Haaf, Timotht Simms, Matthias F. Limberger, Christian Schmahl, Christine Unckel, Klaus Lieb, and Marsha M. Linehan, “Effectiveness of Inpatient Dialectical Behavioral Therapy for Borderline Personality Disorder: A Controlled Trial,” Behaviour Research and Therapy, Vol. 42, No. 5, May 2004, pp. 487–499. Brent, David A., and Jeffrey Bridge, “Firearms Availability and Suicide: Evidence, Interventions, and Future Directions,” American Behavioral Scientist, Vol. 46, No. 9, May 2003, pp. 1192–1210. Breton, Jean-Jaques, Richard Boyer, Henriette Bilodeau, Sylvaine Raymond, Natacha Joubert, and Marie-Andrée Nantel, “Is Evaluative Research on Youth Suicide Programs Theory-Driven? The Canadian Experience,” Suicide and Life-Threatening Behavior, Vol. 32, No. 2, Summer 2002, pp. 176–190. Briggs, Stephen, Liz Webb, Jonathan Buhagiar, and Gaby Braun, “Maytree: A Respite Center for the Suicidal: An Evaluation,” Crisis: The Journal of Crisis Intervention and Suicide Prevention, Vol. 28, No. 3, 2007, pp. 140–147. Britt, Thomas W., Tiffany M. Greene-Shortridge, Sarah Brink, Quyen B. Nguyen, Jaclyn Rath, Anthony L. Cox, Charles W. Hoge, and Carl Andrew Castro, “Perceived Stigma and Barriers to Care for Psychological Treatment: Implications for Reactions to Stressors in Different Contexts,” Journal of Social and Clinical Psychology, Vol. 27, No. 4, 2008, pp. 317–335. Broadhead, W. Eugene, Andrew C. Leon, Myrna M. Weissman, James E. Barrett, Robert S. Blacklow, Thomas T. Gilbert, Martin B. Keller, Mark Olfson, and Edmund S. Higgins, “Development and Validation of the SDDS-PC Screen for Multiple Mental Disorders in Primary Care,” Archives of Family Medicine, Vol. 4, No. 3, March 1995, pp. 211–219. Brown, Gregory K., Thomas Ten Have, Gregg R. Henriques, Sharon X. Xie, Judd E. Hollander, and Aaron T. Beck, “Cognitive Therapy for the Prevention of Suicide Attempts: A Randomized Controlled Trial,” Journal of the American Medical Association, Vol. 294, No. 5, August 3, 2005, pp. 563–570. Brown, Melanie M., and Julie Goldstein Grumet, “School-Based Suicide Prevention with African American Youth in an Urban Setting,” Professional Psychology, Research and Practice, Vol. 40, No. 2, April 2009, pp. 111–117. Bruce, Martha L., Thomas Ten Have, Charles F. Reynolds III, Ira I. Katz, Herbert C. Schulberg, Benoit H. Mulsant, Gregory K. Brown, Gail J. McAvay, Jane L. Pearson, and George S. Alexopoulos, “Reducing Suicidal Ideation and Depressive Symptoms in Depressed Older Primary Care Patients: A Randomized Controlled Trial,” Journal of the American Medical Association, Vol. 291, No. 9, March 3, 2004, pp. 1081–1091. Bryan, Craig J., Luther E. Dhillon-Davis, and Kieran K. Dhillon-Davis, “Emotional Impact of a Video-Based Suicide Prevention Program on Suicidal Viewers and Suicide Survivors,” Suicide and Life-Threatening Behavior, Vol. 39, No. 6, December 2009, pp. 623–632. Campbell, Frank R., Louis Cataldie, John McIntosh, and Kari Millet, “An Active Postvention Program,” Crisis: The Journal of Crisis Intervention and Suicide Prevention, Vol. 25, No. 1, 2004, pp. 30–32.

164 Bibliography

Capp, Kim, Frank P. Deane, and Gordon Lambert, “Suicide Prevention in Aboriginal Communities: Application of Community Gatekeeper Training,” Australian and New Zealand Journal of Public Health, Vol. 25, No. 4, August 2001, pp. 315–321. Cerel, Julie, and Frank Campbell, “Suicide Survivors Seeking Mental Health Services: A Preliminary Examination of the Role of an Active Postvention Model,” Suicide and Life-Threatening Behavior, Vol. 38, No. 1, February 2008, pp. 30–34. Chagnon, François, Janie Houle, Isabelle Marcoux, and Johanne Renaud, “Control-Group Study of an Intervention Training Program for Youth Suicide Prevention,” Suicide and Life-Threatening Behavior, Vol. 37, No. 2, April 2007, pp. 135–144. Chan, Sally Wai-chi, Wai-tong Chien, and Steve Tso, “The Qualitative Evaluation of a Suicide Prevention and Management Programme by General Nurses,” Journal of Clinical Nursing, Vol. 17, No. 21, November 2008, pp. 2884–2894. ———, “Evaluating Nurses’ Knowledge, Attitude and Competency After an Education Programme on Suicide Prevention,” Nurse Education Today, Vol. 29, No. 7, October 2009a, pp. 763–769. ———, “Provision and Evaluation of a Suicide Prevention and Management Programme by Frontline Nurses in Hong Kong,” Hong Kong Medical Journal, Vol. 15, Suppl. 6, October 2009b, pp. 4–8. Chan, Sandra S., Vivian P. Y. Leung, Joshua Tsoh, S. W. Li, C. S. Yu, Gabriel K. K. Yu, T. K. Poon, P. C. Pan, W. F. Chan, Yeates Conwell, Linda C. Lam, and Helen F. K. Chiu, “Outcomes of a Two-Tiered Multifaceted Elderly Suicide Prevention Program in a Hong Kong Chinese Community,” American Journal of Geriatric Psychiatry, Vol. 19, No. 2, February 2011, pp. 185–196. Chapman, S., P. Alpers, K. Agho, and M. Jones, “Australia’s 1996 Gun Law Reforms: Faster Falls in Firearm Deaths, Firearm Suicides, and a Decade Without Mass Shootings,” Injury Prevention, Vol. 12, No. 6, 2006, pp. 365–372. Chinman, Matthew, Joie Acosta, Patricia Ebener, Q Burkhart, Michael Clifford, Maryann Corsello, Tim Duffey, Sarah Hunter, Margaret Jones, Michael Lahti, Patrick S. Malone, Susan Paddock, Andrea Phillips, Susan Savell, Peter C. Scales, and Nancy Tellett-Royce, “Establishing and Evaluating the Key Functions of an Interactive Systems Framework Using an Assets–Getting To Outcomes Intervention,” American Journal of Community Psychology, forthcoming. ———, “Intervening with Practitioners to Improve the Quality of Prevention: One Year Findings from a Randomized Trial of Assets–Getting To Outcomes,” Journal of Primary Prevention, forthcoming. Ciffone, Jerry, “Suicide Prevention: A Classroom Presentation to Adolescents,” Social Work, Vol. 38, No. 2, March 1993, pp. 197–203. ———, “Suicide Prevention: An Analysis and Replication of a Curriculum-Based High School Program,” Social Work, Vol. 52, No. 1, January 2007, pp. 41–49. Cigularov, Konstantin, Peter Chen, Beverly W. Thurber, and Lorann Stallones, “Investigation of the Effectiveness of a School-Based Suicide Education Program Using Three Methodological Approaches,” Psychological Services, Vol. 5, No. 3, August 2008, pp. 262–274. Clark, Tanisha R., Monica M. Matthieu, Alan Ross, and Kerry L. Knox, “Training Outcomes from Samaritans of New York Suicide Awareness and Prevention Programme Among Community- and School- Based Staff,” British Journal of Social Work, Vol. 40, No. 7, October 2010, pp. 2223–2238. Cooper-Patrick, Lisa, Rosa M. Crum, and Daniel E. Ford, “Identifying Suicidal Ideation in General Medical Patients,” Journal of the American Medical Association, Vol. 272, No. 22, December 14, 1994, pp. 1757–1762. Cross, Wendi, Monica M. Matthieu, Julie Cerel, and Kerry L. Knox, “Proximate Outcomes of Gatekeeper Training for Suicide Prevention in the Workplace,” Suicide and Life-Threatening Behavior, Vol. 37, No. 6, 2007, pp. 659–670. Cross, Wendi F., David Seaburn, Danette Gibbs, Karen Schmeelk-Cone, Ann Marie White, and Eric D. Caine, “Does Practice Make Perfect? A Randomized Control Trial of Behavioral Rehearsal on Suicide Prevention Gatekeeper Skills,” Journal of Primary Prevention, Vol. 32, Nos. 3–4, August 2011, pp. 195–211.

165 RAND Suicide Prevention Program Evaluation Toolkit

Cusimano, Michael D., and Mojib Sameem, “The Effectiveness of Middle and High School–Based Suicide Prevention Programmes for Adolescents: A Systematic Review,” Injury Prevention, Vol. 17, No. 1, February 2011, pp. 43–49. Daigle, Marc, Louise Beausoleil, Jacques Brisoux, Sylvaine Raymond, Lucie Charbonneau, and Julie Desaulniers, “Reaching Suicidal People with Media Campaigns: New Challenges for a New Century,” Crisis: The Journal of Crisis Intervention and Suicide Prevention, Vol. 27, No. 4, 2006, pp. 172–180. Dedic, Gordana J., and Panic Milivoje, “Suicide Prevention Program in the Army of Serbia and Montenegro,” Military Medicine, Vol. 172, No. 5, May 2007, pp. 551–555. Defense Centers of Excellence for Psychological Health and Traumatic Brain Injury, Program Evaluation Guide, Arlington, Va., July 2012. As of March 19, 2013: http://www.dcoe.health.mil/PsychologicalHealth/Program_Evaluation.aspx Derogatis, Leonard R., and Patricia A. Cleary, “Confirmation of the Dimensional Structure of the SCL-90: A Study in Construct Validation,” Journal of Clinical Psychology, Vol. 33, No. 4, October 1977, pp. 981–989. Deykin, E. Y., C. C. Hsieh, N. Joshi, and J. J. McNamarra, “Adolescent Suicidal and Self-Destructive Behavior: Results of an Intervention Study,” Journal of Adolescent Health Care, Vol. 7, No. 2, March 1986, pp. 88–95. Diekstra, René F., “The Prevention of Suicidal Behavior: Evidence for the Efficacy of Clinical and Community-Based Programs,” International Journal of Mental Health, Vol. 21, No. 3, Fall 1992, pp. 69–87. Dietrich, Sandra, Roland Mergl, Philine Freudenberg, David Althaus, and Ulrich Hegerl, “Impact of a Campaign on the Public’s Attitudes Towards Depression,” Health Education Research, Vol. 25, No. 1, February 2010, pp. 135–150. Eccleston, Lynne, and Laura Sorbello, “The RUSH Program—Real Understanding of Self-Help: A Suicide and Self-Harm Prevention Initiative Within a Prison Setting,” Australian Psychologist, Vol. 37, No. 3, November 2002, pp. 237–244. Eggert, Leona L., and Jerald R. Herting, “Preventing Teenage Drug Abuse: Exploratory Effects of Network Social Support,” Youth and Society, Vol. 22, No. 4, June 1991, pp. 482–524. Eggert, Leona L., Christine D. Seyi, and Liela J. Nicholas, “Effects of a School-Based Prevention Program for Potential High School Dropouts and Drug Abusers,” Substance Use and Misuse, Vol. 25, No. 7, 1990, pp. 773–801. Eggert, Leona L., Elaine A. Thompson, Jerald R. Herting, and Liela J. Nicholas, “Preventing Adolescent Drug Abuse and High School Dropout Through an Intensive School-Based Social Network Development Program,” American Journal of Health Promotion, Vol. 8, No. 3, January–February 1994, pp. 202–215. ———, “Prevention Research Program: Reconnecting At-Risk Youth,” Issues in Mental Health Nursing, Vol. 15, No. 2, March–April, 1994, pp. 107–135. ———, “Reducing Suicide Potential Among High-Risk Youth: Tests of a School-Based Prevention Program,” Suicide and Life-Threatening Behavior, Vol. 25, No. 2, 1995, pp. 276–296. Eggert, Leona L., Elaine A. Thompson, Brooke P. Randell, and Kenneth C. Pike, “Preliminary Effects of Brief School-Based Prevention Approaches for Reducing Youth Suicide,” Journal of Child and Adolescent Psychiatric Nursing, Vol. 15, No. 2, April 2002, pp. 48–64. Evidence-Based Prevention and Intervention Support Center, Evidence-Based Intervention Reports, University Park, Pa., July 2011. As of March 19, 2013: http://www.episcenter.psu.edu/EBIReports Farberow, Norman L., “The Los Angeles Survivors-After-Suicide Program: An Evaluation,” Crisis: The Journal of Crisis Intervention and Suicide Prevention, Vol. 13, No. 1, 1992, pp. 23–34. First, Michael B., Robert L. Spitzer, and Janet B. W. Williams, Structured Clinical Interview for DSM-IV Axis I Disorders, Research Version, Non-Patient Edition (SCID-I/NP), New York: New York State Psychiatric Institute, November 2002.

166 Bibliography

First Nations Centre, National Aboriginal Health Organization, Assessment and Planning Tool Kit for Suicide Prevention in First Nations Communities, Canada, 2005. As of March 19, 2013: http://www.naho.ca/documents/fnc/english/FNC_SuicidePreventionToolkit.pdf Fiske, Amy, and Patrick Arbore, “Future Directions in late life suicide prevention,” Omega: Journal of Death and Dying, Vol. 42, No. 1, 2000–2001, pp. 37–53. Fitzpatrick, Kathleen Kara, Tracy K. Witte, and Norman B. Schmidt, “Randomized Controlled Trial of a Brief Problem-Orientation Intervention for Suicidal Ideation,” Behavior Therapy, Vol. 36, No. 4, Fall 2005, pp. 323–333. Florentine, Julia Buus, and Catherine Crane, “Suicide Prevention by Limiting Access to Methods: A Review of Theory and Practice,” Social Science and Medicine, Vol. 70, No. 10, May 2010, pp. 1626–1632. Frankish, C. James, “Crisis Centers and Their Role in Treatment: Suicide Prevention Versus Health Promotion,” Death Studies, Vol. 18, No. 4, July–August 1994, pp. 327–339. Freedenthal, Stacey, “Adolescent Help-Seeking and the Yellow Ribbon Suicide Prevention Program: An Evaluation,” Suicide and Life-Threatening Behavior, Vol. 40, No. 6, December 2010, pp. 628–639. Gask, Linda, Claire Dixon, Richard Morriss, Louis Appleby, and Gillian Green, “Evaluating STORM Skills Training for Managing People at Risk of Suicide,” Journal of Advanced Nursing, Vol. 54, No. 6, June 2006, pp. 739–750. Gaynes, Bradley N., Suzanne L. West, Carol Ford, Paul Frame, Jonathan Klein, and Kathleen N. Lohr, Screening for Suicide Risk: A Systematic Evidence Review for the U.S. Preventive Services Task Force, Rockville, Md.: Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services, 2004. Goldney, Robert D., and Lanny Berman, “Postvention in Schools: Affective or Effective?” Crisis: The Journal of Crisis Intervention and Suicide Prevention, Vol. 17, No. 3, 1996, pp. 98–99. Goldston, David B., Christine M. Walrath, Richard McKeon, Richard W. Puddy, Keri M. Lubell, Lloyd B. Potter, and Michael S. Rodi, “The Garrett Lee Smith Memorial Suicide Prevention Program,” Suicide and Life-Threatening Behavior, Vol. 40, No. 3, June 2010, pp. 245–256. Gould, Madelyn S., “Suicide and the Media,” Annals of the New York Academy of Sciences, Vol. 932, No. 1, April 2001, pp. 200–224. Gould, Madelyn S., Ted Greenberg, Drew M. Velting, and David Shaffer, “Youth Suicide Risk and Preventive Interventions: A Review of the Past 10 Years,” Journal of the American Academy of Child and Adolescent Psychiatry, Vol. 42, No. 4, April 2003, pp. 386–405. Gould, Madelyn S., John Kalafat, Jimmie Lou HarrisMunfakh, and Marjorie Kleinman, “An Evaluation of Crisis Hotline Outcomes, Part 2: Suicidal Callers,” Suicide and Life-Threatening Behavior, Vol. 37, No. 3, June 2007, pp. 338–352. Gould, Madelyn S., Frank A. Marrocco, Marjorie Kleinman, John Graham Thomas, Katherine Mostkoff, Jean Cote, and Mark Davies, “Evaluating Iatrogenic Risk of Youth Suicide Screening Programs: A Randomized Controlled Trial,” Journal of the American Medical Association, Vol. 293, No. 13, April 6, 2005, pp. 1635–1643. Gould, Madelyn S., Jimmie L. H. Munfakh, Marjorie Kleinman, and Allison M. Lake, “National Suicide Prevention Lifeline: Enhancing Mental Health Care for Suicidal Individuals and Other People in Crisis,” Suicide and Life-Threatening Behavior, Vol. 42, No. 1, February 2012, pp. 22–35. Grossman, Janet, Arnim Dontes, Markus J. P. Kruesi, James Pennington, and Micahel Fendrich, “Emergency Nurses’ Responses to a Survey About Means Restriction: An Adolescent Suicide Prevention Strategy,” Journal of the American Psychiatric Nurses Association, Vol. 9, No. 3, June 2003, pp. 77–85. Gullestrup, Jorgen, Belinda Lequertier, and Graham Martin, “MATES in Construction: Impact of a Multimodal, Community-Based Program for Suicide Prevention in the Construction Industry,” International Journal of Environmental Research and Public Health, Vol. 8, No. 11, November 2011, pp. 4180–4196. Gunnell, David, and Stephen Frankel, “Prevention of Suicide: Aspirations and Evidence,” BMJ, Vol. 308, No. 6938, May 7, May 7, 1994, pp. 1227–1233.

167 RAND Suicide Prevention Program Evaluation Toolkit

Haas, Ann, Bethany Koestner, Jill Rosenberg, David Moore, Steven J. Garlow, Jan Sedway, Linda Nicholas, Herbert Hendin, John J. Mann, and Charles B. Nemeroff, “An Interactive Web-Based Method of Outreach to College Students at Risk for Suicide,” Journal of American College Health, Vol. 57, No. 1, July–August, 2008, pp. 15–22. Hakanen, Jari, and Maila Upanne, “Evaluation Strategy for Finland’s Suicide Prevention Project,” Crisis: The Journal of Crisis Intervention and Suicide Prevention, Vol. 17, No. 4, 1996, pp. 167–174. Hall, Barry, and Peter Gabor, “Peer Suicide Prevention in a Prison,” Crisis: The Journal of Crisis Intervention and Suicide Prevention, Vol. 25, No. 1, 2004, pp. 19–26. Harkavy Friedman, Jill M., and Gregory M. Asnis, “Assessment of Suicidal Behavior: A New Instrument,” Psychiatric Annals, Vol. 19, No. 7, 1989, pp. 382–387. Harkavy-Friedman, Jill M., The Harkavy-Asnis Suicide Scales, New York: Albert Einstein College of Medicine, 1985. Hamilton, Max, “A Rating Scale for Depression,” Journal of Neurosurgery and Psychiatry, Vol. 23, February 1960, pp. 56–62. Hegerl, Ulrich, David Althaus, Armin Schmidtke, and Günter Niklewski, “The Alliance Against Depression: 2-Year Evaluation of a Community-Based Intervention to Reduce Suicidality,” Psychological Medicine: A Journal of Research in Psychiatry and the Allied Sciences, Vol. 36, No. 9, September 2006, pp. 1225–1233. Hegerl, Ulrich, Roland Mergl, Inga Havers, Armin Schmidtke, Harmut Lehfeld, Günter Niklewski, and David Althaus, “Sustainable Effects on Suicidality Were Found for the Nuremberg Alliance Against Depression,” European Archives of Psychiatry and Clinical Neuroscience, Vol. 260, No. 5, 2010, pp. 401–406. Henggeler, Scott W., Melisa D. Rowland, Jeff Randall, David M. Ward, Susan G. Pickrel, Phillippe B. Cunningham, Stacey L. Miller, James Edwards, Joseph J. Zealberg, and Lisa D. Hand, “Home-Based Multisystemic Therapy as an Alternative to the Hospitalization of Youths in Psychiatric Crisis: Clinical Outcomes,” Journal of the American Academy of Child and Adolescent Psychiatry, Vol. 38, No. 11, November 1999, pp. 1331–1339. Ho, Wen-Wei, Wei-Jen Chen, Chi-Kung Ho, Ming-Been Lee, Chen-Chung Chen, and Frank Huang-Chih Chou, “Evaluation of the Suicide Prevention Program in Kaohsiung City, Taiwan, Using the CIPP Evaluation Model,” Community Mental Health Journal, Vol. 47, No. 5, October 2011, pp. 542–550. Hooven, Carole, Jerald R. Herting, and Karen A. Snedker, “Long-Term Outcomes for the Promoting CARE Suicide Prevention Program,” American Journal of Health Behavior, Vol. 34, No. 6, 2010, pp. 721–736. Huey, Stanley J., Jr., Scott W. Henggeler, Melisa D. Rowland, Colleen A. Halliday-Boykins, Phillippe B. Cunningham, Susan G. Pickrel, and James Edwards, “Multisystemic Therapy Effects on Attempted Suicide by Youths Presenting Psychiatric Emergencies,” Journal of the American Academy of Child and Adolescent Psychiatry, Vol. 43, No. 2, February 2004, pp. 183–190. Hunter, Sarah B., Matthew Chinman, and Patricia A. Ebener, “Promoting Success: A Getting To Outcomes Guide to Implementing Continuous Quality Improvement in Community Service Organizations,” unpublished research, 2011. Husky, Mathilde M., Adam Kaplan, Leslie McGuire, Laurie Flynn, Christine Chrostowski, and Mark Olfson, “Identifying Adolescents at Risk Through Voluntary School-Based Mental Health Screening,” Journal of Adolescence, Vol. 34, No. 3, June 2011, pp. 505–511. Husky, Mathilde M., Leslie McGuire, Laurie Flynn, Christine Chrostowski, and Mark Olfson, “Correlates of Help-Seeking Behavior Among At-Risk Adolescents,” Child Psychiatry and Human Development, Vol. 40, No. 1, March 2009, pp. 15–24. Husky, Mathilde M., Kathleen Miller, Leslie McGuire, Laurie Flynn, and Mark Olfson, “Mental Health Screening of Adolescents in Pediatric Practice,” Journal of Behavioral Health Services and Research, Vol. 38, No. 2, April 2011, pp. 159–169. Indelicate, Natalie Aree, Anca Mirsu-Paun, and Wayne D. Griffin, “Outcomes of a Suicide Prevention Gatekeeper Training on a University Campus,” Journal of College Student Development, Vol. 52, No. 3, May–June 2011, pp. 350–361.

168 Bibliography

Jobes, David A., Craig J. Bryan, and Tracy A. Neal-Walden, “Conducting Suicide Research in Naturalistic Clinical Settings,” Journal of Clinical Psychology, Vol. 65, No. 4, April 2009, pp. 382–395. Joffe, Paul, “An Empirically Supported Program to Prevent Suicide in a College Student Population,” Suicide and Life-Threatening Behavior, Vol. 38, No. 1, February 2008, pp. 87–103. Johnson, Renee M., Elaine M. Frank, Mark Ciocca, and Catherine W. Barber, “Training Mental Healthcare Providers to Reduce At-Risk Patients’ Access to Lethal Means of Suicide: Evaluation of the CALM Project,” Archives of Suicide Research, Vol. 15, No. 3, July 2011, pp. 259–264. Kalafat, John, “School Approaches to Youth Suicide Prevention,” American Behavioral Scientist, Vol. 46, No. 9, May 2003, pp. 1211–1223. Kalafat, John, Madelyn S. Gould, Jimmie Lou HarrisMunfakh, and Marjorie Kleinman, “An Evaluation of Crisis Hotline Outcomes, Part 1: Nonsuicidal Crisis Callers,” Suicide and Life-Threatening Behavior, Vol. 37, No. 3, June 2007, pp. 322–337. Kalafat, J., M. Madden, D. Haley, S. O’Halloran, and C. DiCara, “Evaluation of Lifelines Classes: A Component of the School-Community Based Maine Youth Suicide Prevention Project,” unpublished research, 2007. Kalafat, John, and William D. Neigher, “Experimental and Pragmatic Approaches to the Incidence of Adolescent Suicide,” Evaluation and Program Planning, Vol. 14, No. 4, 1991, pp. 377–383. Kaleveld, Lisette, and Brian English, “Evaluating a Suicide Prevention Program: A Question of Impact,” Health Promotion Journal of Australia, Vol. 16, No. 2, August 2005, pp. 129–133. Kann, Laura, Steven A. Kinchen, Barbara I. Williams, James G. Ross, Richard Lowry, Jo Anne Grunbaum, and Lloyd J. Kolbe, “Youth Risk Behavior Surveillance—United States, 1999,” Morbidity and Mortality Weekly Report, Vol. 49, No. SS-5, June 9, 2000. Kato, Takahiro A., Yuriko Suzuki, Ryoko Sato, Daisuke Fujisawa, Kumi Uehara, Naoki Hashimoto, Yasunori Sawayama, Jun Hayashi, Shigenobu Kanba, and Kotaro Otsuka, “Development of 2-Hour Suicide Intervention Program Among Medical Residents: First Pilot Trial,” Psychiatry and Clinical Neuroscience, Vol. 64, No. 5, October 2010, pp. 531–540. Kazdin, Alan E., Antoinette Rodgers, and Debra Colbus, “The Hopelessness Scale for Children: Psychometric Characteristics and Concurrent Validity,” Journal of Consulting and Clinical Psychology, Vol. 54, No. 2, April 1986, pp. 241–245. Kerkhof, A. J. F., “The Finnish National Suicide Prevention Program Evaluated,” Crisis: The Journal of Crisis Intervention and Suicide Prevention, Vol. 20, No. 2, 1999, pp. 50, 63. Kerkhof, A. J. F., and David C. Clark, “How to Evaluate National Suicide Programs?” Crisis: The Journal of Crisis Intervention and Suicide Prevention, Vol. 19, No. 1, 1998, pp. 2–3. King, Keith A., and Judie Smith, “Project SOAR: A Training Program to Increase School Counselors’ Knowledge and Confidence Regarding Suicide Prevention and Intervention,” Journal of School Health, Vol. 70, No. 10, December 2000, pp. 402–407. King, Keith A., Catherine M. Strunk, and Michael T. Sorter, “Preliminary Effectiveness of Surviving the Teens® Suicide Prevention and Depression Awareness Program on Adolescents’ Suicidality and Self-Efficacy in Performing Help-Seeking Behaviors,” Journal of School Health, Vol. 81, No. 9, September 2011, pp. 581–590. King, Robert, Barry Nurcombe, Leonard Bickman, Leanne Hides, and Wendy Reid, “Telephone Counselling for Adolescent Suicide Prevention: Changes in Suicidality and Mental State from Beginning to End of a Counselling Session,” Suicide and Life-Threatening Behavior, Vol. 33, No. 4, Winter 2003, pp. 400–411. Kinn, Julie T., David D. Luxton, Mark A. Reger, Gregory A. Gahm, Nancy A. Skopp, and Nigel E. Bush, Department of Defense Suicide Event Report: Calendar Year 2010 Annual Report, Washington, D.C.: National Center for Telehealth and Technology, U.S. Department of Defense, 2011. Klimes-Dougan, Bonnie, Chih Yuan, Steven Lee, and Aalaa K. Houri, “Suicide Prevention with Adolescents: Considering Potential Benefits and Untoward Effects of Public Service Announcements,” Crisis: The Journal of Crisis Intervention and Suicide Prevention, Vol. 30, No. 3, 2009, pp. 128–135.

169 RAND Suicide Prevention Program Evaluation Toolkit

Knox, Kerry L., David A. Litts, G. Wayne Talcott, Jill Catalano Feig, and Eric D. Caine, “Risk of Suicide and Related Adverse Outcomes After Exposure to a Suicide Prevention Programme in the US Air Force: Cohort Study,” BMJ, Vol. 327, No. 7428, 2003, pp. 1376–1378. Knox, Kerry L., Steven Pflanz, Gerald W. Talcott, Rick L. Campise, Jill E. Lavigne, Alina Bajorska, Xin Tu, and Eric D. Caine, “The US Air Force Suicide Prevention Program: Implications for Public Health Policy,” American Journal of Public Health, Vol. 100, No. 12, December 2010, pp. 2457–2463. Kostenuik, Marcia, and Mohana Ratnapalan, “Approach to Adolescent Suicide Prevention,” Canadian Family Physician, Vol. 56, No. 8, August 2010, pp. 755–760. Kruesi, Markus J. P., Janet Grossman, James M. Pennington, Paul J. Woodward, David Duda, and Jay G. Hirsch, “Suicide and Violence Prevention: Parent Education in the Emergency Department,” Journal of the American Academy of Child and Adolescent Psychiatry, Vol. 38, No. 3, March 1999, pp. 250–255. Krysinska, Karolina, and Graham Martin, “The Struggle to Prevent and Evaluate: Application of Population Attributable Risk and Preventive Fraction to Suicide Prevention Research,” Suicide and Life-Threatening Behavior, Vol. 39, No. 5, October 2009, pp. 548–557. Kurz, Alexander, and Hans-Jürgen Moller, “Attempted Suicide: Efficacy of Treatment Programs,” Psychiatry and Clinical Neurosciences, Vol. 49, Suppl. 1, May 1995, pp. S99–S103. LaFromboise, Teresa D., and Beth Howard-Pitney, “The Zuni Life Skills Development Curriculum: A Collaborative Approach to Curriculum Development,” American Indian and Alaska Native Mental Health Research, Vol. 4, 1994, pp. 98–121. ———, “The Zuni Life Skills Development Curriculum: Description and Evaluation of a Suicide Prevention Program,” Journal of Counseling Psychology, Vol. 42, No. 4, October 1995, pp. 479–486. Lapierre, Sylvie, Annette Erlangsen, Margda Waern, Diego De Leo, Hirofumi Oyama, Paolo Scocco, Joseph Gallo, Katalin Szanto, Yeates Conwell, Brian Draper, and Paul Quinnett, “A Systematic Review of Elderly Suicide Prevention Programs,” Crisis: The Journal of Crisis Intervention and Suicide Prevention, Vol. 32, No. 2, 2011, pp. 88–98. Leenaars, Antoon A., and David Lester, “Impact of Suicide Prevention Centers on Suicide in Canada,” Crisis: The Journal of Crisis Intervention and Suicide Prevention, Vol. 16, No. 1, 1995, p. 39. Lenth, Russell V., “Some Practical Guidelines for Effective Sample Size Determination,” The American Statistician, Vol. 55, No. 3, August 2001, pp. 187–193. Lesesne, C. A., K. M. Lewis, C. Moore, D. Fisher, D. Green, and A. Wandersman, Promoting Science-Based Approaches to Teen Pregnancy Prevention Using Getting To Outcomes: Draft 2011, unpublished manual, 2011. Lester, David, “Geographical Location of Callers to a Suicide Prevention Center: Note on the Evaluation of Suicide Prevention Programs,” Psychological Reports, Vol. 28, No. 2, April 1971, pp. 421–422. ———, “State Initiatives in Addressing Youth Suicide: Evidence for Their Effectiveness,” Social Psychiatry and Psychiatric Epidemiology, Vol. 27, No. 2, 1992, pp. 75–77. ———, “The Use of the Internet for Counseling the Suicidal Individual: Possibilities and Drawbacks,” Omega (Westport), Vol. 58, No. 3, 2008, pp. 233–250. Leviton, Laura C., Laura Kettel Khan, Debra Rog, Nicola Dawkins, and David Cotton, “Evaluability Assessment to Improve Public Health Policies, Programs, and Practices,” Annual Review of Public Health, Vol. 31, 2010, pp. 213–233. Linehan, Marsha M., unpublished work on multisystemic therapy, 1981. Linehan, Marsha M., Hubert E. Armstrong, Alejandra Suarez, Douglas Allmon, and Heidi L. Heard, “Cognitive-Behavioral Treatment of Chronically Parasuicidal Borderline Patients,” Archives of General Psychiatry, Vol. 48, No. 12, 1991, pp. 1060–1064.

170 Bibliography

Linehan, Marsha M., Katherine Ann Comtois, Angela M. Murray, Milton Z. Brown, Robert J. Gallop, Heidi L. Heard, Kathryn E. Korslund, Darren A. Tutek, Sarah K. Reynolds, and Noam Lindenboim, “Two-Year Randomized Controlled Trial and Follow-Up of Dialectical Behavior Therapy vs Therapy by Experts for Suicidal Behaviors and Borderline Personality Disorder,” Archives of General Psychiatry, Vol. 63, No. 7, 2006, pp. 757–766. Linehan, Marsha M., Judith L. Goodstein, Stevan L. Nielsen, and John A. Chiles, “Reasons for Staying Alive When You Are Thinking of Killing Yourself: The Reasons for Living Inventory,” Journal of Consulting and Clinical Psychology, Vol. 51, No. 2, April 1983, pp. 276–286. Linehan, Marsha M., and Heidi L. Heard, Treatment History Interview (THI), Seattle, Wash.: University of Washington, 1987. Linehan, Marsha M., Heidi L. Heard, and Hubert E. Armstrong, “Naturalistic Follow-Up of a Behavioral Treatment for Chronically Parasuicidal Borderline Patients,” Archives of General Psychiatry, Vol. 50, No. 12, 1993, pp. 971–974. Linehan, Marsha M., Darren A. Tutek, Heidi L. Heard, and Hubert E. Armstrong, “Interpersonal Outcome of Cognitive Behavioral Treatment for Chronically Suicidal Borderline Patients,” American Journal of Psychiatry, Vol. 151, No. 12, December 1994, pp. 1771–1775. Litman, R. E., “Anti-Suicide Program Conducts Controlled Study,” Evaluation, Vol. 3, No. 1–2, 1976, pp. 36–37. Luxton, David D., Julie T. Kinn, Jennifer D. June, Laura W. Pierre, Mark A. Reger, and Gregory A. Gahm, “Caring Letters Project: A Military Suicide Prevention Program,” Crisis: The Journal of Crisis Intervention and Suicide Prevention, Vol. 33, No. 1, 2011, pp. 5–12. Mackenzie, Mhairi, Avril Blamey, Emma Halliday, Margaret Maxwell, Allyson McCollam, David McDaid, Joanne MacLean, Amy Woodhouse, and Stephen Platt, “Measuring the Tail of the Dog That Doesn’t Bark in the Night: The Case of the National Evaluation of Choose Life (the National Strategy and Action Plan to Prevent Suicide in Scotland),” BMC Public Health, Vol. 7, 2007. Maine, Shriyani, Rosalyn Shute, and Graham Martin, “Educating Parents About Youth Suicide: Knowledge, Response to Suicidal Statements, Attitudes, and Intention to Help,” Suicide and Life-Threatening Behavior, Vol. 31, No. 3, Fall 2001, pp. 320–332. Mann, J. John, Alan Apter, Jose Bertolote, Annette Beautrais, Dianne Currier, Ann Haas, Ulrich Hegerl, Jouko Lonnqvist, Kevin Malone, Andrej Marusic, Lars Mehlum, George Patton, Michael Phillips, Wolfgang Rutz, Zoltan Rihmer, Armin Schmidtke, David Shaffer, Morton Silverman, Yoshitomo Takahashi, Airi Varnik, Danuta Wasserman, Paul Yip, and Herbert Hendin, “Suicide Prevention Strategies: A Systematic Review,” Journal of the American Medical Association, Vol. 294, No. 16, October 26, 2005, pp. 2064–2074. Martin, Graham, “On Evaluation,” Australian e-Journal for the Advancement of Mental Health, Vol. 4, No. 1, 2005, pp. 1–3. Martin, Graham, Karolina Krysinska, and Sarah Swannell, “The Struggle to Evaluate Prevention Programs,” Australian e-Journal for the Advancement of Mental Health, Vol. 7, No. 2, 2008, pp. 1–6. Matsubayashi, Tetsuya, and Michiko Ueda, “The Effect of National Suicide Prevention Programs on Suicide Rates in 21 OECD Nations,” Social Science and Medicine, Vol. 73, No. 9, November 2011, pp. 1395–1400. Matthieu, Monica M., Yufei Chen, Mary Schohn, Larry J. Lantinga, and Kerry L. Knox, “Educational Preferences and Outcomes from Suicide Prevention Training in the Veterans Health Administration: One-Year Follow-Up with Healthcare Employees in Upstate New York,” Military Medicine, Vol. 174, No. 11, November 2009, pp. 1123–1131. Matthieu, Monica M., Wendi Cross, Alfonso R. Batres, Charles M. Flora, and Kerry L. Knox, “Evaluation of Gatekeeper Training for Suicide Prevention in Veterans,” Archives of Suicide Research, Vol. 12, No. 2, 2008, pp. 148–154. Matthieu, Monica M., Alan Ross, and Kerry L. Knox, “Program Evaluation of the Samaritans of New York’s Public Education Suicide Awareness and Prevention Training Program,” Brief Treatment and Crisis Intervention, Vol. 6, No. 4, 2006, pp. 295–307.

171 RAND Suicide Prevention Program Evaluation Toolkit

May, Philip A., Patricia Serna, Lance Hurt, and Lemyra M. Debruyn, “Outcome Evaluation of a Public Health Approach to Suicide Prevention in an American Indian Tribal Nation,” American Journal of Public Health, Vol. 95, No. 7, July 2005, pp. 1238–1244. Mazza, James J., “School-Based Suicide Prevention Programs: Are They Effective?” School Psychology Review, Vol. 26, No. 3, 1997, pp. 382–396. McAuliffe, Nora, and Lynda Perry, “Making It Safer: A Health Centre’s Strategy for Suicide Prevention,” Psychiatric Quarterly, Vol. 78, No. 4, December 2007, pp. 295–307. McDaniel, William W., Marion Rock, and Jon R. Grigg, “Suicide Prevention at a United States Navy Training Command,” Military Medicine, Vol. 155, No. 4, April 1990, pp. 173–175. McFarland, Bentson H., and Daniel J. Beavers, “Preventive Strategies and Program Evaluation Methods for Chronically Mentally Ill Suicide Attempters,” Comprehensive Psychiatry, Vol. 25, No. 4, July–August 1984, pp. 426–437. McLaughlin, John A., and Gretchen B. Jordan, “Logic Models: A Tool for Telling Your Program’s Performance Story,” Evaluation and Program Planning, Vol. 22, No. 1, February 1999, pp. 65–72. McManus, Bonnie L., Markus J. P. Kruesi, Armin E. Dontes, Christian R. Defazio, John T. Piotrowski, and Paul J. Woodward, “Child and Adolescent Suicide Attempts: An Opportunity for Emergency Departments to Provide Injury Prevention Education,” American Journal of Emergency Medicine, Vol. 15, No. 4, July 1997, pp. 357–360. McNamee, Jane E., and David R. Offord, “Prevention of Suicide,” Canadian Medical Association Journal, Vol. 142, No. 11, June 1, 1990, pp. 1223–1230. Medoff, Marshall H., “An Evaluation of the Effectiveness of Suicide Prevention Centers,” Journal of Behavioral Economics, Vol. 15, Fall 1986, pp. 43–55. Meehan, Sue-Ann, and Yvonne Broom, “Analysis of a National Toll Free Suicide Crisis Line in South Africa,” Suicide and Life-Threatening Behavior, Vol. 37, No. 1, February 2007, pp. 66–78. Mellanby, R. J., N. P. Hudson, R. Allister, C. E. Bell, R. W. Else, D. A. Gunn-Moore, C. Byrne, S. Straiton, and S. M. Rhind, “Evaluation of Suicide Awareness Programmes Delivered to Veterinary Undergraduates and Academic Staff,” Veterinary Record, Vol. 167, No. 19, November 6, 2010, pp. 730–734. Melle, Ingrid, Jan Olav Johannesen, Svein Friis, Ulrik Haahr, Inge Joa, Tor K. Larsen, Stein Opjordsmoen, Bjorn Rund, Erik Simonsen, Per Vaglum, and Thomas McGlashan, “Early Detection of the First Episode of Schizophrenia and Suicidal Behavior,” American Journal of Psychiatry, Vol. 163, No. 5, 2006, pp. 800–804. Middlebrook, Denise L., Pamela L. LeMaster, Janette Beals, Douglas K. Novins, and Spero M. Manson, “Suicide Prevention in American Indian and Alaska Native Communities: A Critical Review of Programs,” Suicide and Life-Threatening Behavior, Vol. 31 Suppl., Spring, 2001, pp. 132–149. Miranda, Regina, Michelle Scott, Roger Hicks, Holly C. Wilcox, Jimmie L. Harris-Munfakh, and David Shaffer, “Suicide Attempt Characteristics, Diagnoses, and Future Attempts: Comparing Multiple Attempters to Single Attempters and Ideators,” Journal of the American Academy of Child and Adolescent Psychiatry, Vol. 47, No. 1, January 2008, pp. 32–40. Mishara, Brian L., François Chagnon, and Marc Daigle, “Comparing Models of Helper Behavior to Actual Practice in Telephone Crisis Intervention: A Silent Monitoring Study of Calls to the U.S. 1-800-SUICIDE Network,” Suicide and Life-Threatening Behavior, Vol. 37, No. 3, June 2007a, pp. 291–307. ———, “Which Helper Behaviors and Intervention Styles Are Related to Better Short-Term Outcomes in Telephone Crisis Intervention? Results from a Silent Monitoring Study of Calls to the U.S. 1-800-SUICIDE Network,” Suicide and Life-Threatening Behavior, Vol. 37, No. 3, June 2007b, pp. 308–321. Mishara, Brian L., and Marc S. Daigle, “Effects of Different Telephone Intervention Styles with Suicidal Callers at Two Suicide Prevention Centers: An Empirical Investigation,” American Journal of Community Psychology, Vol. 25, No. 6, December 1997, pp. 861–885. Mishara, Brian L., Janie Houle, and Brigitte Lavoie, “Comparison of the Effects of Four Suicide Prevention Programs for Family and Friends of High-Risk Suicidal Men Who Do Not Seek Help Themselves,” Suicide and Life-Threatening Behavior, Vol. 35, No. 3, June 2005, pp. 329–342.

172 Bibliography

Mishara, Brian, and Mette Ystgaard, “Exploring the Potential for Primary Prevention: Evaluation of the Befrienders International Reaching Young Europe Pilot Program in Denmark,” Crisis: The Journal of Crisis Intervention and Suicide Prevention, Vol. 21, No. 1, 2000, pp. 4–5, 7. Motto, Jerome A., and Alan G. Bostrom, “A Randomized Controlled Trial of Postcrisis Suicide Prevention,” Psychiatric Services, Vol. 52, No. 6, June 2001, pp. 828–833. Neimeyer, Robert A., and Angela M. Pfeiffer, “Evaluation of Suicide Intervention Effectiveness,” Death Studies, Vol. 18, No. 2, March–April 1994, pp. 131–166. Nelson, Franklyn L., “Evaluation of a Youth Suicide Prevention School Program,” Adolescence, Vol. 22, No. 88, Winter 1987, pp. 813–825. Nordentoft, M., J. Branner, K. Drejer, B. Mejsholm, H. Hansen, and B. Petersson, “Effect of a Suicide Prevention Centre for Young People with Suicidal Behaviour in Copenhagen,” European Psychiatry, Vol. 20, No. 2, March 2005, pp. 121–128. O’Carroll, Patrick W., Lloyd B. Potter, and James A. Mercy, “Programs for the Prevention of Suicide Among Adolescents and Young Adults,” Morbidity and Mortality Weekly Report, Vol. 43, No. RR-6, April 22, 1994, pp. 1–7. Omar, Hatim A., “A Model Program for Youth Suicide Prevention,” International Journal of Adolescent Medicine and Health, Vol. 17, No. 3, July–September, 2005, pp. 275–278. Orbach, Israel, and Hanna Bar-Joseph, “The Impact of a Suicide Prevention Program for Adolescents on Suicidal Tendencies, Hopelessness, Ego Identity, and Coping,” Suicide and Life-Threatening Behavior, Vol. 23, No. 2, Summer 1993, pp. 120–129. Overholser, James C., Alyce H. Hemstreet, Anthony Spirito, and Stuart Vyse, “Suicide Awareness Programs in the Schools: Effects of Gender and Personal Experience,” Journal of the American Academy of Child and Adolescent Psychiatry, Vol. 28, No. 6, 1989, pp. 925–930. Oyama, Hirofumi, Motoi Fujita, Masahiro Goto, Hiroshi Shibuya, and Tomoe Sakashita, “Outcomes of Community-Based Screening for Depression and Suicide Prevention Among Japanese Elders,” Gerontologist, Vol. 46, No. 6, December 2006, pp. 821–826. Oyama, Hirofumi, Masahiro Goto, Motoi Fujita, Hiroshi Shibuya, and Tomoe Sakashita, “Preventing Elderly Suicide Through Primary Care by Community-Based Screening for Depression in Rural Japan,” Crisis: The Journal of Crisis Intervention and Suicide Prevention, Vol. 27, No. 2, 2006, pp. 58–65. Oyama, Hirofumi, Junichi Koida, Tomoe Sakashita, and Keiko Kudo, “Community-Based Prevention for Suicide in Elderly by Depression Screening and Follow-Up,” Community Mental Health Journal, Vol. 40, No. 3, June 2004, pp. 249–263. Oyama, Hirofumi, Yutaka Ono, Naoki Watanabe, Eriko Tanaka, Seijiro Kudoh, Tomoe Sakashita, Shinji Sakamoto, Keiko Neichi, Kyoko Satoh, Kenji Nakamura, and Kimio Yoshimura, “Local Community Intervention Through Depression Screening and Group Activity for Elderly Suicide Prevention,” Psychiatry and Clinical Neuroscience, Vol. 60, No. 1, February 2006, pp. 110–114. Oyama, Hirofumi, Naoki Watanabe, Yutaka Ono, Tomoe Sakashita, Yuka Takenoshita, Manabu Taguchi, Tohru Takizawa, Reiko Miura, and Keiko Kumagai, “Community-Based Suicide Prevention Through Group Activity for the Elderly Successfully Reduced the High Suicide Rate for Females,” Psychiatry and Clinical Neuroscience, Vol. 59, No. 3, June 2005, pp. 337–344. Page, Andrew, Richard Taylor, David Gunnell, Greg Carter, Stephen Morrell, and Graham Martin, “Effectiveness of Australian Youth Suicide Prevention Initiatives,” British Journal of Psychiatry, Vol. 199, November 2011, pp. 423–429. Pearson, Jane L., Barbara Stanley, Cheryl King, and Celia Fisher, Issues to Consider in Intervention Research with Persons at High Risk for Suicidality, Bethesda, Md.: National Institute of Mental Health, January 2001. Pena, Juan B., and Eric D. Caine, “Screening as an Approach for Adolescent Suicide Prevention,” Suicide and Life-Threatening Behavior, Vol. 36, No. 6, December 2006, pp. 614–637.

173 RAND Suicide Prevention Program Evaluation Toolkit

Pierce, D. W., “The Predictive Validation of a Suicide Intent Scale: A Five Year Follow-Up,” British Journal of Psychiatry, Vol. 139, No. 5, 1981, pp. 391–396. Pirkis, Jane, “Suicide and the Media,” Psychiatry, Vol. 8, No. 7, 2009, pp. 269–271. Ploeg, J., D. Ciliska, M. Dobbins, S. Hayward, H. Thomas, and J. Underwood, “A Systematic Overview of Adolescent Suicide Prevention Programs,” Canadian Journal of Public Health, Vol. 87, No. 5, September–October, 1996, pp. 319–324. Power, P. J., R. J. Bell, R. Mills, T. Herrman-Doig, M. Davern, L. Henry, H. P. Yuen, A. Khademy-Deljo, and P. D. McGorry, “Suicide Prevention in First Episode Psychosis: The Development of a Randomised Controlled Trial of Cognitive Therapy for Acutely Suicidal Patients with Early Psychosis,” Australian and New Zealand Journal of Psychiatry, Vol. 37, No. 4, August 2003, pp. 414–420. Radloff, Lenore Sawyer, “The Ces-D Scale,” Applied Psychological Measurement, Vol. 1, No. 3, July 1977, pp. 385–401. Ramberg, Inga-Lill, and Danuta Wasserman, “Benefits of Implementing an Academic Training of Trainers Program to Promote Knowledge and Clarity in Work with Psychiatric Suicidal Patients,” Archives of Suicide Research, Vol. 8, No. 4, 2004a, pp. 331–343. ———, “Suicide-Preventive Activities in Psychiatric Care: Evaluation of an Educational Programme in Suicide Prevention,” Nordic Journal of Psychiatry, Vol. 58, No. 5, October 2004b, pp. 389–394. Ramchand, Rajeev, Joie Acosta, Rachel M. Burns, Lisa H. Jaycox, and Christopher G. Pernin, The War Within: Preventing Suicide in the U.S. Military, Santa Monica: Calif.: RAND Corporation, MG-953-OSD, 2011. As of March 19, 2013: http://www.rand.org/pubs/monographs/MG953.html RAND Corporation, “Getting To Outcomes®: Improving Community-Based Prevention,” web page, undated. As of March 19, 2013: http://www.rand.org/health/projects/getting-to-outcomes.html Randell, Brooke P., Leona L. Eggert, and Kenneth C. Pike, “Immediate Post Intervention Effects of Two Brief Youth Suicide Prevention Interventions,” Suicide and Life-Threatening Behavior, Vol. 31, No. 1, Spring 2001, pp. 41–61. ———, “‘Immediate Post Intervention Effects of Two Brief Youth Suicide Prevention Interventions’: Reply,” Suicide and Life-Threatening Behavior, Vol. 32, No. 4, Winter 2002, pp. 454–456. Reisch, Thomas, Petra Schlatter, and Wolfgang Tschacher, “Efficacy of Crisis Intervention,” Crisis: The Journal of Crisis Intervention and Suicide Prevention, Vol. 20, No. 2, 1999, pp. 78–85. Rodgers, Philip L., Howard S. Sudak, Morton M. Silverman, and David A. Litts, “Evidence-Based Practices Project for Suicide Prevention,” Suicide and Life-Threatening Behavior, Vol. 37, No. 2, April 2007, pp. 154–164. Roskar, Saska, Anja Podlesek, Maja Zorko, Rok Tavcar, Mojca Z. Dernovsek, Urban Groleger, Milan Mirjanic, Nusa Konec, Evgen Janet, and Andrej Marusic, “Effects of Training Program on Recognition and Management of Depression and Suicide Risk Evaluation for Slovenian Primary-Care Physicians: Follow-Up Study,” Croatian Medical Journal, Vol. 51, No. 3, June 2010, pp. 237–242. Rossi, Peter Henry, Mark W. Lipsey, and Howard E. Freeman, Evaluation: A Systematic Approach, Thousand Oaks, Calif.: Sage Publications, 2004. Rotheram-Borus, Mary Jane, and Jon Bradley, “Triage Model for Suicidal Runaways,” American Journal of Orthopsychiatry, Vol. 61, No. 1, January 1991, pp. 122–127. Rotheram-Borus, Mary Jane, John Piacentini, Coleen Cantwell, Thomas R. Belin, and Juwon Song, “The 18-Month Impact of an Emergency Room Intervention for Adolescent Female Suicide Attempters,” Journal of Consulting and Clinical Psychology, Vol. 68, No. 6, 2000, pp. 1081–1093. Rotheram-Borus, Mary Jane, John Piacentini, Ronan Van Rossem, Flemming Graae, Coleen Cantwell, David Castro-Blanco, Sutherland Miller, and Julie Feldman, “Enhancing Treatment Adherence with a Specialized Emergency Room Program for Adolescent Suicide Attempters,” Journal of the American Academy of Child and Adolescent Psychiatry, Vol. 35, No. 5, May 1996, pp. 654–663.

174 Bibliography

Rowland, Melisa D., Colleen A. Halliday-Boykins, Scott W. Henggeler, Phillippe B. Cunningham, Terry G. Lee, Markus J. P. Kruesi, and Steven B. Shapiro, “A Randomized Trial of Multisystemic Therapy with Hawaii’s Felix Class Youths,” Journal of Emotional and Behavioral Disorders, Vol. 13, No. 1, Spring 2005, pp. 13–23. Rozanov, Vsevolod A., Alexander N. Mokhovikov, and Richard Stiliha, “Successful Model of Suicide Prevention in the Ukraine Military Environment,” Crisis: The Journal of Crisis Intervention and Suicide Prevention, Vol. 23, No. 4, 2002, pp. 171–177. Rudd, M. David, M. Hasan Rajab, and David T. Orman, “Effectiveness of an Outpatient Intervention Targeting Suicidal Young Adults: Preliminary Results,” Journal of Consulting and Clinical Psychology, Vol. 64, No. 1, February 1996, pp. 179–190. Rutz, Wolfgang, “Preventing Suicide and Premature Death by Education and Treatment,” Journal of Affective Disorders, Vol. 62, Nos. 1–2, January 2001, pp. 123–129. Rutz, W., L. Knorring, and J. Walinder, “Frequency of Suicide on Gotland After Systematic Postgraduate Education of General Practitioners,” Acta Psychiatrica Scandinavica, Vol. 80, No. 2, August 1989, pp. 151–154. Sarason, Irwin G., Barbara R. Sarason, Edward N. Shearin, and Gregory R. Pierce, “A Brief Measure of Social Support: Practical and Theoretical Implications,” Journal of Social and Personal Relationships, Vol. 4, No. 4, November 1987, pp. 497–510. Saunders, John B., Olaf G. Aasland, Thomas F. Babor, Juan R. de la Fuente, and Marcus Grant, “Development of the Alcohol Use Disorders Identification Test (AUDIT): WHO Collaborative Project on Early Detection of Persons with Harmful Alcohol Consumption–II,” Addiction, Vol. 88, No. 6, 1993, pp. 791–804. Schoenwald, Sonja K., David M. Ward, Scott W. Henggeler, and Melisa D. Rowland, “Multisystemic Therapy Versus Hospitalization for Crisis Stabilization of Youth: Placement Outcomes 4 Months Postreferral,” Mental Health Services Research, Vol. 2, No. 1, 2000, pp. 3–12. Scott, Michelle, Holly Wilcox, Yanling Huo, J. Blake Turner, Prudence Fisher, and David Shaffer, “School- Based Screening for Suicide Risk: Balancing Costs and Benefits,” American Journal of Public Health, Vol. 100, No. 9, September 2010, pp. 1648–1652. Scott, Michelle A., Holly C. Wilcox, Irvin Sam Schonfeld, Mark Davies, Roger C. Hicks, J. Blake Turner, and David Shaffer, “School-Based Screening to Identify At-Risk Students Not Already Known to School Professionals: The Columbia Suicide Screen,” American Journal of Public Health, Vol. 99, No. 2, February 2009, pp. 334–339. Shaffer, David, Ann Garland, Veronica Vieland, Maureen Underwood, and Carey Busner, “The Impact of Curriculum-Based Suicide Prevention Programs for Teenagers,” Journal of the American Academy of Child and Adolescent Psychiatry, Vol. 30, No. 4, July 1991, pp. 588–596. Shaffer, David, Michelle Scott, Holly Wilcox, Carey Maslow, Roger Hicks, Christopher P. Lucas, Robin Garfinkel, and Steven Greenwald, “The Columbia SuicideScreen: Validity and Reliability of a Screen for Youth Suicide and Depression,” Journal of the American Academy of Child and Adolescent Psychiatry, Vol. 43, No. 1, January 2004, pp. 71–79. Shaffer, David, Veronica Vieland, Ann Garland, Mary Rojas, Maureen Underwood, and Carey Busner, “Adolescent Suicide Attempters: Response to Suicide-Prevention Programs,” Journal of the American Medical Association, Vol. 264, No. 24, December 26, 1990, pp. 3151–3155. Shim, Ruth S., and Michael T. Compton, “Pilot Testing and Preliminary Evaluation of a Suicide Prevention Education Program for Emergency Department Personnel,” Community Mental Health Journal, Vol. 46, No. 6, December 2010, pp. 585–590. Simpson, Grahame, Bernie Franke, and Lauren Gillett, “Suicide Prevention Training Outside the Mental Health Service System: Evaluation of a State-Wide Program in Australia for Rehabilitation and Disability Staff in the Field of Traumatic Brain Injury,” Crisis: The Journal of Crisis Intervention and Suicide Prevention, Vol. 28, No. 1, 2007, pp. 35–43.

175 RAND Suicide Prevention Program Evaluation Toolkit

Smith, Steven W., Ann P. Daunic, and Gregory G. Taylor, “Treatment Fidelity in Applied Educational Research: Expanding the Adoption and Application of Measures to Ensure Evidence-Based Practice,” Education and Treatment of Children, Vol. 30, No. 4, November 2007, pp. 121–134. Spirito, Anthony, James Overholser, Sally Ashworth, and John Morgan, “Evaluation of a Suicide Awareness Curriculum for High School Students,” Journal of the American Academy of Child and Adolescent Psychiatry, Vol. 27, No. 6, November 1988, pp. 705–711. Stelmachers, Zigfrids T., “Current Status of Program Evaluation Efforts,” Suicide and Life-Threatening Behavior, Vol. 6, No. 2, Summer 1976, pp. 67–78. Stone, Deborah M., Catherine W. Barber, and Lloyd Potter, “Public Health Training Online: The National Center for Suicide Prevention Training,” American Journal of Preventive Medicine, Vol. 29, No. 5, Suppl. 2, December 2005, pp. 247–251. Streiner, David L., and Kenneth S. Adam, “Evaluation of the Effectiveness of Suicide Prevention Programs: A Methodological Perspective,” Suicide and Life-Threatening Behaviors, Vol. 17, No. 2, Summer 1987, pp. 93–106. Stuart, Carol, Judith Kelly Waalen, and Echo Haelstromm, “Many Helping Hearts: An Evaluation of Peer Gatekeeper Training in Suicide Risk Assessment,” Death Studies, Vol. 27, No. 4, May 2003, pp. 321–333. Suicide Prevention Action Network, Suicide Prevention: Prevention Effectiveness and Evaluation, Washington, D.C., 2001. As of March 19, 2013: http://www.sprc.org/sites/sprc.org/files/library/prevtoolkit.pdf “Suicide Prevention Evaluation in a Western Athabaskan American Indian Tribe—New Mexico, 1988–1997,” Morbidity and Mortality Weekly Report, Vol. 47, No. 13, April 10, 1998, pp. 257–261. Szanto, Katalin, Sandor Kalmar, Herbert Hendin, Zoltan Rihmer, and J. John Mann, “A Suicide Prevention Program in a Region with a Very High Suicide Rate,” Archives of General Psychiatry, Vol. 64, No. 8, August 2007, pp. 914–920. Takada, Misato, and Satoro Shima, “Characteristics and Effects of Suicide Prevention Programs: Comparison Between Workplace and Other Settings,” Industrial Health, Vol. 48, No. 4, 2010, pp. 416–426. Thompson, Elaine A., Leona L. Eggert, and Jerald R. Herting, “Mediating Effects of an Indicated Prevention Program for Reducing Youth Depression and Suicide Risk Behaviors,” Suicide and Life-Threatening Behavior, Vol. 30, No. 3, Fall 2000, pp. 252–271. Thompson, Elaine A., Leona L. Eggert, Brooke P. Randell, and Kenneth C. Pike, “Evaluation of Indicated Suicide Risk Prevention Approaches for Potential High School Dropouts,” American Journal of Public Health, Vol. 91, No. 5, May, 2001, pp. 742–752. Tierney, R. J., “Suicide Intervention Training Evaluation: A Preliminary Report,” Crisis: The Journal of Crisis Intervention and Suicide Prevention, Vol. 15, No. 2, 1994, pp. 69–76. Tompkins, Tanya L., and Jody Witt, “The Short-Term Effectiveness of a Suicide Prevention Gatekeeper Training Program in a College Setting with Residence Life Advisers,” Journal of Primary Prevention, Vol. 30, No. 2, 2009, pp. 131–149. Tompkins, Tanya L., Jody Witt, and Nadia Abraibesh, “Does a Gatekeeper Suicide Prevention Program Work in a School Setting? Evaluating Training Outcome and Moderators of Effectiveness,” Suicide and Life-Threatening Behavior, Vol. 40, No. 5, November 2010, pp. 506–515. Tsai, Wen-Pei, Long-Yau Lin, Wen-Li Chang, Hui-Chin Chang, and Ming-Chih Chou, “The Effects of the Suicide Awareness Program in Enhancing Community Volunteers’ Awareness of Suicide Warning Signs,” Archives of Psychiatric Nursing, Vol. 24, No. 1, February 2010, pp. 63–68. Tsai, Wen-Pei, Long-Yau Lin, Hui-Chin Chang, Li-Shiuan Yu, and Ming-Chih Chou, “The Effects of the Gatekeeper Suicide-Awareness Program for Nursing Personnel,” Perspectives in Psychiatric Care, Vol. 47, No. 3, July 2011, pp. 117–125. Turner, Ralph M., “Naturalistic Evaluation of Dialectical Behavior Therapy-Oriented Treatment for Borderline Personality Disorder,” Cognitive and Behavioral Practice, Vol. 7, No. 4, Fall 2000, pp. 413–419.

176 Bibliography

Van der Sande, R., L. van Rooijen, E. Buskens, E. Allart, K. Hawton, Y. van der Graaf, and H. van Engeland, “Intensive In-Patient and Community Intervention Versus Routine Care After Attempted Suicide: A Randomised Controlled Intervention Study,” British Journal of Psychiatry, Vol. 171, No. 1, July 1997, pp. 35–41. Verheul, Roel, Louise M. C. van den Bosch, Maarten W. J. Koeter, Maria A. J. De Ridder, Theo Stijnen, and Wim Van Den Brink, “Dialectical Behaviour Therapy for Women with Borderline Personality Disorder,” British Journal of Psychiatry, Vol. 182, No. 2, February 2003, pp. 135–140. Vieland, Veronica, Barry Whittle, Ann Garland, Roger Hicks, and David Shaffer, “The Impact of Curriculum-Based Suicide Prevention Programs for Teenagers: An 18-Month Follow-Up,” Journal of the American Academy of Child and Adolescent Psychiatry, Vol. 30, No. 5, September 1991, pp. 811–815. Wasserman, Danuta, Vladimir Carli, Camilla Wasserman, Alan Apter, Judit Balazs, Julia Bobes, Renata Bracale, Romuald Brunner, Cendrine Bursztein-Lipsicas, Paul Corcoran, Doina Cosman, Tony Durkee, Dana Feldman, Julia Gadoros, Francis Guillemin, Christian Haring, Jean-Pierre Kahn, Michael Kaess, Helen Keeley, Dragan Marusic, Bogdan Nemes, Vita Postuvan, Stella Reiter-Theil, Franz Resch, Pilar Saiz, Marco Sarchiapone, Merike Sisask, Airi Varnik, and Christina W. Hoven, “Saving and Empowering Young Lives in Europe (SEYLE): A Randomized Controlled Trial,” BMC Public Health, Vol. 10, 2010. Weiner, I. W., “The Effectiveness of a Suicide Prevention Program,” Mental Hygiene, Vol. 53, No. 3, July 1969, pp. 357–363. Welu, Thomas C., “A Follow-Up Program for Suicide Attempters: Evaluation of Effectiveness,” Suicide and Life-Threatening Behavior, Vol. 7, No. 1, Spring 1977, pp. 17–20. Wholey, Joseph S., Harry P. Hatry, and Kathryn E. Newcomer, eds., Handbook of Practical Program Evaluation, Vol. 19, San Francisco, Calif.: Jossey-Bass, 2010. Wilcox, Holly C., Sheppard G. Kellam, C. Hendricks Brown, Jeanne M. Poduska, Nicholas S. Ialongo, Wei Wang, and James C. Anthony, “The Impact of Two Universal Randomized First- and Second-Grade Classroom Interventions on Young Adult Suicide Ideation and Attempts,” Drug and Alcohol Dependence, Vol. 95, Suppl. 1, 2008, pp. S60–S73. Wislar, Joseph S., Janet Grossman, Markus J. P. Kruesi, Michael Fendrich, Carole Franke, and Nancy Ignatowicz, “Youth Suicide-Related Visits in an Emergency Department Serving Rural Counties: Implications for Means Restriction,” Archives of Suicide Research, Vol. 4, No. 1, 1998, pp. 75–87. W. K. Kellogg Foundation, Logic Model Development Guide, Battle Creek, Mich., October 2000. World Health Organization, Western Pacific Region, Towards Evidenced-Based Suicide Prevention Programmes, 2010. As of March 19, 2013: http://www.wpro.who.int/mnh/TowardsEvidencebasedSPP.pdf Wyman, Peter A., C. Hendricks Brown, Jeff Inman, Wendi Cross, Karen Schmeelk-Cone, Jing Guo, and Juan B. Pena, “Randomized Trial of a Gatekeeper Program for Suicide Prevention: 1-Year Impact on Secondary School Staff,” Journal of Consulting and Clinical Psychology, Vol. 76, No. 1, February 2008, pp. 104–115. Wyman, Peter A., C. Hendricks Brown, Mark LoMurray, Karen Schmeelk-Cone, Mariya Petrova, Qin Yu, Erin Walsh, Xin Tu, and Wei Wang, “An Outcome Evaluation of the Sources of Strength Suicide Prevention Program Delivered by Adolescent Peer Leaders in High Schools,” American Journal of Public Health, Vol. 100, No. 9, September 2010, pp. 1653–1661. Yip, Paul S. F., C. K. Law, King-Wa Fu, Y. W. Law, Paul W. C. Wong, and Ying Xu, “Restricting the Means of Suicide by Charcoal Burning,” British Journal of Psychiatry, Vol. 196, No. 3, March 2010, pp. 241–242. Zenere, Frank J. III, and Philip J. Lazarus, “The Decline of Youth Suicidal Behavior in an Urban, Multicultural Public School System Following the Introduction of a Suicide Prevention and Intervention Program,” Suicide and Life-Threatening Behavior, Vol. 27, No. 4, Winter 1997, pp. 387–402.

177 Evaluating suicide prevention programs can be challenging because suicide is a rare event, data on suicides often lag by several years, and programs tend to have multiple components, making it difficult to discern which characteristics contributed to a given outcome. The RAND Suicide Prevention Program Evaluation Toolkit was designed to help program staff overcome these common challenges to evaluating and planning improvements to their programs. It begins by walking users through the process of developing a program logic model that ties program activities to intermediate outcomes, helping staff better understand the drivers of any changes in long-term outcomes, such as suicide rates. It then offers information about the latest evaluation research, helps users design an evaluation that is appropriate for their program type and available resources and expertise, supports the selection of measures for new evaluations and to augment or enhance ongoing evaluations, and offers basic guidance on how to analyze and use evaluation data for program improvement. Through checklists, worksheets, and templates, the toolkit takes users step by step through the process of identifying whether their programs produce beneficial effects, ultimately informing the responsible allocation of scarce resources. The toolkit’s design and content are the result of a rigorous, systematic review of the program evaluation literature to identify evaluation approaches, measures, and tools used elsewhere and will be particularly useful to coordinators and directors of suicide prevention programs in the U.S. Department of Defense, Veterans Health Administration, community-based settings, and state and local health departments. A companion report, Devel- opment and Pilot Test of the RAND Suicide Prevention Program Evaluation Toolkit, offers additional background on the toolkit’s design and refinement.

$70.00 NATIONAL DEFENSE RESEARCH INSTITUTE

T L-111- O S D