CUA

THE CATHOLIC UNIVERSITY OF AMERICA

National Catholic School of Social Service Washington, DC 20064

SSS 757 Evaluation of Social Work Programs Fall, 2014 (3 credits)

Wendy Whiting Blome, LICSW, PhD 202-319-5458 [email protected]

© This course outline is the property of NCSSS and the instructor and may be distributed with written permission.

I. COURSE PURPOSE

Building upon the foundation research course, this course presents the basic theoretical and methodological concepts of program evaluation and seeks to engage students in a critical examination of research evidence. The class includes quantitative and qualitative approaches, non-experimental and experimental designs, and statistical and other analytical techniques. Students are challenged to think about evaluation in the context of the budget and time limitations that exist within agencies. Through writing a research proposal, the students will apply their knowledge to a real world situation. Attention is given to ethical concerns in program evaluation as well as to issues related to research with diverse and at-risk populations.

II. COMPETENCIES AND PRACTICE BEHAVIORS

The Council on Social Work Education (CSWE) requires that students meet 10 core competencies, which are operationalized as practice behaviors. Each course is designed to cover one or more of the ten core competencies and each course is also designed to cover some, but not all of the practice behaviors within a competency. Upon completion of this course, students will be able to demonstrate the following practice behaviors within the noted competencies:

1 Date Revised August 2014 Competency Practice Behaviors

Ethical Practice: Social workers are 1. Social workers recognize the complex knowledgeable about the value base of the ethical challenges that arise in a variety of profession, its ethical standards, and macro practice settings principals. 2. Social workers are able to identify and Social workers can apply ethical apply ethical principles to an ethical knowledge to practice settings. dilemma they have experienced in a practice setting.

Critical Thinking: Apply critical thinking 1. Social workers distinguish, critically to inform and communicate professional analyze, and integrate multiple sources of judgments knowledge, including research-based knowledge, theoretical frameworks, and practice wisdom, in an ongoing process of assessment and intervention.

2. Social workers effectively communicate their ideas when producing a written document or giving a presentation or briefing on a particular issue.

Research Based Practice: Engage in 1. Social workers critically evaluate and research-informed practice and practice- utilize theoretical frameworks and informed research empirical research relevant to the problems and/or populations that are relevant to their macro practice.

2. Social workers demonstrate the capacity to apply the evaluation continuum from needs assessment to process and outcome evaluation.

Engage, Assess, Intervene, Evaluate: 1. Social workers participate in Engage, assess, and intervene with groups, monitoring and evaluating the processes organizations, and communities. used and the outcome of their interventions to seek organizational, community, or societal change.

2. Social workers are able to develop an evaluation proposal that includes a data analysis plan and potential implications based on the outcome of their evaluation. III. EDUCATIONAL OBJECTIVES

1. To appreciate the purposes of evaluation as used by social workers.

2 2. To understand research designs used in program evaluation.

3. To recognize the strengths and limitations of each research design.

4. To be aware of the ethical and value issues that arise in conducting program evaluation studies.

5. To be able to select appropriate measurement strategies in developing an evaluation plan.

6. To know the statistical methods used in program evaluation.

7. To appreciate the procedures used in conducting a qualitative evaluation.

8. To understand the impact of budgetary and time limitations on evaluation design.

9. To recognize the political and social implications of program evaluation results.

10. To appreciate the need for agencies to invest in program evaluation activities and to offer evidence based services to people in need.

11. To understand the connection between program evaluations with vulnerable populations and the promotion of economic and social justice.

12. To demonstrate competence in developing a reasoned critique of a published social work program evaluation study.

13. To exhibit proficiency in developing and writing a proposal for conducting a social work program evaluation study.

IV. COURSE REQUIREMENTS

A. Required Texts

Rossi, P., Lipsey, M., & Freeman, H. (2004). Evaluation: A systematic approach (7th ed.). Thousand Oaks, CA: Sage Publications.

Salkind, N. (2014). Statistics for people who (think they) hate statistics (5th ed). Thousand Oaks, CA.: Sage Publications, Inc.

Yuen, F., Terao, K., & Schmidt, A. (2009). Effective grant writing and program evaluation for human service professionals. Hoboken, NJ: John Wiley & Sons.

The Program Manager’s Guide to Evaluation, Administration for Children and Families, Office of Planning, Research, and Evaluation, (2nd ed): http://www.acf.hhs.gov/programs/opre/other_resrch/pm_guide_eval/report s/pmguide/program_managers_guide_to_eval2010.pdf

3 Date Revised August 2014 Other required readings are available through web links or Blackboard.

B. Recommended Reading

Bamberger, M., Rugh, J., & Mabry, L. (2006). Real world evaluation: Working under budget, time, data, and political constraints. Thousand Oaks, CA: Sage Publications, Inc.

Royse, D., Thyer, B., & Padgett, D. (2010). Program evaluation: An introduction. Belmont, CA: Wadsworth Press.

C. Other Recommended Resources and Media

National Catholic School of Social Service: http://ncsss.cua.edu/ Administration for Children, Youth and Families, Office of Planning, Research and Evaluation: http://www.acf.hhs.gov/programs/opre/ Framework for Program Evaluation in Public Health: ftp://ftp.cdc.gov/pub/Publications/mmwr/rr/rr4811.pdf American Evaluation Association: http://www.eval.org/ University of Wisconsin—Extension, Program Development and Evaluation: http://www.uwex.edu/ces/pdande/evaluation/ Basic Guide to Program Evaluation, Free Management Library: http://managementhelp.org/evaluatn/fnl_eval.htm#anchor1575679

D. Course Assignments

Assignment 1: Assessment of Agency Evaluation and Research Activities. Each student is required to complete an interview with his or her field instructor to determine the degree to which program evaluation activities are occurring within the agency, how the results of the evaluation are being used by the agency, and how individual social workers are involved in evaluation activities. This interview should be summarized in a 2 to 3 page descriptive paper. Due Class 3.

Assignment 2: Evaluation Proposal—Part 1. Students are required to develop an evaluation plan suitable for submission as part of a grant proposal. The assignment will be divided into two parts. Part 1 will include the problem statement, literature search, evaluation question(s), and whether the question(s) will be addressed through a needs assessment, process evaluation, or outcome evaluation. This 6 to 8 page paper should be written in APA style. Due Class 7.

Assignment 3: Critical Review of Assigned Articles. In classes 5, 6, and 7 articles from the program evaluation literature are assigned. The student will write a 5-7 page critical review of one of the articles discussing the research problem, methodology, data analysis, and findings. APA style is expected. Due Class 9.

Assignment 4: Evaluation Proposal—Part 2. This section of the evaluation proposal will include the research design, sampling plan, data collection and analysis plan, and policy and/or programmatic implications. Students should

4 revise Part 1, based on the instructor’s comments, in order to make it consistent with Part 2. The grade for Part 1 will not be changed, but the paper as a whole will be graded for Part 2. Part 2 is an additional 6 to 8 pages. Due Class 13.

Assignment 5: Final Examination. An in-class final examination will be held during the University exam period. Date to be discussed.

E. Grading Policy and Weights of Assignments

Grades will be based on the CUA Grading Policy as described in the Graduate Announcements. Full credit will not be given for assignments that are submitted late. No credit will be given for assignments submitted after they have been reviewed in class. The following provides weights for the various course assignments:

Assessment of Agency 10% Critical Reviews (10% each) 20% Evaluation Proposal—Part 1 15% Evaluation Proposal—Part 2 20% Final Examination 25% Class Participation 10%

Grading System Letter Grade Numeric Range A 95 – 100 A- 90 – 94 B+ 87 – 89 B 83 – 86 B- 80 – 82 C 70 - 79 F 0 - 69

5 Date Revised August 2014 F. Preparation, Attendance & Participation

Students are required to attend classes and are expected to participate meaningfully in class discussion/exercises and online forums as required. The class participation grade will be determined by the instructor’s perception of the student’s preparation for and contributions to class discussion/activities. Different students will make different kinds of contributions. Some will have an easy time with spontaneous interactions while others will be more comfortable making planned statements about key ideas from the readings or other sources. Both types of contributions are valued. Unexcused absences will lower the attendance and participation grade. More than three unexcused absences will result in a grade of F.

G. Course and Instructor Evaluation

NCSSS requires electronic evaluation of this course and the instructor. At the end of the semester, the evaluation form may be accessed at http://evaluations.cua.edu/evaluations using your CUA username and password. Additional, informal written or verbal feedback to the instructor during the semester is encouraged and attempts will be made to respond to requests.

V. CLASS EXPECTATIONS

Please refer to NCSSS Announcements, or appropriate Program Handbook for Academic Requirements (http://ncsss.cua.edu/courses/index.cfm), including scholastic and behavioral requirements.

NCSSS is committed to creating an open and inclusive learning environment where all members - including students, faculty, administrators, and staff – strive to listen to and learn from one another. We recognize that in a multicultural society, it is inevitable that issues or tensions relative to diversity and different life experiences will arise. It is how we handle these events that matters. Therefore, when such issues occur – inside or outside of the classroom - we agree to engage in respectful and productive discussion with one another until learning is enhanced and understanding is deepened by all involved.

A. Scholastic Expectations All written work should reflect the original thinking of the writer, cite references where material is quoted or adapted from existing sources, adhere to APA format, and should be carefully proof read by the student before submission to the instructor for grading.

B. Behavioral Requirements Students are expected to maintain accepted standards of professional conduct and personal integrity in the classroom. Students should:  Attend all classes and contribute constructively to the classroom culture,

6  Recognize and avoid behavior that jeopardizes the learning/teaching environment of other students or the instructor,  Demonstrate competence in planning academic activities and in following through on those plans,  Reasonably respond to and respect others’ reactions to one’s comments or actions in the classroom,  Use an appropriate level of class time and instructor’s time and attention in and out of class, and  Behave in a manner that is consistent with the ethical principles of the social work profession.

C. Academic Honesty Joining the community of scholars at CUA entails accepting the standards, living by those standards, and upholding them. Please refer to University Policy (http://graduatestudies.cua.edu/currentstudents/academintgrt.cfm) and appropriate Program Handbooks.

D. Confidentiality Each student is expected to adhere to the Confidentiality Agreement that is signed at the beginning of every semester. This agreement covers “practice materials” in classes, supervisory sessions, case conferences, seminars, and other educational settings within the NCSSS BSW or MSW programs are for professional learning purposes only and are subject to strict professional standards of confidentiality. These same standards of confidentiality also extend to various forms of written communication and peer consultation.

Adherence to these standards means all students refrain from communicating beyond the classroom setting about practice material that is presented in class. I will also refrain from using social media outlets (blogs, twitter, Facebook, etc.) or email to discuss practice settings, program responsibilities and projects with individuals who are not in teaching or supervision roles directly related to the situation.

E. Accommodations Students with physical, learning, psychological or other disabilities wishing to request accommodations must identify with the Disability Support Services (DSS) and submit documentation of a disability. If you have documented such a disability to DSS that requires accommodations or an academic adjustment, you much present that documentation to your instructors and arrange a meeting with as soon as possible to discuss these accommodations.

F. Use of Electronic Devices Laptops can be used for note-taking during class. Please turn off all cell phones or other devices that would disrupt the learning environment of the classroom.

7 Date Revised August 2014 Class Schedule

Class Topics and Readings

1 Overview of Statistics and Research Terms

Required Reading

NCSSS Review sheet: Distributed in class

Salkind, Chapter 1: Statistics or sadistics? It’s up to you

Yuen, Terao, & Schmidt, Chapter 1: Introduction

Yuen, Terao, & Schmidt, Chapter 2: Community and target populations, service providers, and funding sources.

Basic Guide to Program Evaluation, Free Management Library: http://managementhelp.org/evaluatn/fnl_eval.htm#anchor1575679 Read the section on Myths about Program Evaluation and review the rest of the document so you can use it as a reference when writing your papers.

2 Introducing Program Evaluation, Identifying Issues, and Formatting Questions

Required Reading

Rossi, Chapter 1: An overview of program evaluation

Rossi, Chapter 2: Tailoring evaluations

Rossi, Chapter 3: Identifying issues and formulating questions

3 Developing and Using Logic Models

Required Reading

Bliss, D. (2007). Implementing an outcomes measurement system in substance abuse treatment programs. Administration in Social Work, 31(4), 83-101.

Carman, J. (2010). The accountability movement: What’s wrong with this theory of change. Nonprofit and Voluntary Sector Quarterly, 39(2), 256- 274.

Children’s Bureau, Child Welfare Information Gateway website, review logic model materials: http://www.childwelfare.gov/management/effectiveness/

8 Kellogg Foundation. (2004). Logic model development guide. Author: Battle Creek, MI. Retrieved from: http://ww2.wkkf.org/DesktopModules/WKF.00_DmaSupport/ViewDoc.asp x?LanguageID=0&CID=284&ListID=28&ItemID=2813669&fld=PDFFile

Savaya, R., & Waysman, M. (2005). The logic model: A tool for incorporating theory in development and evaluation of programs. Administration in Social Work, 29(2), 85-103.

4 Developing Program Theory and Needs Assessment

Required Reading

Rossi, Chapter 4: Assessing the need for a program

Rossi, Chapter 5: Expressing and assessing program theory

Kidd, S., & Scrimenti, K. (2004). Evaluating child and youth homelessness: The example of New Haven, Connecticut. Evaluation Review, 28(4), 325-341.

Rios-Ellis, B., Frantes, J., D’Anna, L., Dwyer, M., Lopez-Zetina, J., & Ugarte, C. (2008). Addressing the need for access to culturally and linguistically appropriate HIV/AIDS prevention for Latinos. Journal of Immigrant Minority Health, 10, 445–460.

Salkind, Chapter 7: Hypotheticals and you: Testing your questions

5 Conducting Process Evaluations

Required Reading

Rossi, Chapter 6: Assessing and monitoring program process

Esbensen, F., Matsuda, K., Taylor, T., & Peterson, D. (2011). Multimethod strategy for assessing program fidelity: The national evaluation of the revised GREAT program. Evaluation Review, 35(1), 14-39.

Forsetlund, L, Talseth, K., Bradley, P., Nordheim, L., & Bjorndal, A. (2003). Many a slip between cup and lip: Process evaluation of a program to promote and support evidence-based public health practice. Evaluation Review, 27(2), 179-209.

Salkind, Chapter 11: t(ea) for two: Tests between the means of different groups

Salkind, Chapter 12: t(ea) for two (again): Tests between the means of related groups

9 Date Revised August 2014 6 Evaluating Outcomes I

Required Reading

Rossi, Chapter 7: Measuring and monitoring program outcomes

Bamberger, Chapter 14: Sampling for real world evaluation

English, D., Brummel, S., & Martens, P. (2009) Fatherhood in the child welfare system: Evaluation of a pilot project to improve father involvement. Journal of Public Child Welfare, 3(3), 213-234.

Mears, S., Yaffe, J., & Harris, N. (2009). Evaluation of wraparound services for severely emotionally disturbed youth. Research on Social Work Practice, 19(6), 678-685.

Salkind, Chapter 6: Just the truth: An introduction to understanding reliability and validity

7 Evaluating Outcomes II

Required Reading

Rossi, Chapter 8: Assessing program impact: Randomized field experiments

Rossi, Chapter 9: Assessing program impact: Alternative designs

Brooks, F., Russell, D., & Fisher, R. (2006). ACORN’s accelerated income redistribution project: A program evaluation. Research on Social Work Practice, 16(4), 369-381.

Johnson, K., & Wagner, D. (2005). Evaluation of Michigan’s foster care case management system. Research on Social Work Practice, 15(5), 372-380.

Laudet, A, Stanick, V., & Sands, B. (2007). An exploration of the effect of on- site 12-step meetings on post-treatment outcomes among polysubstance dependent outpatient clients. Evaluation Review, 31(6), 613-646.

Salkind, Chapter 17: What to do when you’re not normal—Chi-square and some other nonparametric tests

8 Analyzing Data for Program Evaluation

Required Reading

Rossi, Chapter 10: Detecting, interpreting, and analyzing program effects

10 Corcoran, J. & Pillai, V. (2007). Effectiveness of secondary pregnancy prevention programs: A meta-analysis. Research on Social Work Practice, 17(1), 5-18.

Crew, R., & Lamothie, S. (2003). Evaluating the efficiency of private sector organizations in helping welfare beneficiaries find employment. Evaluation Review, 27(2), 151-164.

Salkind, Chapter 9: Significantly significant: What it means for you and me

Salkind, Chapter 13: Two groups too many? Try analysis of variance

Salkind, Chapter 15: Cousins or just good friends? Testing relationships using the correlation coefficient

9 Conducting Cost Benefit and Cost Effectiveness Analyses

Required Reading

Rossi, Chapter 11: Measuring efficiency

McLaughlin, T. (2011). Using common themes: Cost-effectiveness of permanent supported housing for people with mental illness. Research on Social Work Practice, 21(4), 404-411.

McNutt,J. (2011). Is social work advocacy worth the cost? Issues and barriers to an economic analysis of social work political practice. Research on Social Work Practice, 21(4), 397-403.

Shanahan, M., Lancsar, E., Haas, M., Lind, B., Weatherburn, D., & Chen, S. (2004). Cost-effectiveness analysis of the New South Wales Adult Drug Court Program. Evaluation Review, 28(1), 3-27.

10 Using Qualitative Methods in Program Evaluation

Required Reading

Bamberger, Chapter 12: Qualitative evaluation methods

Jenson, C., Spath, B., & Kerman, B. (2009). Developing strong helping alliances in family reunification. Journal of Public Child Welfare, 3(4), 331-353.

Powell, E., & Renner, M. (2003). Analyzing qualitative data. University of Wisconsin—Extension. Retrieved from http://learningstore.uwex.edu/pdf/G3658-12.pdf

11 Date Revised August 2014 FRIENDS National Resource Center for Community-Based Child Abuse Prevention. (2009). Using qualitative data in program evaluation: Telling the story of a prevention program. Retrieved from http://friendsnrc.org/joomdocs/qualitativedata.pdf

11 Developing an Evaluation Proposal—I

Required Reading

Bamberger, Chapter 3: Not enough money: Addressing budget constraints

Royse, D., Thyer, B., & Padgett, D. Chapter 15: Writing evaluation proposals, reports, and journal articles.

Contents of an evaluation plan: http://managementhelp.org/evaluation/program- evaluation-guide.htm#anchor1586742

The Program Manager’s Guide to Evaluation, Administration for Children and Families, Office of Planning, Research, and Evaluation, (2nd ed): How do you hire and manage an outside evaluator? Pages 19-28: http://www.acf.hhs.gov/programs/opre/other_resrch/pm_guide_eval/reports /pmguide/program_managers_guide_to_eval2010.pdf

12 Developing an Evaluation Proposal—II

Required Reading

Salkind, Chapter 4: A picture really is worth a thousand words

Salkind, Chapter 21: The ten commandments of data collection

Yuen, Terao, & Schmidt, Chapter 3: Basic research methods and program evaluation

Yuen, Terao, & Schmidt, Chapter 4: Program planning and evaluation

Yuen, Terao, & Schmidt, Chapter 6: Program evaluation

13 Putting It Together: Model Program

Required Reading

Bamberger, Chapter 16: Bringing it all together: Applying real world evaluation approaches to each stage of the evaluation process

The Program Manager’s Guide to Evaluation, Administration for Children and Families, Office of Planning, Research, and Evaluation, (2nd ed): ): How can you report on what you have learned? Pages 87-95:

12 http://www.acf.hhs.gov/programs/opre/other_resrch/pm_guide_eval/reports /pmguide/program_managers_guide_to_eval2010.pdf

14 Using Evaluation Results

Required Reading

Rossi, Chapter 12: The social context of evaluation

The Program Manager’s Guide to Evaluation, Administration for Children and Families, Office of Planning, Research, and Evaluation, (2nd ed): http://www.acf.hhs.gov/programs/opre/other_resrch/pm_guide_eval/report s/pmguide/program_managers_guide_to_eval2010.pdf Read pages 77-90.

15 Final Examination—TBA

13 Date Revised August 2014