External Evaluation Report October 2015 Final Report Prepared by Josh Lang, MA and Atiera Coleman, MA Center for Urban Initiatives and Research University of Wisconsin-Milwaukee Acknowledgement The UWM Center for Urban Initiatives and Research wishes to express their sincere gratitude to Erin Guest and Sean Luedke of Public Allies for their assistance in data collection, data access, and survey development for this evaluation . 1 Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research Contents I. ............................................................................................................................................... Summary4 II. ........................................................................................................................................ Background7 Evaluation Team ............................................................................................................................ 7 What is Public Allies? .................................................................................................................... 9 Ten-Month Apprenticeship Program ......................................................................................... 9 Program Reach ........................................................................................................................... 110 III................................................................................................................Evaluation Description13 Purpose of Evaluation ................................................................................................................. 13 Overall Evaluation Design .......................................................................................................... 13 Evaluation Questions .................................................................................................................. 14 IV. .............................................................................................................Evaluation Components15 Partner Organizations .......................................................................................................... 15 Partner Organization Outcome Measures................................................................................ 16 Outcome Measure1 .................................................................................................................. 16 Outcome Measure 2 ................................................................................................................ 17 Data Collection Instruments ...................................................................................................... 17 Data Collection Procedures ........................................................................................................ 18 Results ........................................................................................................................................... 17 Sample Size ............................................................................................................................... 17 Improved Technical Capacity ................................................................................................ 22 Capacity Change Scores .......................................................................................................... 31 Adaptive Capacity .................................................................................................................... 33 Implications .................................................................................................................................. 35 Recommendations ....................................................................................................................... 36 Limitations ................................................................................................................................... 37 Current Ally Apprentices ..................................................................................................... 37 Current Ally Retrospective and Post Ally Survey Outcome Measures ................................. 38 Outcome Measure 1: Increased 21st Century Skills ............................................................. 38 Socially Responsible Leadership Scale (SRLS-R2) ............................................................. 38 Collaboration ............................................................................................................................ 39 Controversy with Civility. ....................................................................................................... 39 Leadership ................................................................................................................................ 39 Self Confidence ........................................................................................................................ 39 Resilience .................................................................................................................................. 41 Outcome Measure 2: Increased Civic Knowledge ............................................................... 41 Outcome Measure 3: Increased Interest in Higher Education ......................................... 41 Data Collection Instruments ...................................................................................................... 42 Data Collection Procedures ........................................................................................................ 42 Results ........................................................................................................................................... 42 Current Ally, Retrospective Survey Between-group Outcomes Comparison. ................. 45 2 Prepared by University of Wisconsin – Milwaukee Center for Urban Initiatives and Research Current Ally, Post Survey Between-group Outcomes Comparison. ................................. 46 Current Allies: Disconnected Allies Outcome Comparison ............................................... 47 Current Ally, Connected Ally Outcome Comparison .......................................................... 48 Implications .................................................................................................................................. 53 Public Allies Alumni .............................................................................................................. 56 Alumni Ally Outcome Measures ................................................................................................ 57 Outcome Measure 1: Civic engagement ................................................................................ 57 Outcome Measure 2: Increased Employment in Family Sustaining Careers.................. 57 Outcome Measure 3: Increased Pursuit of Higher Education .......................................... 57 Data Collection Instruments ...................................................................................................... 58 Data Collection Procedures ........................................................................................................ 58 Results ........................................................................................................................................... 60 Alumni Ally Outcome Comparison. ...................................................................................... 60 Alumni Allies, Disconnected During the Program Outcome Comparison ...................... 60 Alumni Allies, Connected Ally Outcome Comparison. ....................................................... 62 Alumni Pursuit of Higher Education .................................................................................... 62 Alumni Increased Employment ............................................................................................. 63 Implications .................................................................................................................................. 65 Limitations ................................................................................................................................... 67 Public Allies Site Directors ................................................................................................. 67 Data Collection Instruments and Procedures...................................................................... 68 Results ....................................................................................................................................... 69 Implications .............................................................................................................................. 74 Recommendations ................................................................................................................... 75 Limitations ............................................................................................................................... 75 Appendix A: CNCS Charts ......................................................................................................... 75 Appendix B: Partner Organization Theory of Change ........................................................... 76 Appendix C: Capacity Self-Assessment for Partner Organizations ..................................... 77 Appendix D: Ally-focused Theory of Change .......................................................................... 89 Appendix E