Final Report on the Evaluation of the National Science Foundation Louis Stokes Alliances for Minority Participation Program Prepared for the National Science Foundation Directorate for Education and Human Resources Division of Research, Evaluation and Communication Program for Evaluation and Equity Research (PEER) The Urban Institute 2100 M Street NW Washington, DC 20037 November 2005 Final Report Evaluation of the National Science Foundation Louis Stokes Alliances for Minority Participation Program Prepared under Contract REC 9912176 Beatriz Chu Clewell Clemencia Cosentino de Cohen Lisa Tsui Laurie Forcier Ella Gao Nicole Young Nicole Deterding Caroline West Program for Evaluation and Equity Research (PEER) The Urban Institute Prepared for The National Science Foundation Directorate for Education and Human Resources Division of Research, Evaluation and Communication November 2005 The data collection, analysis and reporting of this material was conducted in accordance with OMB No. 3145-0190. Note: Any opinions, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the United States Government or The Urban Institute. National Science Foundation, Division of Research, Evaluation and Communication, Evaluation of the Louis Stokes Alliance for Minority Participant Program, NSF 05-XXX (Arlington, VA, 2005). THE URBAN INSTITUTE The Urban Institute is a nonprofit, nonpartisan policy research and educational organization established in Washington, D.C., in 1968. Its staff investigates the social, economic, and governance problems confronting the nation and evaluates the public and private means to alleviate them. The Institute disseminates its research findings through publications, its web site, the media, seminars, and forums. Through work that ranges from broad conceptual studies to administrative and technical assistance, Institute researchers contribute to the stock of knowledge available to guide decisionmaking in the public interest. Conclusions or opinions expressed in this report are those of the authors and do not necessarily reflect the views of officers or trustees of the Institute, advisory groups, or any organizations that provide financial support to the Institute. Program for Evaluation and Equity Research (PEER) The Urban Institute’s Program for Evaluation and Equity Research (PEER) focuses on education research. PEER staff conduct studies in the fields of educational attainment, educational access, minorities in mathematics and science, teacher education, teacher recruitment and retention, and educational assessments. Much of PEER’s work has centered around evaluation studies—many of these large, multisite, multi-method evaluations of programs to increase educational access and success among underrepresented groups, as well as programs to increase the teaching pool. Support for PEER comes from multiple sources, including the National Science Foundation and the American Association for the Advancement of Science, as well as private foundations, such as the Ford Foundation, DeWitt Wallace Reader’s Digest Fund, Lumina Foundation, and GE Foundation. Acknowledgments We would like to acknowledge the invaluable cooperation and assistance that we received from Louis Stokes Alliances for Minority Participation (LSAMP) program directors, staff, faculty, and student participants (both past and present), which enabled us to collect a wealth of data on which this evaluation rests. Their willingness to share their experiences and insights contributed greatly to our understanding of how individual projects function, and the commonalities and differences that exist within the overall Program. We are particularly grateful to LSAMP staff members and participants at the case study institutions for effectively coordinating and graciously hosting the case study site visits, which proved to be informative as well as enjoyable. We are also grateful to the staff at the National Science Foundation (NSF) for their support and guidance. In particular, we are indebted to A. James Hicks, the program officer for LSAMP, for his strong leadership and responsiveness to our requests for assistance throughout the project. We are also very appreciative of Elmima Johnson, our program officer, whose encouragement and guidance facilitated our work. We benefited tremendously from the invaluable assistance and feedback received from Bernice Anderson, acting director of human resource development. John Tsapogas in Social Behavioral and Economic Sciences/Science Resource Statistics was extremely helpful in answering questions about the National Survey of Recent College Graduates (NSRCG) and facilitating our access to the NSRCG data. Our special thanks to members of the project’s advisory panel: Gerald E. Gipp, Carlos Rodriguez, William Sibley, and Floraline I. Stevens. The insight and suggestions they offered early in the evaluation were helpful in refining its design and contributed substantially to the evaluation’s direction. We wish to acknowledge the invaluable assistance of NuStats, our subcontractor on this project. In particular, we are deeply grateful for the tremendous contributions that Rob Santos, vice president of NuStats, made to our study. Early on and throughout the project, his expertise and advice strengthened our work. Finally, we wish to thank our colleagues at the Urban Institute: LaTasha Holloway, who provided assistance throughout the long process of data collection, analysis, and report writing and Gary Gates, who advised us on the use of STATA during the analysis phase. The Evaluation Team THE URBAN INSTITUTE Program for Evaluation and Equity Research (PEER) Contents List of Tables and Figures viii Executive Summary 1 Section I: Introduction Statement of the Problem 4 Brief Description and Overview of LSAMP 5 Section II: Evaluation Design and Methodology Evaluation Questions 7 Evaluation Design and Methodology 8 Section III: Process Evaluation Findings Telephone Interviews 19 Critical Issues Emerging from the Case Studies 28 Section IV: The LSAMP Model Research and Theory Underlying Approaches and Strategies Academic and Social Integration: The Tinto Model of Student Retention 33 Socialization into Science: The Professionalization of Scientists 34 The LSAMP Model 35 Section V: Summative Evaluation Impact on Student Participants 40 Impact on the STEM Workforce 48 Impact on the Knowledge Base: The LSAMP Model 49 Impact on Participating IHEs 52 Section VI: Conclusions and Recommendations Conclusions 54 Recommendations 55 References 57 Appendix A: Literature Review on Effective Strategies to Increase Diversity in STEM Fields A-1 Appendix B: Telephone Interview Protocols B-1 Appendix C: LSAMP Graduate Survey Materials C-1 Appendix D: Site Visit Interview and Focus Group Protocols D-1 Appendix E: Case Studies E-1 Appendix F: Summative Evaluation Tables F-1 THE URBAN INSTITUTE vii Program for Evaluation and Equity Research (PEER) List of Tables and Figures Table 1: Number of Interviews and Focus Groups Conducted at Each Alliance 12 Table 2: Respondents to the LSAMP Retrospective Survey by Mode Employed 15 Table 3: Composition of the National Comparative Sample 17 Figure 1: Timing of LSAMP Site Visits 11 Figure 2: Number of Alliances Created, by Year 19 Figure 3: Types of Advisory Committees 20 Figure 4: Precollege Development 23 Figure 5: Student Academic Development Activities 24 Figure 6: Student Professional Development Activities 24 Figure 7: Types of Mentoring Programs 25 Figure 8: Faculty Development 26 Figure 9: Curriculum Development Activities 26 Figure 10: Graduate Studies Development 27 Figure 11: Community College Components 27 Figure 12: Top 5 Components 28 Figure 13: The LSAMP Model 36 Figure 14: Strategies and Approaches: Elements of the LSAMP Model 37 Figure 15: Undergraduate GPA: LSAMP Participants 41 Figure 16: Degrees Sought: Post-Bachelor’s Coursework 42 Figure 17: Field of Study Pursued at Graduate Level: LSAMP Participants 43 Figure 18: Degrees Completed at Time of Survey 44 Figure 19: Graduate Coursework, Degrees Pursued, and Degrees Completed 47 THE URBAN INSTITUTE viii Program for Evaluation and Equity Research (PEER) Executive Summary The Louis Stokes Alliances for Minority Participation (LSAMP) Program was established in 1991 by the National Science Foundation (NSF) to develop strategies to increase the quality and quantity of minority students who successfully complete baccalaureate degrees in science, technology, engineering, and mathematics (STEM) and who continue on to graduate studies in these fields. The Urban Institute was commissioned to conduct an evaluation of the LSAMP Program, an evaluation that would answer questions about the structure and implementation of LSAMP and its impact on students, participating institutions of higher education (IHEs), and the diversity of the STEM workforce. The information presented in this report comes from the Urban Institute’s multiyear evaluation of the LSAMP program. The LSAMP Program began with grants to six multi-institution Alliances across the country. Today there are 34 Alliances with over 450 participating institutions that have produced thousands of STEM degrees. Distinguishing it from traditional scholarship programs, LSAMP takes a multidisciplinary approach to student development and retention, creating partnerships among colleges, universities, national research laboratories, business and industry, and other federal agencies in order to accomplish its goals. Hands-on research experiences