Final Report on the Evaluation of the National Science Foundation Louis Stokes Alliances for Minority Participation Program
Prepared for the National Science Foundation Directorate for Education and Human Resources Division of Research, Evaluation and Communication
Program for Evaluation and Equity Research (PEER) The Urban Institute 2100 M Street NW Washington, DC 20037
November 2005
Final Report
Evaluation of the National Science Foundation Louis Stokes Alliances for Minority Participation Program
Prepared under Contract REC 9912176
Beatriz Chu Clewell Clemencia Cosentino de Cohen Lisa Tsui Laurie Forcier Ella Gao Nicole Young Nicole Deterding Caroline West
Program for Evaluation and Equity Research (PEER) The Urban Institute
Prepared for The National Science Foundation Directorate for Education and Human Resources Division of Research, Evaluation and Communication
November 2005
The data collection, analysis and reporting of this material was conducted in accordance with OMB No. 3145-0190.
Note: Any opinions, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the United States Government or The Urban Institute.
National Science Foundation, Division of Research, Evaluation and Communication, Evaluation of the Louis Stokes Alliance for Minority Participant Program, NSF 05-XXX (Arlington, VA, 2005).
THE URBAN INSTITUTE
The Urban Institute is a nonprofit, nonpartisan policy research and educational organization established in Washington, D.C., in 1968. Its staff investigates the social, economic, and governance problems confronting the nation and evaluates the public and private means to alleviate them. The Institute disseminates its research findings through publications, its web site, the media, seminars, and forums. Through work that ranges from broad conceptual studies to administrative and technical assistance, Institute researchers contribute to the stock of knowledge available to guide decisionmaking in the public interest. Conclusions or opinions expressed in this report are those of the authors and do not necessarily reflect the views of officers or trustees of the Institute, advisory groups, or any organizations that provide financial support to the Institute.
Program for Evaluation and Equity Research (PEER)
The Urban Institute’s Program for Evaluation and Equity Research (PEER) focuses on education research. PEER staff conduct studies in the fields of educational attainment, educational access, minorities in mathematics and science, teacher education, teacher recruitment and retention, and educational assessments. Much of PEER’s work has centered around evaluation studies—many of these large, multisite, multi-method evaluations of programs to increase educational access and success among underrepresented groups, as well as programs to increase the teaching pool. Support for PEER comes from multiple sources, including the National Science Foundation and the American Association for the Advancement of Science, as well as private foundations, such as the Ford Foundation, DeWitt Wallace Reader’s Digest Fund, Lumina Foundation, and GE Foundation.
Acknowledgments
We would like to acknowledge the invaluable cooperation and assistance that we received from Louis Stokes Alliances for Minority Participation (LSAMP) program directors, staff, faculty, and student participants (both past and present), which enabled us to collect a wealth of data on which this evaluation rests. Their willingness to share their experiences and insights contributed greatly to our understanding of how individual projects function, and the commonalities and differences that exist within the overall Program. We are particularly grateful to LSAMP staff members and participants at the case study institutions for effectively coordinating and graciously hosting the case study site visits, which proved to be informative as well as enjoyable.
We are also grateful to the staff at the National Science Foundation (NSF) for their support and guidance. In particular, we are indebted to A. James Hicks, the program officer for LSAMP, for his strong leadership and responsiveness to our requests for assistance throughout the project. We are also very appreciative of Elmima Johnson, our program officer, whose encouragement and guidance facilitated our work. We benefited tremendously from the invaluable assistance and feedback received from Bernice Anderson, acting director of human resource development. John Tsapogas in Social Behavioral and Economic Sciences/Science Resource Statistics was extremely helpful in answering questions about the National Survey of Recent College Graduates (NSRCG) and facilitating our access to the NSRCG data.
Our special thanks to members of the project’s advisory panel: Gerald E. Gipp, Carlos Rodriguez, William Sibley, and Floraline I. Stevens. The insight and suggestions they offered early in the evaluation were helpful in refining its design and contributed substantially to the evaluation’s direction.
We wish to acknowledge the invaluable assistance of NuStats, our subcontractor on this project. In particular, we are deeply grateful for the tremendous contributions that Rob Santos, vice president of NuStats, made to our study. Early on and throughout the project, his expertise and advice strengthened our work.
Finally, we wish to thank our colleagues at the Urban Institute: LaTasha Holloway, who provided assistance throughout the long process of data collection, analysis, and report writing and Gary Gates, who advised us on the use of STATA during the analysis phase.
The Evaluation Team
THE URBAN INSTITUTE Program for Evaluation and Equity Research (PEER)
Contents
List of Tables and Figures viii
Executive Summary 1 Section I: Introduction Statement of the Problem 4 Brief Description and Overview of LSAMP 5
Section II: Evaluation Design and Methodology Evaluation Questions 7 Evaluation Design and Methodology 8
Section III: Process Evaluation Findings Telephone Interviews 19 Critical Issues Emerging from the Case Studies 28
Section IV: The LSAMP Model Research and Theory Underlying Approaches and Strategies Academic and Social Integration: The Tinto Model of Student Retention 33 Socialization into Science: The Professionalization of Scientists 34 The LSAMP Model 35 Section V: Summative Evaluation Impact on Student Participants 40 Impact on the STEM Workforce 48 Impact on the Knowledge Base: The LSAMP Model 49 Impact on Participating IHEs 52 Section VI: Conclusions and Recommendations Conclusions 54 Recommendations 55
References 57
Appendix A: Literature Review on Effective Strategies to Increase Diversity in STEM Fields A-1 Appendix B: Telephone Interview Protocols B-1 Appendix C: LSAMP Graduate Survey Materials C-1 Appendix D: Site Visit Interview and Focus Group Protocols D-1 Appendix E: Case Studies E-1 Appendix F: Summative Evaluation Tables F-1
THE URBAN INSTITUTE vii Program for Evaluation and Equity Research (PEER)
List of Tables and Figures
Table 1: Number of Interviews and Focus Groups Conducted at Each Alliance 12 Table 2: Respondents to the LSAMP Retrospective Survey by Mode Employed 15 Table 3: Composition of the National Comparative Sample 17
Figure 1: Timing of LSAMP Site Visits 11 Figure 2: Number of Alliances Created, by Year 19 Figure 3: Types of Advisory Committees 20 Figure 4: Precollege Development 23 Figure 5: Student Academic Development Activities 24 Figure 6: Student Professional Development Activities 24 Figure 7: Types of Mentoring Programs 25 Figure 8: Faculty Development 26 Figure 9: Curriculum Development Activities 26 Figure 10: Graduate Studies Development 27 Figure 11: Community College Components 27 Figure 12: Top 5 Components 28 Figure 13: The LSAMP Model 36 Figure 14: Strategies and Approaches: Elements of the LSAMP Model 37 Figure 15: Undergraduate GPA: LSAMP Participants 41 Figure 16: Degrees Sought: Post-Bachelor’s Coursework 42 Figure 17: Field of Study Pursued at Graduate Level: LSAMP Participants 43 Figure 18: Degrees Completed at Time of Survey 44 Figure 19: Graduate Coursework, Degrees Pursued, and Degrees Completed 47
THE URBAN INSTITUTE viii Program for Evaluation and Equity Research (PEER)
Executive Summary
The Louis Stokes Alliances for Minority Participation (LSAMP) Program was established in 1991 by the National Science Foundation (NSF) to develop strategies to increase the quality and quantity of minority students who successfully complete baccalaureate degrees in science, technology, engineering, and mathematics (STEM) and who continue on to graduate studies in these fields. The Urban Institute was commissioned to conduct an evaluation of the LSAMP Program, an evaluation that would answer questions about the structure and implementation of LSAMP and its impact on students, participating institutions of higher education (IHEs), and the diversity of the STEM workforce. The information presented in this report comes from the Urban Institute’s multiyear evaluation of the LSAMP program.
The LSAMP Program began with grants to six multi-institution Alliances across the country. Today there are 34 Alliances with over 450 participating institutions that have produced thousands of STEM degrees. Distinguishing it from traditional scholarship programs, LSAMP takes a multidisciplinary approach to student development and retention, creating partnerships among colleges, universities, national research laboratories, business and industry, and other federal agencies in order to accomplish its goals. Hands-on research experiences and interactions with mentors to build minority student interest in STEM are LSAMP’s other key characteristics.
The Urban Institute evaluation of this program included both process and summative components, seeking to understand both the implementation of the program and its success in meeting stated goals. The process component of the evaluation utilized qualitative methods to identify aspects of the LSAMP projects that promoted or inhibited the achievement of program goals. Data collection methods included a review of LSAMP project documents, telephone interviews with project staff of all Alliances, case studies involving 18 institutions in three Alliances, and a literature review of research on effective strategies to increase diversity in STEM. The analysis of this information indicates that, at the institutional level, a supportive environment that includes adequate provision of resources and support of faculty and top administrators facilitated the achievement of program goals; at the Alliance level, collaborative activities among partner institutions that result in the leveraging and sharing of both tangible and intangible resources were similarly important. Lack of financial resources and adverse national, state, or institutional political climates were the most common challenges to program success. The process evaluation also revealed that, despite expected variation in practices among Alliances, a recognizable LSAMP model does emerge. That model can be understood as a merging of two prominent streams of research and theory: a model of student retention (the Tinto model) that emphasizes integration of students into the academic institution and the notion of “disciplinary socialization,” which is the process through which students become socialized into science as a profession.
In order to answer questions about the program’s impact on participating institutions and to examine education and career outcomes for participating students, the summative component of the evaluation utilized a combination of quantitative and qualitative methods. Institutional impacts were measured using interviews with program staff and Alliance site visits, while student outcomes were explored through a retrospective survey of funded LSAMP participants who graduated from the program between 1992 and 1997.
Institutional Outcomes. Project staff members who were interviewed at participating IHEs believe that involvement in the program enables institutions to retain and graduate more STEM students by substantially expanding their capability to develop and support STEM student talent. Staff also believe that LSAMP had an impact on participating institutions by changing the institutional culture, policies, and
THE URBAN INSTITUTE 1 Program for Evaluation and Equity Research (PEER)
practices to encourage the recruitment, retention, and graduation of underrepresented minorities (URMs)1 in STEM majors.
Student Outcomes. Analysis of survey data revealed that the vast majority of program graduates sought additional education after their bachelor’s degrees, and two-thirds of participants later enrolled in graduate school—working toward a master’s, PhD, or professional degree. One in four LSAMP graduates had completed a STEM graduate degree at the time of the survey. Finally, the majority of LSAMP graduates reported that the program had been helpful as they sought their bachelors’ degrees in STEM and had influenced their decisions to attend graduate school. More than 90 percent reported that they either had recommended or would recommend LSAMP to others.
National Comparison. In order to examine the difference between these outcomes and those of STEM graduates nationally, LSAMP graduates’ progress in the STEM pipeline was compared to that of nationally representative samples of underrepresented minorities and white and Asian students (using longitudinal data from NSF’s National Survey of Recent College Graduates). This analysis revealed that LSAMP participants pursued post-bachelor’s coursework, enrolled in graduate programs, and completed advanced degrees at greater rates than national comparison groups. The difference in graduate school enrollment and completion is largely due to the significantly higher percentage of LSAMP students pursuing and completing degrees in STEM fields. In terms of the final phase in the STEM pipeline, LSAMP participants were observed joining the STEM workforce in similar proportions to national samples.
Summary of Conclusions and Recommendations
The information learned about the LSAMP program through the process and summative evaluations resulted in three main conclusions and five key recommendations.
Conclusions
1. LSAMP met its stated goal of increasing the quality and quantity of students who successfully complete LSAMP-supported STEM baccalaureate programs. As the program expanded, the share of national URM undergraduate STEM degrees earned by LSAMP participants increased, coinciding with an increase in the national production of URM bachelor’s degree recipients in STEM. On measures of undergraduate academic performance, LSAMP students were found to outperform national comparison samples. 2. The LSAMP program exceeded its stated goal of increasing the number of students matriculating into programs of graduate study in STEM. The LSAMP program produced underrepresented minority students who enroll in and attain graduate degrees in STEM at a higher percentage rate than that of a national sample of underrepresented minority students, and a national sample of white and Asian STEM baccalaureate degree recipients. 3. LSAMP’s strategies and approaches constitute a discrete, identifiable program model, grounded in research and theory, that can be tested and replicated. The identification and description of this successful model signifies a critical advance in the knowledge base of intervention program models.
1 The term “underrepresented minorities” is used to describe racial/ethnic groups who are not represented in the pool of scientific and engineering professionals commensurate with their representation in the general U.S. population, namely, African Americans, Hispanic Americans, and American Indians.
THE URBAN INSTITUTE 2 Program for Evaluation and Equity Research (PEER)
Recommendations
1. Increase data collection efforts. Areas of attention should include undergraduate retention/attrition information and up-to-date tracking and contact information for program graduates. Such information would allow for continued analyses of the program’s impact. 2. Strengthen the focus on community college students. Community colleges enroll over half of all underrepresented minority students in postsecondary education and thus provide a promising source of potential STEM students. In light of the program’s success in retaining URM students who begin their degrees in community colleges, increased attention to this component is recommended. 3. Expand the program to offer graduate school tuition and support to LSAMP graduates. LSAMP graduates who did not continue taking courses after attaining a bachelor’s degree were significantly more likely to cite financial reasons for not doing so than were URMs or white and Asian students in the comparison samples. Given LSAMP’s success in preparing students to enter and complete graduate degrees, extending the program’s offerings to include financial incentives to encourage these students to enter graduate STEM programs seems a worthwhile investment.2- 4. Emphasize successful factors in selecting sites to receive LSAMP awards. In awarding LSAMP grants, the program should continue to consider three criteria: (1) evidence of institutional and faculty support; (2) history of, or plans for, a strong collaborative relationship among partners; and (3) well-defined plan and the capacity to provide the integrated services that comprise the LSAMP model. 5. Replicate and expand the LSAMP program. The LSAMP model, unlike most intervention efforts for increasing URM participation in STEM, encourages and supports the synergistic efforts of institutional partners, laying the foundation for systemic institutional change. Given LSAMP’s demonstrated success, it is important that efforts to replicate and disseminate the model be increased.
2 NSF recently initiated a program, Bridge to the Doctorate, to provide graduate school tuition and support to LSAMP graduates.
THE URBAN INSTITUTE 3 Program for Evaluation and Equity Research (PEER)
Section I. Introduction
Statement of the Problem
The U.S. science, technology, engineering, and mathematics (STEM) workforce continues to face the challenge of increasing the participation of women and minorities. While strides have been made to address this shortage, women and underrepresented minorities are still not represented in the U.S. STEM workforce in parity with their percentages in the total workforce population. Recently, a confluence of trends has focused the spotlight on the nation’s need to develop the talent of underrepresented groups in STEM. These include a surge in the college-age population comprised of minorities, declines in science and engineering (S&E) graduate degrees to white students, declining enrollment of foreign students in S&E graduate programs, expectation of a high retirement rate in the S&E workforce, and rapid job growth in the S&E employment sector. These trends have led the National Science Board (NSB) to conclude that the “number of native-born S&E graduates entering the workforce is likely to decline unless the Nation intervenes to improve success in educating S&E students from all demographic groups, especially those that have been underrepresented in S&E careers” (NSB, 2003, p. 1, emphasis added).
In 1991, the National Science Foundation (NSF)—responding to its charge from the Congress to “undertake or support a comprehensive science and engineering education program to increase the participation of minorities in science and engineering” (42 U.S.C. 1885b)—designed the Louis Stokes Alliances for Minority Participation (LSAMP) program. The main goal of LSAMP was to encourage and facilitate access to careers in STEM fields for underrepresented populations. The LSAMP program’s approach to fulfilling its goal addresses several of the often intractable barriers that inhibit minorities from pursuing careers in science and engineering.
Students from underrepresented minority groups (URMs)3 face obstacles at different points in the STEM pipeline that make it difficult for them to attain postsecondary degrees in STEM. First, many students fail to enter higher education prepared to pursue STEM degrees due to inadequacies in K–12 training. Improvements in math and science education including stronger curricula, well-trained teachers, and availability of technological equipment are needed to prepare students adequately for college. Programs that provide academic enrichment and expose students to science experimentation enable precollege students to maintain interest in STEM.
A second obstacle to diversity in STEM is attrition from the STEM pipeline at the college and university level. Underrepresented minorities are less likely than whites and Asians to complete baccalaureate degrees in a STEM major. Reasons that have been cited for this underrepresentation are lack of role models, greater interest in majors unrelated to science and math, poor quality of STEM teaching, inflexible curricula, lack of adequate academic guidance or advice, and low faculty expectations. Those minority students who are successful in obtaining their baccalaureate science degrees often lack the laboratory research skills that make them competitive for graduate school admission. The juncture between undergraduate and graduate school is another point that students leave the STEM education pipeline.
Finally, the high cost of tuition also serves as a barrier for minority students and affects their access to, and retention in, higher education. Tuition costs are one of the factors that have led many students to begin their college education at two-year institutions. There is a strong movement and need for four-year colleges and universities to become more diligent in developing articulation agreements with community colleges. And while community colleges enroll close to half of all students from groups traditionally underrepresented in STEM disciplines, only 26 percent of all students at two-year colleges transfer to
3 Includes African Americans, Hispanic Americans, and American Indians.
THE URBAN INSTITUTE 4 Program for Evaluation and Equity Research (PEER)
four-year institutions. The LSAMP program is designed as an intervention to help minority students overcome many of the problems and barriers they face as they transition into college and progress toward graduation.
Brief Description and Overview of LSAMP
The Louis Stokes Alliances for Minority Participation (LSAMP) program was established in 1991. It was designed to develop comprehensive strategies intended to increase the quality and quantity of minority students who successfully complete baccalaureate degrees in science, technology, engineering, and mathematics (STEM) and who continue on to graduate studies in these fields. Originally named the Alliance for Minority Participation (AMP), the program was renamed in 1999 in honor of former Congressman Louis Stokes. Congressman Stokes served in the U.S. House of Representatives for 30 years and was a leader and pioneer of congressional efforts to improve minority health and the education of minority health professionals, scientists, and engineers. Despite the name change, the program’s goals and design remained the same. LSAMP’s continued goal is to make a significant contribution to the attainment of a diverse, internationally competitive, and globally engaged workforce of scientists, engineers, and well-prepared citizens. LSAMP strives to nurture students’ desire to pursue research in STEM fields and to facilitate NSF’s long-term goal of increasing the production of doctorates in STEM fields. In accordance with that goal, LSAMP has encouraged awardees to develop activities and services that persuade minority students to persist through to graduate school. As it aims to make a significant positive impact on STEM fields, LSAMP’s success will be measured by the program’s ability to bring about a significant increase in the number of underrepresented minorities graduating with baccalaureate STEM degrees and persisting through to graduate study.
The LSAMP program is managed by NSF’s Directorate for Education and Human Resources’ Division of Human Resource Development (HRD). This office administers LSAMP awards at the national level, as it does with other programs with similar goals and objectives. LSAMP awards are distributed in five-year phases. The level of funding provided under LSAMP depends upon the scope of the proposal. The NSF contribution to a project usually ranges between $300,000 and $1 million per year. Projects are selected for the award based on the intellectual merit of their proposed activities and the excellence of their proposal’s plan to broaden the participation of underrepresented minority groups.
LSAMP encourages its awardees to create Alliances that forge partnerships among academic communities (both two-year and four-year institutions) and encourages the inclusion of government agencies and laboratories, businesses and industries, and professional organizations. NSF recognizes that community colleges provide a large pool of minority students that could enter the STEM pipeline. LSAMP awardees are encouraged to promote interaction and collaboration between community colleges and four-year institutions through shared student activities like research experiences and scientific conferences. In this way, Alliances can have an impact on a wide range of students and can share resources in order to fully support students through the pipeline. LSAMP awardees select strategies and approaches that are tailored to their institutional setting and are likely to result in the achievement of program goals. In all cases, potential awardees must describe their plans to create Alliances that work together through joint planning and resource commitment. A project’s proposed activities also must show a reasonable and comprehensive effort that focuses on improving the undergraduate educational experience.
LSAMP project leaders use several mechanisms to communicate with national leadership at NSF. Project directors and other staff members have often been invited to the NSF headquarters in Virginia for what are called “reverse site visits.” These visits give project leaders an opportunity to talk to NSF about their project’s progress. LSAMP projects also have very strict data collection requirements, which enable NSF to monitor the program’s growth. Data are collected annually on student demographics, academic
THE URBAN INSTITUTE 5 Program for Evaluation and Equity Research (PEER)
progress, faculty involvement, and activities by the projects themselves. Institutional-level data are then assembled by the Alliance’s central office staff and used to produce an Alliance-wide annual report. These reports generally showcase the Alliance’s yearly and cumulative progress, budget activity, nonacademic partnerships, as well as student-faculty publications and other significant accomplishments. Much of the student demographic data are reported through the MARS (Monitoring and Reporting System) database, an electronic project reporting system developed by NSF.
The LSAMP program began with grants to six Alliances producing fewer than 4,000 underrepresented minorities with baccalaureate degrees in STEM fields. Today there are 34 Alliances with over 450 participating institutions that have produced thousands of STEM degrees. The program has also influenced an increase in minority enrollment in STEM majors from 35,670 in 1991 to over 205,000 in 2003. LSAMP attributes much of its success to the Alliance structure within which its awardees work. Alliance structures exist in different forms: citywide (e.g., New York City), statewide (e.g., California, North Carolina), and multistate (e.g., Florida-Georgia). Most Alliances have a lead institution where the central office is located and where, generally, the project director is employed.
LSAMP Alliances have several opportunities to exchange ideas and best practices amongst themselves. The NSF/EHR Division of Human Resource Development (HRD) sponsors a joint annual meeting where project leaders (including LSAMP project directors) and staff come together and discuss issues facing the STEM student community. This conference offers opportunities to share ideas and gain important information about recruiting, retaining, and producing professionals in STEM fields. Another means for support and collaboration is the national LSAMP listserv. This form of electronic communication allows those who coordinate LSAMP projects across the country to exchange information about opportunities to be distributed to students. The listserv, moderated by the LSAMP senior program director, is often used as an instrument to advertise summer research internships, fellowships, and scholarships. Some LSAMP projects sponsor their own summer research opportunities for LSAMP students, generally in conjunction with their university or another similar science and math intervention program. These summer experiences are sometimes opened up to LSAMP students nationwide in another attempt to promote collaboration across Alliances.
LSAMP awardees implement a variety of activities and services in order to accomplish the goals of the program. These activities focus on strengthening academic skills through student support, academic enrichment, and research skill development. Participants receive a stipend for engaging in LSAMP- sponsored activities. LSAMP projects coordinate a wide range of activities and services based on the needs of their student populations and in accordance with the focus of their award phase.
LSAMP has several components that distinguish it from traditional scholarship programs. Instead of focusing on aiding individual institutions or students through financial support, LSAMP takes a multidisciplinary and comprehensive approach through the creation of Alliances that generate productive partnerships among colleges, universities, national research laboratories, the business and industry community, and federal agencies that help accomplish its goals. In addition, its major emphasis on offering activities designed to sustain minority student interest in STEM fields and graduate study through hands-on research experiences and interactions sets it apart from national programs with similar goals.
This final report of the findings of our evaluation of the LSAMP program consists of six sections, including this introduction. Section II describes the evaluation design as well as the methodology used to conduct the evaluation. Section III discusses the findings of the process component of the evaluation. Section IV describes the components that make up the LSAMP model and explores the model’s links to the existing theoretical and research literature. Section V presents the findings of the summative portion of the evaluation. The report ends with the study conclusions and recommendations, found in Section VI.
THE URBAN INSTITUTE 6 Program for Evaluation and Equity Research (PEER)
Section II. Evaluation Design and Methodology
In 2000, the National Science Foundation contracted with the Urban Institute to conduct an evaluation of the Louis Stokes Alliances for Minority Participant program. This section presents the questions, design, and methodology used in carrying out the evaluation.
Evaluation Questions
Our evaluation, which contained both process and summative components, used a mixed-methods approach to answer the main evaluation questions.
Process Evaluation
The process evaluation concerned the implementation of LSAMP and focused on the following questions:
• How are LSAMP programs being implemented?
• What components/strategies have accelerated the attainment of program goals?
• What factors have inhibited the attainment of program goals?
• Is there a recognizable LSAMP model?
This component of the evaluation used qualitative methods to identify crucial components of LSAMP as well as factors that seemed to promote or inhibit the achievement of program goals. An important aspect of this component was to assess whether or not—in spite of expected variations in practices—the LSAMP Alliances were operating according to a recognizable model, and whether that model could be traced to general theories of student retention and persistence in science that could inform future efforts to achieve and sustain diversity in the S&E workforce.
Summative Evaluation
The summative component of the evaluation sought to document the impact of LSAMP in a number of areas as shown by the following questions:
What has been the impact of LSAMP on
• participants?
• the diversity of the STEM workforce?
• the knowledge base of promoting diversity in STEM?
• participating IHEs?
This component used a combination of quantitative and qualitative methods to answer the relevant questions. A major feature of the summative component was a retrospective survey of all LSAMP participants who graduated with an S&E major from the program’s inception in 1992 to 1997 to determine their progress in the S&E pipeline. These participants were then compared to matched counterparts on a national level to determine the impact of having participated in LSAMP.
THE URBAN INSTITUTE 7 Program for Evaluation and Equity Research (PEER)
Evaluation Design and Methodology
The evaluation framework combines both process and summative evaluations, documenting the mix of strategies used by the Alliances in order to develop models of the processes that yield LSAMP outcomes. The methods undertaken for both of these evaluation components are discussed in this section.
1. Process Evaluation
The process component of the evaluation relied primarily on qualitative data. Methods included a thorough review of LSAMP project documents, telephone interviews with project staff members of all Alliances, case study site visits to a sample of three Alliances, and a literature review of research on effective strategies to increase diversity in STEM.
Project Document Review In order to become familiar with project structures and strategies employed by the Alliances, the evaluation team undertook a project document review. This process involved the retrieval and review of relevant project documents from NSF files. Evaluation staff from the Urban Institute were granted access to office space and photocopying equipment and spent approximately one week at the National Science Foundation reviewing LSAMP files and making photocopies of the most relevant documents. Documents considered “relevant” included proposals, cooperative agreements, and annual reports4 as well as descriptive reports, “best practices”-type publications, and any program reviews or evaluations that may have been prepared by internal project staff or outside contractors. Once the retrieval process was complete, documents were organized and filed by Alliance.
Evaluation team members reviewed all retrieved LSAMP documents and prepared short project abstracts for each of the 27 Alliances based on the information contained in the project files. Project documents were closely studied for their descriptions of project activities and a matrix identifying common and uncommon program components was generated. The background information gleaned from project documents further facilitated refinement of the evaluation design, and assisted in the development of appropriate data collection instruments to be utilized during telephone interviews and case study site visits.
Telephone Interviews Telephone interviews were conducted with LSAMP project staff to gather formative data on project goals, history, and function, as well as on major project activities. These interviews were conducted with staff from all 27 Alliances.
Development of the Interview Protocols. Protocols for the telephone interviews were developed based on the evaluation design and information taken from the LSAMP project documents. The protocol was field tested in a telephone environment by UI staff in early October 2001. UI staff debriefed interviewees for their feedback following the field test and took notes regarding respondents’ comments and criticisms. These debriefing sessions revealed that while project directors were usually most able to respond to questions pertaining to the overall project, others such as a project manager or coordinator were more suitable in answering questions about the daily operation of the project and project components. Consequently, the initial telephone interview protocol was split into two parts. Part I contained questions about the daily operations and was generally directed to the project director, while Part II focused on more general questions about the project and its components and was generally directed to the project
4 Updated annual reports for each of the Alliances were requested from NSF periodically over the course of the project.
THE URBAN INSTITUTE 8 Program for Evaluation and Equity Research (PEER)
manager or coordinator (refer to appendix B for copies of these instruments). These protocols were submitted for OMB clearance and approval was obtained.
Interviews. In preparation for the interviews, senior staff conducted a telephone interview training session for junior members of the evaluation team. Subsequently, evaluation team members reviewed assigned project files as well as program activity reports generated for each Alliance from the LSAMP project database (MARS),5 and customized individual interview protocols as appropriate. An introductory script was used in conjunction with interview protocols. NSF provided the evaluation team with contact information for key project staff who were reached through email or telephone to schedule the Part I interview. Telephone Interview Part I covered the following topics: