Robert Noyce Program Evaluation Fact Sheet

Robert Noyce Teacher Scholarship Program Evaluation Highlights

University of Minnesota Evaluation Team Frances Lawrenz, PI Anica Bowe Maureen Braam Allison Kirchhoff Pey-Yan Liou

June 2010 2

The following are highlights of the Robert Noyce Teacher Scholarship Program (Noyce program) evaluation conducted by the University of Minnesota (U of MN) Noyce evaluation team from 2005-10. Descriptions of the Noyce program are available from the National Science Foundation (NSF) online at http://www.nsf.gov/funding/pgm_summ.jsp? pims_id=5733&org=DUE&from=home . The program evaluation was extensive and resulted in several detailed databases and reports about different aspects of the evaluation. These can be obtained from our Web site http://cehd.umn.edu/EdPsych/NOYCE/. The evaluation was not longitudinal, although some time-related inferences can be drawn because the array of projects in the Noyce program had been in operation for varying amounts of time.

Survey data were gathered in 2007 from 66 principal investigators (PIs) of the projects (including projects awarded between 2002-6), 90 disciplinary faculty (80 STEM and 10 Education) whom PIs identified as interacting with the Noyce projects, and 555 Noyce program scholarship and fellowship recipients (henceforth referred to as scholars) at varying points in their career paths. In late 2007-8, data were gathered from 19 interviews with school district representatives identified by the PIs and in 2007-9 from 38 scholar interviews. In late 2009-10, interview data was also gathered from 16 STEM content faculty at various institutions where the Noyce program was situated. The instruments for collecting these data were jointly developed by the evaluation team and the Noyce project representatives through a series of in-person and virtual meetings during the first years of the evaluation project. The demographic data on the scholars were obtained from the ORC Macro International, Inc. Noyce program monitoring data (ORC database) provided by the PIs.

Scholar Characteristics

In the ORC database the PIs reported their scholars as having diverse backgrounds and strong content preparation in terms of GPA and majors. Because the scholars who responded to the U of MN survey were a subset of the scholars in the ORC database, scholar characteristics were obtained from it. Although the GPAs for all scholars were not reported, the mean of reported GPAs was 3.5 at the time they received funding. The scholars were reported as majoring in a range of STEM areas: 39% mathematics, 31% biology, 7% chemistry, 6% physics, 5% geology/environmental science, 4% engineering, 2% physical science, 1% computer science, and 18.5% indicated having “other” majors (e.g., bacteriology, chemical engineering, astrophysics) (percents don’t add to 100% because more than one major could be reported for each scholar.) Sixty-five percent of the scholars were female and approximately one third of the scholars were of color. Blacks, Hispanics, and Asians each were approximately one third of the population of scholars of color with very small numbers of American Indian and Native Hawaiian. The proportion of scholars of color was higher than the proportion of teachers of color in the existing STEM teaching force (9-14% depending on subject and grade range1). On the scholar survey of the scholars who were teaching, 59% reported that they were likely to assume a leadership position within the next three years; 22% reported that they were already in leadership positions.

Teacher Preparation/Certification Programs

1 Weiss, I. Banilower, E. McMahon, K. and Smith, P. (2001) 2000 National Survey of Science and Mathematics Education. Horizon Research, Inc. http://2000survey.horizon-research.com/. 3

Based on data from the PI survey, the teacher preparation/certification programs exhibit a good deal of variety both within and across institutions and an array of traditional and alternative certification programs. One PI reported having eight different certification tracks. Most PIs (70%) reported having one or two tracks. Sixty percent of PIs said their teacher preparation/certification programs had students begin full-time teaching after completion of the program while 40% had students start full-time teaching before that. The amount of time to obtain certification varied from 6-9 months (3%) to over 15 months (45%). About 90% of the PIs reported that the Noyce funding had increased their ability to recruit students with STEM content degrees and those committed to working in high need schools. About 80% reported that the Noyce funding had improved their ability to recruit underrepresented students. Forty percent said the Noyce funding improved their relationships with business or industry even though it is not a requirement of the Noyce program.

When asked about how they collaborate with local school districts, 95% of the PIs said the school districts were used to provide practicum sites. Twenty-six percent of the PIs said that the school districts had agreed to hire all of the scholars who successfully completed the teacher preparation/certification program.

The PIs reported a diverse range of opportunities available to the scholars as well as a core of common opportunities directly linked to experiences in, and education about, high need settings. PIs were asked which of five field experiences were required in their programs. The two most commonly required experiences were supervised high need classroom teaching experiences (84%) and experiences in schools with high need students (78%). However, 10% indicated their program required education field experiences outside of school with high need youth. PIs were also asked about which of 11 different activities were required, offered or not available as part of their teacher preparation/certification program. The most commonly required activity was education about different cultures (83%). Many more activities were reported as being offered than as being required.

STEM Faculty Perceptions

The STEM faculty responses showed that the STEM faculty were supportive of the Noyce projects. Eighty-nine percent reported that the Noyce program had a minimal to strong positive effect on their institutions’ preparation/certification programs. In addition, a majority of STEM faculty (65%) reported believing that STEM majors going into education are similar in ability to other STEM majors.

The analysis of the STEM faculty interviews indicated that the STEM faculty participants were involved at three different levels with the Noyce program and these levels were influenced by a variety of contextual elements. Although all participants had positive perceptions of teaching and STEM education, those who were part of an institutional culture which favored STEM education reform outreach, faculty members of small institutions/departments, were Noyce program Co- PIs, or had education programs within STEM departments seemed more heavily impacted by the Noyce program.

School District Perceptions of the Effect of the Noyce Program

The district interviewees said they perceived the effect of the Noyce program through direct interactions with the scholars. The most prominent perception was that of a “recruitment pipeline” in which the districts were able to find higher numbers and better quality math and science teachers. One quarter of interviewees also mentioned that they felt there were more minority and disadvantaged teaching candidates available because of the Noyce program. Another common response was that the Noyce program enhanced the district’s already strong relationship with the teacher preparation/certification program and provided another avenue for collaboration between the school district and the preparation/certification institution.

Issues Affecting Influence of the Noyce Program

This evaluation project was not longitudinal so the influence of the program was assessed by perceived effect rather than an over-time assessment of scholars remaining in high need schools. Through the use of statistical grouping techniques (factor and cluster analyses), the scholars who were teaching were organized into three clusters. Scholars within the clusters were similar in terms of their perceptions of the effect of the funding on becoming teachers and on becoming teachers in high need schools. The three groups were: 1) less committed to becoming a teacher and teaching in a high need school, 2) highly committed to becoming a teacher but not to teaching in a high need school, and 3) highly committed to becoming a teacher and teaching in a high need school. Scholars in the less committed clusters correspondingly reported more influence of the funding. In other words, if scholars were committed to teaching in high need schools, they were less influenced by the funding. Significantly more scholars of color than white scholars were in the more committed to teach and to teach in high need schools cluster (less influence of funding). More scholars with high GPAs were in the cluster of more influence of the Noyce funding. Scholars identifying themselves as career changers were distributed evenly across the three clusters. Not surprisingly, significantly more scholars who had decided to be teachers before hearing about the Noyce program viewed the funding as less influential in their decisions to become teachers. Funding was perceived as important to all scholars for completing their preparation/certification and becoming teachers.

Influence of the Noyce Program on Career Paths

The scholar interviews revealed a complex structure of the scholars’ paths to teaching in high need schools. Choosing teaching as a career is influenced by the scholar’s background in terms of personal background, content preparation, educational role models, and previous careers. The scholars’ backgrounds also interact with accepting the Noyce funding which in turn interacts with the choice of teaching as a career. Once the Noyce funding is accepted and the student is in a teacher preparation/certification program, the most salient influences are various types of support and experiences with high need schools and children. These teacher preparation/certification program characteristics contribute to the now certified teacher choosing where to teach. In addition, this decision is affected by the Noyce funding, the scholars’ backgrounds—such as where they are from—and the chosen school setting. The school setting includes such things as mentoring, student behavior, administration, colleagues, safety and income, with administration and colleagues playing a major role. The school setting in turn

1 Weiss, I. Banilower, E. McMahon, K. and Smith, P. (2001) 2000 National Survey of Science and Mathematics Education. Horizon Research, Inc. http://2000survey.horizon-research.com/. 5 interacts with job satisfaction. Job satisfaction is influenced by the scholars’ content backgrounds which affect the types of classes taught and the opportunity for leadership positions.

Influence of the Noyce Funding on Becoming Teachers and Teaching in High Need Schools

Two-level HLM analyses were performed on two groups of scholars: the “all scholars group” and “current STEM teacher group.” Results from the two-level HLM analyses revealed that most of the variance in the influence of the Noyce funding occurred at the scholar level rather than the program level for both groups. In addition, more variance at the program level occurred for scholars’ perceptions of the influence of the Noyce funding to become teachers than for explaining scholars’ perceptions of the influence of the Noyce funding on teaching in high need schools. Noyce funding at the program level showed a positive relationship with the all scholars’ and STEM teachers’ perceptions of becoming teachers and teaching in high need schools. At the scholar level only a few of the predictors had significant relationships with the outcome variables, and these relationships were not necessarily consistent. For the all scholar group, the results indicated that 1) non-whites tended to have higher perceptions of the effect of the funding on becoming teachers and teaching in high need schools than whites; 2) including curricula and activities for high need schools in Noyce programs played a role in increasing scholars’ perceptions of the effect of Noyce funding on their decision to teach in high need schools; and 3) scholars’ paths to teaching increased the effect of the funding on scholars’ decisions to become teachers. For the STEM teacher group, the results indicated that 1) scholars’ paths to teaching increased the effect of the funding on scholars’ decisions to become teachers; 2) scholars’ perceptions of the effect of the Noyce program on their decision to become teachers had a positive relationship with their district/school high need environment; and 3) the more current STEM teachers’ personal beliefs toward teaching increased, the higher they perceived the effect of Noyce funding on their decisions to teach in high need schools.

Conclusions about the Effect of Scholarship Programs

About programs  Noyce teacher preparation/certification programs are diverse but all include some specific preparation for teaching in high need schools  A majority of STEM faculty believe that STEM majors going into education are similar in ability to other STEM majors  School districts perceive that more high quality and diverse job candidates are being provided through the Noyce program  Noyce scholars perceive that the most beneficial aspects of teacher preparation/certification programs include experiences in high need settings and ongoing support  The degree of STEM faculty investment in the Noyce program depends on being involved as a Co-PI, institutional/departmental size and culture, having STEM education housed with STEM departments and personal interest in STEM education reform

About scholars  Scholars join preparation/certification programs for a wide variety of reasons

 The path to teaching in a high need school is complex and involves a variety of different influences  Becoming a teacher and becoming a teacher in a high need setting appear to be separate constructs  Job satisfaction affects whether or not scholars stay in high need settings. Job satisfaction is influenced by the school’s administration, availability of supportive colleagues and the existence of a stimulating environment including leadership opportunities  The proportion of Noyce scholars of color (33%) is higher than the proportion of teachers of color in the existing STEM teaching force (9-14% depending on subject and grade range)  Although the results about race (whites and non-whites) are mixed, race is an important variable in explaining the effect of Noyce funding on scholars in becoming teachers and teaching in high need schools, especially for non-whites

About funding  Funding is important to all scholars for completing their preparation/certification and becoming teachers  Funding is viewed as most influential by scholars who had not intended to teach until they heard about the scholarship  Funding appears to be more influential in encouraging scholars to teach in high need schools than encouraging them to be teachers  Perceptions of the effect of funding on commitment to teach and to teach in a high need school appear to be different for different groups of scholars. More specifically, race appears to be an important variable, however, its effect is dependent on the presence of other variables  Noyce funding at the program level has a positive influence on scholars’ perceptions of becoming teachers and teaching in high need schools

Recommendations for Further Investigation

Any future investigations of the Noyce program should employ sophisticated designs involving multiple variables in order to build upon the framework of findings provided by this evaluation. Suggestions for future studies include:  Longitudinal studies examining what happens as time from certification increases. This would include questions such as: What proportion of scholars go into teaching? What do they do if they don’t pursue teaching? How long do they stay in teaching? How long do they stay in high need teaching? Why do they leave?  Investigations of the effect of Noyce funding on recruitment into preparation/certification programs through the use of quasi-experimental designs where different types of recruitment strategies are employed and what types of students (e.g., career changers, students of color) the strategies attract are determined  Studies of the effect of Noyce funding compared to other types of funding on students remaining as teachers in high need schools

 Large scale structural equation modeling/path analytic studies to examine the quantitative relationships among the variables on the path to careers in teaching identified through the scholar interviews  Comparative studies of different types of preparation/certification program activities and experiences in terms of which are most related to students remaining in high need settings  Investigations of the nature of support/mentoring (from preparation/certification programs and school districts) during teachers’ early years in high need settings  A descriptive study on the types of non-whites being recruited. Of interest is to know what proportions are scholarship recipients, stipend recipients, as well as career changers. Furthermore, how do these proportions compare to the proportion of whites in the above three categories?

1 Weiss, I. Banilower, E. McMahon, K. and Smith, P. (2001) 2000 National Survey of Science and Mathematics Education. Horizon Research, Inc. http://2000survey.horizon-research.com/.