The National Evaluation of NASA’s Science, Engineering, Mathematics and Aerospace Academy (SEMAA) Program March 31, 2010 Cambridge, MA Bethesda, MD Chicago, IL Durham, NC Prepared for Hadley, MA National Aeronautics and Space Administration (NASA) Office of Education Prepared by Alina Martinez, Abt Associates Inc. Clemencia Cosentino de Cohen, The Urban Institute Abt Associates Inc. W. Carter Smith Kenyon Maree Amanda Parsad Caroline Shlager David Cook The Urban Institute Lisa Tsui Education Development Center, Inc. Abigail Jurist Levy THIS PAGE INTENTIONALLY BLANK Acknowledgements This report is the result of the collaborative effort of numerous individuals. This study was conducted with the assistance of Cristofer Price, Jennifer Carney, and Jan Nicholson of Abt Associates Inc.; Beatriz Chu Clewell, Erin Mudd Gilfenbaum, and Alla Myrvoda of The Urban Institute; Estella Sena, Kevin McMahon and Joe Cella of Abt‐SRBI; and Erica Fields and Barbara Berns of the Education Development Center, Inc. The SEMAA evaluation team would like to thank the students and parents at participating SEMAA sites for their contributions in completing surveys for this study. We gratefully acknowledge the cooperation and assistance received from the directors of former and active SEMAA sites who participated in various aspects of this evaluation. Their thoughtful and detailed responses allowed us to field the impact study, and collect a wealth of information on the experiences of SEMAA programs across the country. Special thanks to the former SEMAA site directors and students and parents, who gave generously of their time to assist us in pilot testing the study protocols. Our work was much facilitated by the support and guidance of the staff at NASA. We would like to thank the SEMAA Management Team and NASA Glenn Research Center (Educational Programs Office) and the National SEMAA Office, Paragon TEC, Inc., who dug into their archives to search for documentation we requested. We are grateful for the guidance and support of staff from the NASA Office of Education, including Brian Yoder, Manager for Program Evaluation, and Patrick Murphy, who served as the program officer for the evaluation, as well as Joyce Winterton, James Stofan, Shelley Canright, and Bernice Alston. Lastly, we also thank the reviewers of this report for the helpful feedback provided. The data collection, analysis and reporting of this material was conducted in accordance with OMB Control No. 2700‐0136 (expiration April 30, 2010). Abt Associates Inc. National Evaluation of SEMAA i THIS PAGE INTENTIONALLY BLANK Contents Executive Summary ....................................................................................................................... v Chapter 1: Introduction ................................................................................................................. 1 The SEMAA Program....................................................................................................................2 A National Evaluation of SEMAA..................................................................................................6 The Issue and Related Factors .....................................................................................................6 The importance of high school course‐taking ...................................................................7 Affective factors influencing STEM persistence: Interest and self‐efficacy ......................8 The role of parents in children’s STEM outcomes...........................................................10 Out‐of‐school STEM enrichment activities......................................................................11 Conceptual Framework..............................................................................................................12 SEMAA’s theory of change ..............................................................................................12 How the SEMAA program theory informed the evaluation............................................17 Chapter 2. Study Design............................................................................................................... 19 Research Questions ...................................................................................................................19 Implementation Study Methodology.........................................................................................19 Impact Study Methodology .......................................................................................................20 Sources of data ................................................................................................................20 Sample for the impact study ...........................................................................................21 Analytic approach for the impact study ..........................................................................23 Chapter 3: Implementation of the SEMAA Program ..................................................................... 25 SEMAA Elements........................................................................................................................25 Project goals ....................................................................................................................25 Project management and governance ............................................................................26 Project components and implementation ......................................................................27 Students served...............................................................................................................30 SEMAA providers.............................................................................................................32 External support, partnerships and collaborations.........................................................33 SEMAA Implementation.............................................................................................................35 SEMAA models and model selection...............................................................................35 Facilitating implementation ............................................................................................38 Monitoring and evaluation..............................................................................................38 Sustainability ...................................................................................................................39 Chapter 4: Characteristics of SEMAA Participants ........................................................................ 41 Background of Participants........................................................................................................41 Baseline Levels of Student Outcomes........................................................................................43 Desire to do science.........................................................................................................44 Participation in informal science‐related activities.........................................................44 Interest in informal science activities..............................................................................44 Abt Associates Inc. National Evaluation of SEMAA iii Change in interest in informal science activities ............................................................ 45 Engagement in school STEM activities............................................................................ 45 Anxiety toward science ................................................................................................... 45 Self‐confidence in science............................................................................................... 45 Interest in high school STEM coursework....................................................................... 46 Interest in college STEM coursework.............................................................................. 46 Interest in STEM careers ................................................................................................. 46 Value of science to society.............................................................................................. 46 Baseline Levels of Parental Outcomes ...................................................................................... 47 Parental support for students’ prior out‐of‐school science activities ............................ 47 Parental support for student’s future out‐of‐school science activities .......................... 48 Parental encouragement of high school STEM coursework........................................... 48 Parental perception of student’s need to understand science/math as adult............... 48 Parental encouragement of STEM fields in college ........................................................ 48 Chapter 5: Outcomes of SEMAA................................................................................................... 51 Overall SEMAA Impacts ............................................................................................................. 51 Impacts for Prior SEMAA Participants and First‐time Participants ........................................... 53 Exploration of Variation in Participation ................................................................................... 53 Chapter 6: Conclusions and Implications .....................................................................................