Scientizing Science Policy: Implications for Science, Technology, and Innovation Policy and R&D Evaluation Gouk Tae Kim Dissertation submitted to the faculty of the Virginia Polytechnic Institute and State University in partial fulfillment of the requirements for the degree of Doctor of Philosophy In Science and Technology Studies Saul Halfon, Chair Daniel Breslau Stephen D. Nelson Aaron D. Schroeder July 12, 2012 Blacksburg, Virginia, USA Keywords: science of science policy, government performance management reform, social contract of science, science and technology studies, R&D evaluation Copyright 2012, Gouk Tae Kim Scientizing Science Policy: Implications for Science, Technology, and Innovation Policy and R&D Evaluation Gouk Tae Kim ABSTRACT In this dissertation research, I try to deepen the understanding of the logic and history behind the science of science policy approaches and to substitute for this scientific evidence-based science policy model an evidence-critical and -informed model in which scientific and democratic claims are promoted simultaneously. The science of science policy, or what I call the scientizing science policy (SSP) discourse, is a strategic response of science policy community members to the following two socio-political developments: the government performance management reform movement and a new social contract of science. These two developments have motivated the science policy community to construct new science R&D management strategies that make science R&D investment more effective and economically beneficial than before. Former Presidential Science Advisor John Marburger played an important role in articulating an SSP approach at the federal level that opened up a political space for the larger SSP discourse to emerge and take hold. Other heterogeneous science policy community actors, including science agency managers and academic researchers, have also engaged and played major roles in shaping the premises, strategies, and directions that make up the SSP discourse by articulating their own approaches to SSP. The SSP discourse constitutes a series of strategies such as economizing and quantifying R&D investment decisions. In particular, to implement the ideas of performance reform and a new social contract of science in the field of science policy and management, the SSP community members have prioritized the development of data, models, and evidence related to federal R&D investment by funding studies on new scientific data, tools, and quantitative methods through the National Science Foundation (NSF) Science of Science and Innovation Policy (SciSIP) program. Interagency collaboration organized and supported by the Office of Science and Technology Policy (OSTP) is another key feature promoted by the SSP community. Through this research of the rise and development of the SSP discourse, I emphasize the following aspects that are relevant to both science policy practice and research community members. First, the SSP discourse demonstrates the influence of the performance reform movement on science, technology, and innovation policy and R&D management. Second, the SSP discourse has the strong potential to shift science policy makers’ focus from planning and implementing to evaluating federal R&D programs. Third, the SSP discourse not only reflects, but also promotes the tendency of public policy makers, politicians, and the public to rely on scientific claims and evidence when they are engaged in discussions or policy decision making processes related to science and technology. Fourth, the SSP discourse alters the balance of authority and influence among science policy actors, including science agency managers, scientists, and executive branch offices in the decision making process on federal R&D priority and investment. Fifth, even though there are conflicts and disagreements among science policy community members on the visions and future of the NSF SciSIP program, the SSP discourse is valuable as a space in which heterogeneous science policy research and practice community members can interact, learn from each other, and collaborate to develop U.S. science, technology, and innovation policy. I conclude by proposing an evidence-critical and -informed science policy in which the SSP discourse contributes to promoting democratic values in the science policy decision process. In particular, the evidence-critical and -informed model focuses on not only using scientific data and evidence when making federal R&D decisions, but also on promoting the democratic and deliberative process in monitoring R&D activities’ performance and social outcomes. In this model, I view the public as a legitimate stakeholder for evaluating federal R&D investment. This evidence-informed model can be implemented under the SSP discourse if the new R&D data, models, and tools developed by the NSF SciSIP-funded research are coupled with a new government performance website in which the public can access information about federal R&D activities as well as provide feedback about R&D investments to science policy makers. iii ACKNOWLEDGEMENTS I want to express my deepest appreciation to my committee chair, Dr. Saul Halfon, and my three committee members, including Dr. Daniel Breslau, Dr. Stephen Nelson, and Dr. Aaron Schroeder, for their academic advice in producing my dissertation. I thank Harvard Kennedy School STS fellow program director Dr. Sheila Jasanoff for offering me such a wonderful academic journey in her program to deepen my understanding of STS. I also appreciate for the NSF STS Program for providing a research grant to me for improving my dissertation research. And I would like to special thank to my wife, Eun-Hee Kim, my son, Isaac Jae- Hyun Kim, and my parents for their great support of my study. iv TABLE OF CONTENTS Abstract................................................................................................................... ii Acknowledgments.................................................................................................. iv Chapter 1: Introduction………………………....................................................... 1 1. Science of Science Policy Initiative 4 2. Main Research Questions 7 3. Analytical Frameworks to Investigate the SSP Discourse 10 (1) New Social Contract of Science (Chapter 3) 10 (2) Performance Management Reform Movement (Chapter 4-I) 11 (3) Performance Management Reform in Science Policy (Chapter 4- II) 12 (4) International Dynamics of the SSP (Chapter 5) 13 (5) Politics of Science & SSP (Chapter 6) 13 (6) Comparative Analysis (Chapter 7) 16 4. Qualitative Research Methods and Interviews 17 (1) Qualitative Research Methods 18 (2) Design and Goals of Research Interviews 20 5. Benefits and Significance of the Research 23 Chapter 2: Case Analysis: NSF’s SciSIP and OSTP’s SoSP…………………….. 26 1. Science Politics & the Empirical Evidence Based Model 26 (1) Marburger’s Two Keynote Speeches on Scientizing Science Policy 29 (2) OSTP, Interagency Task Group, and the Science of Science Policy Roadmap 32 (3) NSF’s Science of Science and Innovation Policy (SciSIP) 37 (4) Congressional Hearing on SciSIP 47 2. The SSP and the New Era of Evaluation of Federal R&D investments 49 (1) Needs for a Developing New Science Policy Evaluation Approach 50 (2) Evaluation of R&D Activities and the SSP 52 (3) SciSIP Funded Research and the Possible Impacts of the SSP on R&D Evaluation 54 (4) Emphasis of Extrinsic Criteria & Large-Scale Data 57 (5) New Ways of Evaluating STI Policies and R&D Programs 58 Chapter 3: Reshaping the Social Contract of Science............................................ 60 1. Definition(s) of Science Policy, Technology Policy and Innovation Policy 60 2. History of the U.S. Science Policy 61 (1) Before World War I 62 (2) Wartime and Post-War Science Policy 66 v (3) Three Phases of Science Policy Development until the 1980s 69 (4) Developing Triple-Helix System 74 3. The Emerging New Social Contract of Science and Science Policy Culture 79 (1) The New Social Contract of Science 79 (2) The Emerging New Science Policy Culture 85 (3) Beyond Science, the Endless Frontier 88 (4) SciSIP and SoSP: New Science Policy Tools 92 Chapter 4: The Government Performance Management Reform Movement in Science Policy........................................................................................... 95 1. SSP, the Arrival of the Government Performance Reform in Science and Science Policy 95 2. Government Performance Management, GPRA and PART 97 (1) GPRA of 1993 99 (2) The President’s Management Agenda and the Program Assessment Rating Tool 103 (3) The PMA and R&D Performance Reform 106 (4) The GPRA Modernization Act – Obama Administration 109 (5) Annual Performance Plan (APP) 112 3. New Relationship Among Scientists, Science Agencies, OMB and White House 114 (1) Performance Reform in Science Policy 117 (2) Balancing Influence and Authority in Governing Science through the SSP 124 (3) Performance Reform in Science Policy & SSP 128 Chapter 5: Interview Data Analysis: External Logics and Internal Dynamics of the SSP…………………………………………………………………... 136 1. Analysis of the Interview Data on the SSP Discourse 138 (1) Diverse Interpretation of the SSP 140 (2) Impacts of the Performance Management/Budget Reform on the SSP 147 (3) Impacts of the Performance Reform & a Revised Social Contract of Science I: Economizing and Mathematizing Federal R&D Decision Making 155 (4) Impacts of the Performance Reform & a Revised Social Contract of Science II: Improved Authority of Scientific Claims