Science Outside the Laboratory Science Outside the Laboratory Measurement in Field Science and Economics MARCEL BOUMANS Associate Professor History and Methodology of Economics at Faculty of Economics and Business, University of Amsterdam and at Faculty of Philosophy, Erasmus University Rotterdam 3 3 Oxford University Press is a department of the University of Oxford. It furthers the University’s objective of excellence in research, scholarship, and education by publishing worldwide. Oxford New York Auckland Cape Town Dar es Salaam Hong Kong Karachi Kuala Lumpur Madrid Melbourne Mexico City Nairobi New Delhi Shanghai Taipei Toronto With offices in Argentina Austria Brazil Chile Czech Republic France Greece Guatemala Hungary Italy Japan Poland Portugal Singapore South Korea Switzerland Thailand Turkey Ukraine Vietnam Oxford is a registered trade mark of Oxford University Press in the UK and certain other countries. Published in the United States of America by Oxford University Press 198 Madison Avenue, New York, NY 10016 © Oxford University Press 2015 All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, without the prior permission in writing of Oxford University Press, or as expressly permitted by law, by license, or under terms agreed with the appropriate reproduction rights organization. Inquiries concerning reproduction outside the scope of the above should be sent to the Rights Department, Oxford University Press, at the address above. You must not circulate this work in any other form and you must impose this same condition on any acquirer. Library of Congress Cataloging-in-Publication Data Boumans, Marcel. Science Outside the Laboratory : measurement in field science and economics / Marcel Boumans. p. cm. Includes bibliographical references and index. ISBN 978–0–19–938828–8 (alk. paper) 1. Economics—Statistical methods. 2. Social sciences—Statistical methods. 3. Social sciences—Fieldwork. 4. Measurement. I. Title. HB137.B66 2015 530.801—dc23 2014046362 135798642 Printed in the United States of America on acid-free paper To Freeke CONTENTS Preface ix 1. Introduction 1 2. Measurement 26 3. Calculus of Observations 57 4. The Problem of Passive Observation 87 5. Clinical Judgment 116 6. Consensus 150 7. Conclusions 173 Bibliography 179 Index 191 vii PREFACE The first signs were quite alarming: At one of our regular visits at the child health center, one month after the birth of our youngest child, the officiating physician detected cardiac arrhythmia, and so we immediately went to the hospital for fur- ther investigation. After all kinds of tests that confirmed the arrhythmia detected by the child health center, the medical staff at the hospital looked even more wor- ried. To investigate how this disorder would evolve and so determine the most appropriate treatment, we received a small box to be connected with stickers to our son’s body so an electrocardiogram over a longer period could be taken. At the appointment a week later to discuss the analysis of the ECG, the physician looked surprisingly less worried, even relaxed. The ECG had not shown the arrhythmia again. Carefully examining our son, he commented that he looked very vivid and healthy, and so concluded that no treatment at all was needed. Because the cause of the arrhythmia was—and remained—unknown, the physician could also have prescribed medication as a precaution. This would probably have meant a lifelong dependence on medication. We are happy he did not; our son is now taller than me. The experience reveals that it was the combination of the test results and the clinical view of the physician that induced the right diagnosis. This book is about what is needed to attain a reliable measurement, particularly when the data out of which the measurement is composed come from different sources, and not all of the different sources of the data derive from clean and con- trolled environments, which I call “laboratories.” One often says that one cannot compare apples to oranges, but this is, however, what one has to do constantly outside these clean and controlled environments, that is, in the “field.” To give a simple example, any index number makes such comparisons: take the Human Development Index, where economic growth, life expectancy, and education are “added” together to measure human development. It is not only that we often need human judgment as one of these required sources of data; the composition of the data has to be designed, and is based on human judgment, too. Mechanical objectivity presumes human judgment to design ix x preface the mechanism. It is, however, just this human judgment, needed as a source of data and design, that is under attack: evidence is piling up showing how “irrational” and “biased” human judgments are, including those of “experts.” In the political domain, one increasingly gets the idea that people need to be “nudged.” But who are the “nudgers”? Experts with a special permit? Who gives the permit? This book is written to explore a methodology of field science to deal with these different sources of evidence, a methodology that aims not at abandoning human judgment but at including it in a “rational” manner. The exploration of this method- ology is an exercise in the philosophy of science-in-practice. These practices include both current and past research, where the past covers mainly the twentieth century. In studying various exchanges and practices, I found that those of the 1950s were of most interest and relevance for the subject of this book. It may be that the reason for this is that in that period shortly after the Second World War, with the positive experiences of operations research groups fresh in mind, interaction between vari- ous social and natural disciplines was considered to be most promising to achieve progress in the field sciences. It is a cliché among authors to say that their book could not have been written without the discussions and interactions in the past few years with wonderful minds, but this does not mean it is less true. First of all I would like to express my grati- tude to the people of the Center for the History of Political Economy (CHOPE) of Duke University: Bruce Caldwell, Craufurd Goodwin, Kevin Hoover, Neil De Marchi, and Roy Weintraub. By inviting me to join them for the Fall semester in 2012, they provided the intellectual shelter I needed to write the first version of this book. I also would like to thank the visiting fellows of CHOPE during that semes- ter, in particular Verena Halsmayer, with whom I could discuss my first ideas about the overall theme of the book. The manuscript was sent to John Davis, Neil De Marchi, Kevin Hoover, and Mary Morgan for their comments. I am very grateful for their constructive responses, which I used to improve the manuscript. I also would like to thank Giora Hon for his trust and constructive advice, the anony- mous Oxford University Press referees for their encouraging and positive reports, and Scott Parris, the Oxford University Press academic book editor, for his enduring trust and support. This book is based on my work on measurement of the past few years. During this period I benefited enormously from the many intellectual stimulating conver- sations I had with members of the Amsterdam History and Methodology Research Group, consisting of the late Mark Blaug, Dirk Damsma, Federico D’Onofrio, Harro Maas, Tiago Mata, Mary Morgan, Geert Reuten, Eric Schliesser, and Andrej Svorencik. In particular I would like to thank Freeke Mulder for the many conver- sations on measurement we had over a much longer period, adequately captured by her motto “Meten is weten,” that is, “Measuring is knowing.” It is to her I have dedicated this book. preface xi Most of the chapters are derived from articles I have published elsewhere. I thank Duke University Press for permission to use excerpts of my “Observations in a Hostile Environment: Morgenstern on the Accuracy of Economic Observations,” History of Political Economy, Annual Supplement to vol. 44, 110–131, copyright 2012, Duke University Press, in chapter 1; “The Problem of Passive Observation,” History of Political Economy 42 (1), 75–110, copyright 2010, Duke University Press, and “Haavelmo’s Epistemology for an Inexact Science,” History of Political Economy 46 (2), 211–229, copyright 2014, Duke University Press, in chapter 4. I thank Elsevier for permission to use excerpts of my “The Role of Models in Measurement outside the Laboratory,” Measurement 46 (2013) 8, 2908–2912, and “Model-based Type B uncertainty evaluations of measurement towards more objective evalu- ation strategies,” Measurement 46 (2013) 9, 3775–3777, in Chapter 2. I thank Sage for permission to use excerpts of my “The Two-Model Problem in Rational Decision Making,” Rationality and Society 23 (2011) 3, 371–400, in chapter 5. I thank Taylor and Francis for permission to use excerpts of my “The Reliability of an Instrument,” Social Epistemology 18 (2004), 215–246, in chapter 3, and “Battle in the Planning Office: Field Experts versus Normative Statisticians,” Social Epistemology 22 (2008) 4, 389–404, in chapter 5. I thank Aksant for permission to use excerpts of my “Measurement and Error Problems (1800–1900): Buys Ballot and Landré’s Critique on the Method of Least Squares,” in The Statistical Mind in Modern Society: The Netherlands, 1850–1940,vol.2:Statistics and Scientific Work, ed. I. H. Stamhuis, P. M. M. Klep and J. G. S. J. van Maarseveen (2008), pp. 179–197, in chapter 3. I thank Springer for permission to use excerpts of my “Model-Based Consensus,” in Experts and Consensus in Social Science: Critical Perspectives from Economics, Sociology, Politics, and Philosophy, ed. C. Martini and M.