Validation and Assessment of Energy Models, Held at the Na
Total Page:16
File Type:pdf, Size:1020Kb
NBS PUBLICATIONS NBS SPECIAL PUBLICATION U.S. DEPARTMENT OF COMMERCE / National Bureau of Standards NATIONAL BUREAU OF STANDARDS The National Bureau of Standards' was established by an act of Congress on March 3, 1901. The Bureau's overall goal is to strengthen and advance the Nation's science and technology and facilitate their effective application for public benefit. To this end, the Bureau conducts research and provides: (1) a basis for the Nation's physical measurement system, (2) scientific and technological services for industry and government, (3) a technical basis for equity in trade, and (4) technical services to promote public safety. The Bureau's technical work is per- formed by the National Measurement Laboratory, the National Engineering Laboratory, and the Institute for Computer Sciences and Technology. THE NATIONAL MEASUREMENT LABORATORY provides the national system of physical and chemical and materials measurement; coordinates the system with measurement systems of other nations and furnishes essential services leading to accurate and uniform physical and chemical measurement throughout the Nation's scientific community, industry, and commerce; conducts materials research leading to improved methods of measurement, standards, and data on the properties of materials needed by industry, commerce, educational institutions, and Government; provides advisory and research services to other Government agencies; develops, produces, and distributes Standard Reference Materials; and provides calibration services. The Laboratory consists of the following centers: Absolute Physical Quantities' — Radiation Research — Thermodynamics and Molecular Science — Analytical Chemistry — Materials Science. THE NATIONAL ENGINEERING LABORATORY provides technology and technical ser- vices to the public and private sectors to address national needs and to solve national problems; conducts research in engineering and applied science in support of these efforts; builds and maintains competence in the necessary disciplines required to carry out this research and technical service; develops engineering data and measurement capabilities; provides engineering measurement traceability services; develops test methods and proposes engineering standards and code changes; develops and proposes new engineering practices; and develops and improves mechanisms to transfer results of its research to the ultimate user. The Laboratory consists of the following centers: Applied Mathematics — Electronics and Electrical Engineering- — Mechanical Engineering and Process Technology' — Building Technology — Fire Research — Consumer Product Technology — Field Methods. THE INSTITUTE FOR COMPUTER SCIENCES AND TECHNOLOGY conducts research and provides scientific and technical services to aid Federal agencies in the selection, acquisition, application, and use of computer technology to improve effectiveness and economy in Government operations in accordance with Public Law 89-306 (40 U.S.C. 759), relevant Executive Orders, and other directives; carries out this mission by managing the Federal Information Processing Standards Program, developing Federal ADP standards guidelines, and managing Federal participation in ADP voluntary standardization activities; provides scientific and technological advisory services and assistance to Federal agencies; and provides the technical foundation for computer-related policies of the Federal Government. The Institute consists of the following centers: Programming Science and Technology — Computer Systems Engineering. 'Headquarters and Laboratories at Gaithersburg, MD, unless otherwise noted; mailing address Washington, DC 20234. 'Some divisions within the center are located at Boulder, CO 80303. ; OF STAK»AXOf UBRABT Validation and Assessment OCT 2 3 1981 of Energy Models Proceedings of a Symposium held at the National Bureau of Standards, Gaithersburg, MD May 19-21, 1980 Edited by: DATE DUE Saul I. Gass National Engineering Laboratory National Bureau of Standards m 1 5 1986 Washington, DC 20234 Sponsored by: Energy Information Administration Department of Energy Washington, DC 20461 and Operations Research Division Center for Apphed Mathematics National Engineering Laboratory National Bureau of Standards Washington, DC 20234 CAYLORO PRINTEO IN U.S.A. U.S. DEPARTMENT OF COMMERCE, Malcolm Baldrige, Secretary NATIONAL BUREAU OF STANDARDS, Ernest Ambler, Director Issued October 1981 Library of Congress Catalog Card Number: 81-600087 National Bureau of Standards Special Publication 616 Nat. Bur. Stand. (U.S.), Spec. Publ.616, 248 pages (Oct. 1981) CODEN; XNBSAV U.S. GOVERNMENT PRINTING OFFICE WASHINGTON: 1981 For sale by the Superintendent of Documents, U.S. Government Printing Office, Washington, D.C. 20402 Price $7.50 (Add 25 percent for other than U.S. mailing) ABSTRACT The Symposium on Validation and Assessment of Energy Models, held at the Na- tional Bureau of Standards (NBS) , Gaithersburg , MD (May 19-21, 1980), was sponsored by the Energy Information Administration (EIA), of the Depart- ment of Energy (DOE), Washington, DC. The symposium was organized by NBS' Op erations Research Division with a two-fold agenda: (1) to summarize the re- cent ideas and advances of model validation and assessment that have been ap- plied to DOE energy models, and (2) to hold workshops on key open questions that are of concern to the validation and assessment research community. Speakers addressed current and future practices, the EIA model validation pro gram, model structure and data, and model credibility. Full-day workshop ses sions were held on the following topics: validating composite models, the measurement of model confidence, model structure and assessment, sensitivity and statistical analysis of models, and model assessment methodologies. This volume documents the symposium proceedings and includes the formal papers pre sented, discussant comments, panel discussions and questions and answers, and summaries of the issues and conclusions reached in the workshops. Key words: Assessment; composite models; data quality; energy models; mathe- matical models; model confidence; model credibility; policy models sensitivity analysis; validation. iii CONTENTS Introductary Remarks Saul I. Gass 1 Welcoming Remarks Lincoln E. Moses 3 Burton H. Colvin 5 Energy Model Evaluation and Analysis: Current Practice David 0. Wood 7 Humanizing Policy Analysis: Confronting the Paradox in Energy Modeling Martin Greenberger 25 Remarks on Wood and Greenberger Papers Ken Hoffman 43 Building Good Models is not Enough Richard Richels 49 Quality Control for Analysis George M. Lady 65 Validation and Assessment Procedures for Energy Models Jerry A. Hausman 79 Energy Model Assessment Procedure Development Project Richard H. F. Jackson 87 Comments on Validation and Assessment Procedures Allen L. Soyster 91 Are Energy Models Credible? David Freedman 93 Model Credibility: How Much Does it Depend Upon Data Quality? C. Roger Glass ey and Harvey J. Greenberg 117 Remarks on Freedman and Glassey-Greenberg Papers Douglas R. Hale 121 David A. Pilati 125 Discussion 129 A Note on the Human Factors Associated with Model Validation and Confidence David A. Pilati 137 Workshop Report 1 — Composite Model Validation Andy S. Kydes and Lewis Rubin 139 Workshop Report 2 —The Measurement of Model Confidence Lambert S. Joel and John S. Maybee 149 V Workshop Report 3 —Model Standards to Aid Assessment Milton L. Holloway 159 Workshop Report 4 — Sensitivity and Statistical Analysis of Models Carl Harris and David Hirshfeld 171 Workshop Report 5—Model Assessment Methodologies PART A: Enumeration of Validation Methodologies James Gruhl and David 0. Wood 183 PART B: Model Assessment Methodologies Thomas J. Woods 201 Panel Discussion on Workshop Reports , 205 Reflections on Modeling and Modeling Assessment Alan J. Goldman 217 Symposium Program and Attendees 243 Papers in this volume, except those by National Bureau of Standards authors, have not been edited or altered by the National Bureau of Standards. Opinions expressed in non-NBS papers are those of the authors, and not necessarily those of the National Bureau of Standards. Non-NBS authors are solely responsible for the content and quality of their submissions. The mention of trade names in the volume is in no sense an endorsement or recommendation by the National Bureau of Standards. vi INTRODUCTORY REMARKS Saul I. Gass Symposium Chairperson Good morning. I would like to welcome you to the Second Symposium on Valida- tion and Assessment of Energy Models sponsored by the Department of Energy and the National Bureau of Standards. The first symposium, which many of you were at, was held in January of 1979. That symposium emphasized the open issues and research aspects of energy model validation and assessment. This symposium highlights the advances, new directions, and major research activities of the last year and a half. We have attempted to do this by lim- iting the invited paper sessions to a few specific topics and to spend almost two full days on research workshops. I think I can best explain why we are here and why 1 think these sjrmposiums are of great importance by quoting from the May 16, 1980 editorial that appeared in Science Magazine . It was written by Judge David L. Bazelon, Senior Circuit Judge of the U. S. Court of Appeals for the District of Colum- bia Circuit. The title of the editorial is "Science, Technology^ and the Court." I quote: "I believe that the judicial responsibility is to monitor and scru- tinize the administrative