NBS SPECIAL PUBLICATION 406 U.S. DEPARTMENT OF COMMERCE / National Bureau of Standards' Computer Performance Evaluation: Report of the 1973 NBS/ACM Workshop NATIONAL BUREAU OF STANDARDS The National Bureau of Standards' was established by an act of Congress March 3, 1901. The Bureau's overall goal is to strengthen and advance the Nation's science and technology and facilitate their effective application for public benefit. To this end, the Bureau conducts research and provides: (1) a basis for the Nation's physical measurement system, (2) scientific and technological services for industry and government, (3) a technical basis for equity in trade, and (4) technical services to promote public safety. The Bureau consists of the Institute for Basic Standards, the Institute for Materials Research, the Institute for Applied Technology, the Institute for Computer Sciences and Technology, and the Office for Information Programs. THE INSTITUTE FOR BASIC STANDARDS provides the central basis within the United States of a complete and consistent system of physical measurement; coordinates that system with measurement systems of other nations; and furnishes essential services leading to accurate and uniform physical measurements throughout the Nation's scientific community, industry, and commerce. The Institute consists of a Center for Radiation Research, an Office of Meas- urement Services and the following divisions: Applied Mathematics — Electricity — Mechanics — Heat — Optical Physics — Nuclear Sciences" — Applied Radiation " — Quantum Electronics '' — Electromagnetics" — Time " and Frequency — Laboratory Astrophysics " — Cryogenics THE INSTITUTE FOR MATERIALS RESEARCH conducts materials research leading to improved methods of measurement, standards, and data on the properties of well-characterized materials needed by industry, commerce, educational institutions, and Government; provides advisory and research services to other Government agencies; and develops, produces, and distributes standard reference materials. The Institute consists of the Office of Standard Reference Materials and the following divisions: Analytical Chemistry — Polymers — Metallurgy — Inorganic Materials — Reactor Radiation — Physical Chemistry. THE INSTITUTE FOR APPLIED TECHNOLOGY provides technical services to promote the use of available technology and to facilitate technological innovation in industry and Government; cooperates with public and private organizations leading to the development of technological standards (including mandatory safety standards), codes and methods of test; and provides technical advice and services to Government agencies upon request. The Institute consists of a Center for Building Technology and the following divisions and offices: Engineering and Product Standards — Weights and Measures — Invention and Innova- tion — Product Evaluation Technology — Electronic Technology — Technical Analysis — Measurement Engineering — Structures, Materials, and Life Safety* — Building * Environment ' — Technical Evaluation and Application — Fire Technology. THE INSTITUTE FOR COMPUTER SCIENCES AND TECHNOLOGY conducts research and provides technical services designed to aid Government agencies in improving cost effec- tiveness in the conduct of their programs through the selection, acquisition, and effective utilization of automatic data processing equipment; and serves as the principal focus within the executive branch for the development of Federal standards for automatic data processing equipment, techniques, and computer languages. The Institute consists of the following divisions: Computer Services — Systems and Software — Computer Systems Engineering — Informa- tion Technology. THE OFFICE FOR INFORMATION PROGRAMS promotes optimum dissemination and accessibility of scientific information generated within NBS and other agencies of the Federal Government; promotes the development of the National Standard Reference Data System and a system of information analysis centers dealing with the broader aspects of the National Measurement System; provides appropriate services to ensure that the NBS staff has optimum accessibility to the scientific information of the world. The Office consists of the following organizational units: Office of Standard Reference Data — Office of Information Activities — Office of Technical Publications — Library — Office of International Relations. 1 Headquarters and Laboratories at Gaithersburg. Maryland, unless otherwise noted; mailing address Washington, D.C. 20234. " Part of the Center for Radiation Research. 3 Located at Boulder. Colorado 80302. * Part of the Center for Building Technology. Computer Performance Evaluation: Report of the 1973 NBS/ACM Workshop Edited by: Thomas E. Bell and Barry W. Boehm TRW Systems Redondo Beach, California 90278 and S. Jeffery Institute for Computer Sciences and Technology National Bureau of Standards Washington. D.C. 20234 U.S. DEPARTMENT OF COMMERCE, Rogers C. B. Morton, Secretary NATIONAL BUREAU OF STANDARDS, Ernest Ambler, Acting Director Issued September 1975 Library of Congress Cataloging in Publication Data Main entry under title: Computer Performance Evaluation. (NBS Special Publication; 406) Supt. of Docs. No.: C13.10:405 1. Electronic Digital Computers— Evaluation. I. Bell, Thomas E. II. Boehm, Barry W. III. Jeflery, Seymour, 1922- . IV. United States. National Bureau of Standards. V. Association for Computing Machinery. VI. Series: United States. National Bureau of Standards. Special Publication ; 406. QC100.U57 No. 406 [QA76.51 389'.08s [621.3819'58'21 75-619080 National Bureau of Standards Special Publication 406 Nat. Bur. Stand. (U.S.), Spec. Publ. 406, 180 pages (Sept. 1975) CODEN: XNBSAV U.S. GOVERNMENT PRINTING OFFICE WASHINGTON: 1975 sale For by the Superintendent of Documents, U.S. Government Printing Office, Washington, D.C. 20402 • Price $2.45 (paper cover) Stock Number 003-003-01.393 (Order by SD Catalog No. C13.10:406). Abstract An ACM/NBS Workshop on Computer Performance Evaluation (CPE) was held in San Diego, Calif, in March 1973. The papers, workshop discussions, conclusions and recommendations presented in this volume address specific problems in making computer performance evaluation a common- place and productive practice. While several of the conclusions indicate that improvements are needed in performance analysis tools, another suggests that improved application of CPE could be achieved by better documentation of analysis field to develop its full potential. Particapants noted that the common emphasis on data collection or modeling, to the exclusion of considering objectives, often seriously degrades the value of performance analysis. Key words: Computer architecture; computer performance evaluation; installation management; measurement; modeling: monitoring; operating systems; performance objectives. iii Foreword The Institute for Computer Sciences and Technology at the National Bureau of Standards, U.S. Department of Commerce, and the Association for Computing Machinery, the nation's largest technical society for computing professionals, have been jointly sponsoring a series of workshops and action conferences on national issues. These workshops were designed to bring together the best talents in the country in their respective areas to establish a consensus on (1) current state of the art, (2) additional action required, and (3) where the responsibility for such action lies. Computer Performance Evaluation (CPE) was selected as a subject of an ACM/NBS Workshop* because of the significant leverage CPE activities can have on computer usage. Under the right conditions, CPE can be a major force for improving the efficiency and effectiveness of computer and related human operations. However, it is important to beware of opportunities to misuse CPE techniques. Collecting vast amounts of data does not guarantee system improvement. Even when one can make an improve- ment to one system parameter, this may have a degrading effect on another parameter. Another serious problem arises when the results of modeling a system are impossible to validate. It is hoped that the publication of these proceedings will be of value to specialized investigators in the particular area discussed, as well as those actively engaged in attacking the broad problem of bringing CPE techniques into productive practice. Thomas E. Bell TRW Systems Group Barry W. Boehm TRW Systems Group S. Jeffrey National Bureau of Standards • The initial planning was supported by a grant from the National Science Foundation. V Contents Page Executive Summary x Keynote Address xii I. Workshop Summary 1 A. Introduction - 1 1. Potential Benefits 1 2. Potential Difficulties 1 3. Sources of Leverage 2 4. Objective of Workshop Report 2 5. Scope of Workshop Report 2 6. A Final Introductory Comment 3 B. Definitions and Critical Distinctions 3 1. Introduction 3 2. Components and Connections 3 8. Performance Improvement Avenues 7 4. Operational Significance of Distinctions 7 5. A Critical Distinction 8 C. Objectives and Criteria 8 1. Introduction 8 2. Consumer Concerns 9 3. Relating Measurement to Evaluation: General Considerations 9 4. Relating Measurement to Evaluation: Some Specifics 10 5. Multivariate Performance Criteria 12 D. Tools and Techniques 13 1. Introduction 13 2. Methodological Tools and Techniques 13 3. Data Collection Tools and Techniques 15 4. Data Analysis Tools and Techniques 17 5. Modeling Tools and Techniques 17 6. Management 17 7. Research and Development 18 E. Recommendations 18 1.