IBM Front cover Understanding IBM Eserver xSeries Benchmarks

What industry benchmarks are used in the xSeries marketplace

How to interpret the results of the benchmarks

How the benchmarks relate to client configurations

David Watts Slavko Bozic Craig

.com/redbooks Redpaper

International Technical Support Organization

Understanding IBM Eserver xSeries Benchmarks

May 2005 Note: Before using this information and the product it supports, read the information in “Notices” on page v.

First Edition (May 2005)

This edition applies IBM Eserver xSeries and BladeCenter servers.

This document created or updated on May 13, 2005.

© Copyright International Business Machines Corporation 2005. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Contents

Notices ...... v Trademarks ...... vi

Preface ...... vii The team that wrote this Redpaper ...... vii Become a published author ...... viii Comments welcome...... ix

Chapter 1. Benchmarks 101 ...... 1 1.1 Types of benchmarks ...... 2 1.2 A level playing field ...... 3 1.3 A rigorous process means credible results ...... 3 1.4 Current industry standard benchmarks ...... 4 1.4.1 Evolving benchmarks ...... 4 1.5 Invalid comparisons ...... 5 1.5.1 Server versus workstation benchmarks ...... 5 1.5.2 Apples-to-apples comparison ...... 5 1.5.3 Performance versus price/performance ...... 5 1.5.4 Results versus reality ...... 6

Chapter 2. IBM and benchmarks ...... 7 2.1 Why IBM runs benchmarks on xSeries servers ...... 8 2.2 Client benefits ...... 8 2.3 Benchmarks important to IBM...... 8 2.4 How xSeries fared...... 10 2.4.1 The xSeries performance lab ...... 10 2.4.2 xSeries and HPC Benchmark Centers ...... 11 2.4.3 IBM Eserver family ...... 11 2.4.4 Enterprise X-Architecture technology ...... 14 2.5 How the industry benefits ...... 14 2.5.1 IBM Center for Microsoft Technologies...... 15 2.5.2 Technology Center ...... 16 2.6 Examples of xSeries benchmark setups ...... 16 2.6.1 TPC-C on an xSeries x445 ...... 16 2.6.2 SPECweb99 on BladeCenter ...... 17

Chapter 3. Industry benchmarks ...... 19 3.1 What benchmarks are available ...... 20 3.1.1 Transaction Processing Performance Council ...... 20 3.1.2 Standard Performance Evaluation Corporation ...... 20 3.1.3 Application and other standardized benchmarks ...... 21 3.2 Rules of engagement ...... 21 3.2.1 TPC rules of engagement ...... 22 3.2.2 SPEC rules of engagement ...... 22 3.3 Selecting the relevant benchmark...... 23 3.4 Workload descriptions...... 24 3.4.1 Online Transaction Processing (OLTP) ...... 24 3.4.2 Decision Support (DSS) ...... 25 3.4.3 Web servers ...... 25

© Copyright IBM Corp. 2005. All rights reserved. iii 3.4.4 Application server and ERP ...... 26 3.4.5 Mail and collaboration ...... 26 3.5 Benchmark descriptions ...... 27 3.5.1 TPC-C ...... 27 3.5.2 TPC-H ...... 28 3.5.3 TPC-W and TPC-App ...... 29 3.5.4 SAP Standard Application Benchmark ...... 31 3.5.5 Oracle Application Standard Benchmark ...... 33 3.5.6 SPEC CPU2000 ...... 34 3.5.7 SPECWeb99 and SEPCWeb99_SSL...... 35 3.5.8 SPECjAppServer ...... 36 3.5.9 SPECjbb2000 ...... 37

Chapter 4. Understanding benchmark results ...... 39 4.1 Examining a benchmark result ...... 40 4.2 Finding information ...... 40 4.3 Disclosure reports ...... 41 4.4 Workload characteristics affect system performance ...... 42 4.5 Types of benchmarks ...... 44 4.5.1 Component level benchmarks...... 44 4.5.2 System-level benchmarks...... 44 4.6 Benchmark result considerations ...... 44 4.6.1 Know what you are comparing ...... 44 4.6.2 Maximum performance versus price/performance ...... 45 4.6.3 Clustered system benchmarks ...... 45 4.7 Benchmark results as part of an evaluation process ...... 46

Chapter 5. Client-related benchmarks...... 49 5.1 Industry benchmarks versus client benchmarks ...... 50 5.2 Alternatives to client benchmarking ...... 50 5.3 Why perform your own benchmarking ...... 52 5.4 What to expect if you must do your own benchmarking ...... 52 5.5 Traps and pitfalls of doing your own benchmarking ...... 52 5.6 Summary...... 55

Abbreviations and acronyms ...... 57

Related publications ...... 59 IBM Redbooks ...... 59 Online resources ...... 59 How to get IBM Redbooks ...... 60 Help from IBM ...... 60

iv Understanding IBM Eserver xSeries Benchmarks Notices

This information was developed for products and services offered in the U.S.A.

IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing, IBM Corporation, North Castle Drive Armonk, NY 10504-1785 U.S.A.

The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you.

This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice.

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk.

IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you.

Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.

This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental.

COPYRIGHT LICENSE: This information contains sample application programs in source language, which illustrates programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs confor