The IBM Eserver Bladecenter JS20

The IBM Eserver BladeCenter JS20

Learn about IBM Eserver BladeCenter JS20 and blade server technology

Understand the PowerPC 970 microprocessor architecture

Plan for, install, and configure BladeCenter JS20

Ben Gibbs Omkhar Arasaratnam Robert Arenburg

Bradley Elkin Rogeli Grima James Kelly

ibm.com/redbooks

International Technical Support Organization

The IBM Eserver BladeCenter JS20

June 2005

SG24-6342-01

Note: Before using this information and the product it supports, read the information in “Notices” on page xi.

Second Edition (June 2005)

This edition applies to the IBM Eserver BladeCenter JS20, type 8842.

© Copyright International Business Machines Corporation 2005. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Contents

Figures ...... vii

Tables ...... ix

Notices ...... xi Trademarks ...... xii

Preface ...... xiii The team that wrote this redbook...... xiii Become a published author ...... xv Comments welcome...... xv

Part 1. Introduction to the BladeCenter JS20...... 1

Chapter 1. Introduction to BladeCenter and blade server technology. . . . 3

Chapter 2. Hardware components ...... 7 2.1 Overview of the BladeCenter infrastructure ...... 8 2.1.1 BladeCenter chassis ...... 8 2.1.2 BladeCenter power options...... 10 2.1.3 Management Module ...... 13 2.1.4 I/O modules...... 16 2.1.5 Blade servers ...... 21 2.2 BladeCenter JS20 ...... 22 2.2.1 Base features ...... 24 2.2.2 Optional features...... 27 2.3 PowerPC 970 and PowerPC 970FX Microprocessors ...... 29 2.3.1 Review of POWER and PowerPC Architecture ...... 30 2.3.2 Vector/SIMD Multimedia eXtension ...... 38 2.3.3 Features of the PowerPC 970 and PowerPC 970FX ...... 42

Chapter 3. Software environment ...... 47 3.1 Operating system support ...... 48 3.1.1 AIX ...... 48 3.1.2 Red Hat Enterprise Linux ...... 48 3.1.3 SUSE LINUX Enterprise Server ...... 49 3.2 System management tools ...... 49 3.2.1 BladeCenter Web interfaces ...... 49 3.2.2 IBM Director ...... 50 3.2.3 IBM Cluster Systems Management...... 50

4.1.2 High-performance, low-latency network requirements ...... 58 4.1.3 Multiple BladeCenter environments ...... 59 4.2 Operating system installation ...... 60 4.2.1 Network installation planning ...... 60 4.3 Systems management...... 63 4.3.1 IBM Director ...... 63 4.3.2 IBM Cluster Systems Management...... 64

Part 2. Implementing the BladeCenter JS20...... 67

Chapter 5. Hardware setup...... 69 5.1 Management Module configuration ...... 70 5.2 LAN Switch I/O Module configuration ...... 73 5.3 Blade server configuration...... 76 5.3.1 Assigning names to blade servers ...... 77 5.3.2 Setting the boot sequence ...... 78 5.4 Firmware ...... 79 5.4.1 Management Module firmware ...... 80 5.4.2 LAN Switch I/O Module firmware ...... 82 5.4.3 BladeCenter JS20 firmware (BIOS) ...... 84 5.4.4 Integrated Systems Management Processor firmware ...... 85 5.5 Providing a console for the JS20 blades ...... 87 5.5.1 Configuring Serial over LAN ...... 89 5.5.2 Using Serial over LAN...... 91 5.5.3 Open Firmware interfaces...... 94 5.5.4 Specifying IP parameters to Open Firmware ...... 96

Chapter 6. Installing Linux ...... 101 6.1 Installing Linux...... 102 6.1.1 Configuring the sources ...... 102 6.1.2 The zImage.initrd file...... 107 6.2 Configuring BOOTP and TFTP ...... 109 6.2.1 BOOTP ...... 110 6.2.2 Trivial File Transfer Protocol ...... 111 6.3 Preparing an unattended install...... 111 6.3.1 Red Hat Kickstart ...... 112 6.3.2 SuSE AutoYaST ...... 114 6.4 Performing an unattended installation...... 116 6.4.1 Open Firmware ...... 117 6.4.2 mkzimage_cmdline: SuSE only...... 117 6.5 Performing a network installation ...... 118 iv The IBM Eserver BladeCenter JS20 6.5.1 SUSE LINUX Enterprise Server 9...... 118 6.5.2 Red Hat Enterprise Linux 3 ...... 121

Chapter 7. Installing AIX on the JS20 ...... 123 7.1 Minimal NIM installation ...... 124 7.2 Adding resources ...... 127 7.3 Adding machine information ...... 130 7.4 Preparing the NIM master ...... 134 7.5 Preparing the client ...... 135

Chapter 8. System management scenarios ...... 139 8.1 IBM Director ...... 140 8.1.1 Setting up an IBM Director Server ...... 142 8.1.2 Installing an IBM Director Agent ...... 145 8.1.3 Using Director to manage JS20s ...... 147 8.2 IBM Cluster Systems Management...... 147 8.2.1 Setting up a CSM management node ...... 148 8.2.2 Installing and managing BladeCenter JS20s using CSM ...... 154

Abbreviations and acronyms ...... 159

Related publications ...... 161 IBM Redbooks ...... 161 Other publications ...... 161 Online resources ...... 162 How to get IBM Redbooks ...... 164 Help from IBM ...... 164

Index ...... 165

Contents v

vi The IBM Eserver BladeCenter JS20 Figures

2-1 BladeCenter chassis front view ...... 8 2-2 BladeCenter chassis rear view ...... 9 2-3 BladeCenter power domains ...... 11 2-4 KVM Management Module ...... 14 2-5 Management Module connectors...... 15 2-6 BladeCenter JS20 physical layout ...... 22 2-7 BladeCenter JS20 block diagram ...... 23 2-8 PowerPC logical processing model ...... 34 2-9 PowerPC registers used by application programs ...... 35 2-10 Scalar and vector operations ...... 39 2-11 PowerPC with VMX logical processing model ...... 40 2-12 VMX registers ...... 41 2-13 PowerPC 970 internal organization ...... 43 4-1 Network logical view ...... 55 4-2 Myrinet network infrastructure ...... 59 5-1 Management Module controls and indicators ...... 71 5-2 IP address configuration ...... 72 5-3 LAN Switch I/O Module IP address ...... 74 5-4 LAN Switch I/O Module setup ...... 75 5-5 Blade server naming ...... 77 5-6 Boot sequence of blades ...... 78 5-7 Viewing firmware levels on BladeCenter ...... 80 5-8 Updating the Management Module firmware ...... 81 5-9 Firmware upgrade of 4-Port Gb Ethernet Switch Module ...... 83 5-10 Blade firmware update ...... 86 5-11 Serial over LAN components ...... 88 5-12 Serial over LAN configuration ...... 90 5-13 Serial over LAN status ...... 91 5-14 Management Module’s command line interface ...... 92 5-15 Setting CRLF in Windows 2000 Telnet ...... 94 5-16 POST checkpoint codes ...... 95 6-1 YaST installation server configuration ...... 104 6-2 Selecting the distribution ...... 105 6-3 Source Configuration pane ...... 106

6-4 Completing the setup ...... 107 6-5 redhat-config-kickstart main window ...... 112 6-6 AutoYaST module within YaST ...... 114 6-7 Main AutoYaST window...... 115

6-11 Source menu ...... 120 6-12 Protocol menu ...... 120 6-13 Installation location ...... 121 7-1 NIM configuration window ...... 124 7-2 Configuring as a NIM Master ...... 125 7-3 Selecting the NIM interface and network name ...... 125 7-4 Choosing to initialize minimal NIM environment ...... 126 7-5 Finalizing the NIM environment settings ...... 126 7-6 lpp_source and SPOT configuration ...... 127 7-7 AIX source ...... 128 7-8 Default resource settings ...... 128 7-9 Completing the resource installation ...... 129 7-10 Adding new machine information to NIM ...... 130 7-11 Host name of new machine ...... 131 7-12 Entering machine-specific information ...... 132 7-13 Entering MAC address information ...... 133 7-14 Installing an operating system from NIM ...... 134 7-15 Setting BOS preferences ...... 135 7-16 Installation language panel ...... 136 7-17 Operating system selection ...... 136 7-18 Installation and language options ...... 137 7-19 Kernel options ...... 138 8-1 Selecting the type of database...... 143 8-2 Database connection information ...... 144 8-3 Setting Director server encryption ...... 145 8-4 Installing the Director Agent...... 146 8-5 Configuring Director Agent security ...... 146

viii The IBM Eserver BladeCenter JS20 Tables

2-1 Power domain A minimum power requirements ...... 12 2-2 Power domain B minimum power requirements ...... 13 2-3 Physical attributes of PowerPC 970 and PowerPC 970FX ...... 42 5-1 BladeCenter and BladeCenter JS20 firmware components ...... 79 5-2 Management Module commands for SoL ...... 93 5-3 Network configuration information worksheet ...... 96

x The IBM Eserver BladeCenter JS20 Notices

This information was developed for products and services offered in the U.S.A. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or service. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing, IBM Corporation, North Castle Drive Armonk, NY 10504-1785 U.S.A. The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk. IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you. Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental. COPYRIGHT LICENSE: This information contains sample application programs in source language, which illustrates programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. You may copy, modify, and distribute these sample programs in any form without payment to IBM for the purposes of developing, using, marketing, or distributing application programs conforming to IBM's application programming interfaces.

The following terms are trademarks of the International Business Machines Corporation in the United States, other countries, or both:

Eserver ® BladeCenter™ POWER3™ Eserver® Domino® POWER4™ developerWorks® IBM® POWER4+™ eServer™ Lotus® POWER5™ ibm.com® PowerPC Architecture™ Redbooks™ iSeries™ PowerPC 601® Redbooks (logo) ™ pSeries® PowerPC 603™ RETAIN® xSeries® PowerPC 604™ RS/6000® AIX 5L™ PowerPC® ServeRAID™ AIX® POWER™ System/360™ AS/400® POWER2™ 3090™

The following terms are trademarks of other companies: Intel, Intel Inside (logos), MMX, and Pentium are trademarks of Intel Corporation in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. UNIX is a registered trademark of The Open Group in the United States and other countries. Other company, product, and service names may be trademarks or service marks of others.

xii The IBM Eserver BladeCenter JS20 Preface

Blade servers are a relatively new technology. They have captured industry focus because of their modular design, which can reduce cost with a more efficient use of valuable floor space. They offer simplified management, which can help to speed such tasks as installing, reprovisioning, updating, and troubleshooting hundreds of blade servers. You can do all of this remotely using one graphical console with IBM® Director systems management tools.

In addition, blade servers provide improved performance by doubling current rack density. By integrating resources and sharing key components, costs decrease and availability increases.

The IBM Eserver® BladeCenter™ boasts innovative modular technology, leadership density, and availability. It was designed to help solve a multitude of real-world problems.

This IBM Redbook takes an in-depth look at the IBM Eserver BladeCenter JS20. This is a two-way blade server for applications requiring 64-bit computing. It is ideal for computer-intensive applications and transactional Internet servers. This IBM Redbook helps you to install, tailor, and configure the IBM Eserver BladeCenter JS20.

The team that wrote this redbook

This redbook was produced by a team of specialists from around the world working at the International Technical Support Organization (ITSO), Austin Center.

Ben Gibbs is a Senior Consulting Engineer with Technonics, Inc. (http://www.technonics.com) in Austin, Texas. He has over 20 years of experience with UNIX®-based operating systems. He started working with the AIX® operating system in November of 1989. His areas of expertise include performance analysis and tuning, operating system internals, and device driver development for the AIX operating system. He is also an IBM Learning Services instructor for advanced AIX classes. He was the project leader for this IBM Redbook. Omkhar Arasaraknum is a Team Leader with IBM Global Services in Canada. He has worked with Linux since 1998 and uses SuSE, Red Hat, and Gentoo. Omkhar is the Linux technical lead within his Service Delivery Center and has

Robert Arenburg is a Senior Technical Consultant in the Solutions Enablement organization in the IBM Systems and Technology Group and is located in Austin,

Texas. He has a PhD in Engineering Mechanics from Virginia Tech. He has worked at IBM for 13 years. His areas of expertise include high performance computing, performance, capacity planning, 3D graphics, solid mechanics, computational, and finite element methods.

Bradley Elkin is a Senior Software Engineer for IBM U.S. He holds a PhD in Chemical Engineering from the University of Pennsylvania and has 17 years of experience in high performance computing. His areas of expertise include applications in computational fluid mechanics, computational chemistry, and bioinformatics. He has written several articles for IBM ^ Development Domain.

Rogeli Grima is a research staff member at the CEPBA IBM Research Institute in Spain. He has three years of experience in applied mathematics. He has also collaborated on the development of BladeCenter JS20 solution for bioinformatics.

James Kelly is a Systems Architect in IBM Australia specializing in deep computing. He has 17 years of experience in system engineering and technical sales support. His current areas of expertise include the solution design, benchmarking, and installation of server clusters running either AIX or Linux, to support the installation of high performance computing applications.

Thanks to the following people for their contributions to this project:

ITSO, Austin Center

Trina Bunting IBM Dallas

Luciano Chavez Matt Davis Martin Gramlich Jon Mason Doug W Oliver Brian Rzycki IBM Austin Anton Blanchard IBM OzLabs, Canberra, Australia

xiv The IBM Eserver BladeCenter JS20 Janet Ellsworth IBM Poughkeepsie, New York

Become a published author

Join us for a two- to six-week residency program! Help write an IBM Redbook dealing with specific products or solutions, while getting hands-on experience with leading-edge technologies. You'll team with IBM technical professionals, Business Partners and/or customers.

Your efforts will help increase product acceptance and customer satisfaction. As a bonus, you'll develop a network of contacts in IBM development labs, and increase your productivity and marketability.

Find out more about the residency program, browse the residency index, and apply online at: ibm.com/redbooks/residencies.html

Comments welcome

Your comments are important to us!

We want our Redbooks™ to be as helpful as possible. Send us your comments about this or other Redbooks in one of the following ways: Use the online Contact us review redbook form found at: ibm.com/redbooks Send your comments in an Internet note to: [email protected] Mail your comments to: IBM Corporation, International Technical Support Organization Dept. JN9B Building 905 Internal Zip 9053D003 11501 Burnet Road Austin, Texas 78758-3493

Preface xv

xvi The IBM Eserver BladeCenter JS20

Part 1

Part 1 Introduction to the BladeCenter JS20

This part introduces and describes the hardware and software components of the IBM Eserver BladeCenter JS20. It also presents planning considerations.

2 The IBM Eserver BladeCenter JS20

Chapter 1. Introduction to BladeCenter and blade server technology

This chapter presents an overview of IBM Eserver BladeCenter and blade server technology. It also covers the hardware for the chassis and the blades.

The term blade server refers to a chassis that can hold a number of hot-swappable devices called blades. Blades come in two varieties: server blades and option blades.

A server blade is an independent server that contains one or more processors and associated memory, disk storage, and network controllers. It runs its own operating system and applications. Each server blade within a system chassis slides into a blade bay2. It plugs into a midplane or backplane to share common infrastructure components. These components may include power supplies, fans, CD-ROM, and floppy drives, Ethernet and Fibre Channel switches, and system ports.

Option blades may be sharable by server blades. Option blades provide additional features, such as controllers for external I/O or disk arrays, power supplies, and so on.

For organizations that seek server consolidation, BladeCenter centralizes servers for increased flexibility, ease of maintenance, reduced cost, and streamlined human resources. Companies that need to install new applications

© Copyright IBM Corp. 2005. All rights reserved. 3 for e-commerce and On Demand Business can achieve speed while ensuring flexibility, scalability, and availability. For such enterprise requirements as file-and-print and collaboration, BladeCenter is designed to offer reliability, flexibility for growth, and cost effectiveness. And clients with compute-intensive applications that need highly available clustering can use the BladeCenter to help achieve high degrees of scalability and performance.

Benefits of the IBM Eserver BladeCenter family The IBM Eserver BladeCenter family of products features a modular design that integrates multiple computing resources into a cost-effective, high-density enclosure for a platform that provides: Lower cost in certain configurations: Due to efficiencies in power usage, heat emissions, and data center floor space utilization, when a configuration of other servers exceeds one rack, it is often less expensive over a multi-year period to use a single rack of BladeCenter chassis and BladeCenter JS20s. Physical server consolidation: A BladeCenter server is an ideal way to replace many uniprocessor or 2-way servers to save space. 14U of rack space for 14 1U servers can be replaced with one 7U BladeCenter chassis. Additional rack space that is normally taken for Ethernet, Fibre Channel, and other switches is eliminated by the integrated switches in the BladeCenter chassis. High availability: The BladeCenter chassis offers redundant and hot-swap components that can prevent failure of the chassis or blade servers when one of those components fail. The chassis has redundant, hot-swap cooling, power, midplane features, management, and I/O switches. It also has a hot-swap media tray, with CD-ROM and floppy drive, that can be removed and serviced while all blade servers are operating. Integrated switch technology: All service is from the front and rear of BladeCenter. There is no need to slide the chassis out of the rack and remove the top cover for service. Also, numerous cables are eliminated, reducing both cabling cost and servicer or administrator time. More integrated systems management features: The BladeCenter chassis includes a management module. This module eliminates the need for individual management adapters, such as the Remote Supervisor Adapter I or II, in each blade server for remote control. It also eliminates the need for RS485 interconnect cabling between the blade servers. SAN optimization: A Fibre Channel switch module installed in a BladeCenter chassis provides storage area network (SAN) access to all blade servers in the chassis without internal cabling.

4 The IBM Eserver BladeCenter JS20 BladeCenter versus an IBM rack server BladeCenter is an ideal solution for certain environments. In other environments, an IBM rack server may be a better fit: Need for a small number of servers: A BladeCenter chassis is required for one blade to be cost effective compared to some stand-alone servers. Therefore, a chassis should be full or nearly full. Need for existing Peripheral Component Interconnect (PCI) adapters: BladeCenter does not include adapter slots as shipped. An optional I/O Expansion Card feature offers one PCI slot per blade, limiting the blade to one integrated development environment (IDE) drive. The expansion feature supports PCI cards designed for BladeCenter. Therefore, a mix of BladeCenter and traditional rack-optimized servers may be appropriate. Need for large internal storage or external SCSI storage: BladeCenter supports a maximum of 80 GB of IDE or 146.8 GB of SCSI storage internally. While there is no provision for external SCSI storage, there is an external Fibre Channel SAN storage option. Using the optional internal SCSI storage feature doubles the space requirement of the blade, cutting in half the number of blade servers that can be installed in a chassis. For large internal storage, the IBM Eserver xSeries® 345 is capable of holding up to 880.8 GB of hot-swap SCSI storage. All rack-mounted xSeries servers can support, either natively or via an optional ServeRAID™ adapter, external SCSI arrays. Need for re-installation or repurposing at end of original: Stand-alone uniprocessor and 2-way servers can be distributed individually for use as departmental file/print servers and other low-horsepower uses. Because blade servers cannot be used without a chassis, the entire chassis-and-blades combo needs to stay together. The original chassis may be useful if older blade servers are discarded and newer, faster ones take their places in the chassis. Power requirements: The data center is wired for 110-120 V power. BladeCenter requires 220-240 V power.

Learn more about IBM Eserver BladeCenter and its components in the IBM Redpaper The Cutting Edge: IBM Eserver BladeCenter, REDP-3581.

The following chapters address the descriptions and considerations of the hardware and software components.

Chapter 1. Introduction to BladeCenter and blade server technology 5

6 The IBM Eserver BladeCenter JS20

Chapter 2. Hardware components

This chapter describes the major hardware components associated with the IBM ^ BladeCenter JS20. It begins by reviewing the core BladeCenter infrastructure. Then it provides a detailed description of the BladeCenter JS20 and the range of options that you can order from IBM. Finally, this chapter reviews the IBM PowerPC® 970 RISC Microprocessor and IBM PowerPC 970FX RISC Microprocessor that are used as main processors on the BladeCenter JS20.

The BladeCenter JS20 is designed to be installed in the IBM ^ BladeCenter Type 8677. This section reviews the BladeCenter infrastructure that is relevant to the installation of the BladeCenter JS20.

2.1.1 BladeCenter chassis The core component of the BladeCenter infrastructure is the BladeCenter chassis. Figure 2-1 illustrates the front view of the BladeCenter chassis. The BladeCenter chassis is designed to be installed in an industry standard 19-inch rack. Each BladeCenter chassis occupies seven rack units. You can install up to six BladeCenter chassis in a single 42U rack such as the IBM NetBAY42 Enterprise Rack.

In the front view of the BladeCenter chassis, you see these standard features: Fourteen bays where you can install blade servers A media bay containing one CD-ROM drive, one diskette drive, and a Universal Serial Bus (USB) port that can be dynamically assigned to any single blade server in the BladeCenter chassis

Figure 2-1 BladeCenter chassis front view

8 The IBM Eserver BladeCenter JS20 Figure 2-2 illustrates the rear view of the BladeCenter chassis. In the rear of the BladeCenter chassis, you see these standard features:

One management module One bay where you can install an optional redundant Management Module

Four bays where you can install optional input/output (I/O) modules A redundant pair of power supply modules Two bays where you can install an additional pair of redundant power supply modules A redundant pair of blowers

Figure 2-2 BladeCenter chassis rear view

Chapter 2. Hardware components 9 Attention: Nonredundant power is not supported in BladeCenter products. Power modules must always be present in power bays 1 and 2. When any blade server or option is in blade bays 7 through 14, power modules must be present in all four power-module bays.

If a power module fails or an ac power failure occurs, BladeCenter units configured for redundant power operation operate in a nonredundant mode, and the blower modules runs at full speed. You must replace the failing power module or restore ac power as soon as possible to regain redundant power operation and to reset the blower modules to their normal operating speed.

As of the date of this publication, these four BladeCenter power-module options are supported: IBM BladeCenter 1200W Power Supply Module (part number 48P7052) IBM BladeCenter 1200W to 1400W Power Supply Upgrade Kit (part number 90P0197) IBM BladeCenter 1800W Power Supply Module (part number 13N0570) IBM BladeCenter 2000W Power Supply Module (part number 26K4816)

Not visible in either view of the BladeCenter chassis are the redundant pair of midplanes in the center of the BladeCenter chassis that interconnect the blade servers installed in the front, and other modules that are installed in the rear of the BladeCenter chassis. The midplanes support the hot-plugging of blade servers and the other BladeCenter modules.

Note: IBM offers a variant of the BladeCenter called the BladeCenter T. The BladeCenter T is designed to meet the special needs of the telecommunications industry and has been tested for NEBS compliance. Since BladeCenter JS20 is not supported in the BladeCenter T at this time, we do not describe BladeCenter T and its associated features in this book.

2.1.2 BladeCenter power options The standard redundant power supplies are installed in power bays 1 and 2 of the BladeCenter chassis, and provide power to the first six blade server bays. This is known as power domain A.

To install blade servers in the remaining bays 7 through 14, you must install an additional pair of redundant power supply modules in power bays 3 and 4. This is known as power domain B. Figure 2-3 shows these two power domains.

10 The IBM Eserver BladeCenter JS20

Figure 2-3 BladeCenter power domains

Chapter 2. Hardware components 11 Attention: Power supply modules must always be installed in pairs. You can have either two power supply modules (in power bays 1 and 2) or four power supply modules (in power bays 1 through 4). Both power supply modules in a redundant pair should have the same capacity.

When setting up the power requirements for the BladeCenter, remember the following criteria: Install the power modules in pairs in a domain. They must match each other in capacity (wattage, amperage, and so on). A power domain operating above the capacity of a single power module results in a nonredundant power condition. In a pair of power modules, a power module that is not connected to a 200–240 volt ac power source results in a nonredundant power condition. To provide true redundant power, connect BladeCenter power modules 1 and 3 to a different 200–240 volt ac power source than to power modules 2 and 4. Connect an installed power module to an ac power source. The module must not be used as a filler.

Attention: To maintain proper system cooling, each unoccupied blade bay must contain a filler blade as well as each unoccupied I/O, management-module, or power-module bay. An installed power module must be powered, but must not be used as a filler.

Table 2-1 shows the minimum power requirements for power domain A.

Table 2-1 Power domain A minimum power requirements Sum of JS20 power units Minimum power module required

Less than 7.4 1200 W (labeled 7.5A)

7.4 up to less than 9.0 1400 W (labeled 9A)

9.0 up to less than 10.0 1800 W (labeled 12A)

10.0 or greater 2000 W (labeled 13.5A)

12 The IBM Eserver BladeCenter JS20 Table 2-2 shows the minimum power requirements for power domain B.

Table 2-2 Power domain B minimum power requirements

Sum of JS20 power units Minimum power module required

Less than 9.9 1200 W (labeled 7.5A)

9.9 up to less than 11.5 1400 W (labeled 9A)

11.5 up to less than 13.4 1800 W (labeled 12A)

13.4 or greater 2000 W (labeled 13.5A)

Each BladeCenter JS20 (1.6 GHz/800 MHz 8842-21X) requires 1.5 power units for two installed microprocessors. For example, if you fully populate all six bays, the total power units is 9.0 (6 bays times 1.5). According to Table 2-1, you need a minimum of the 1800 W power module.

For other configurations, refer to the latest version of IBM ^ BladeCenter Power Module Upgrade Guidelines Technical Update, found at: http://www.ibm.com/pc/support/site.wss/document.do?lndocid=MIGR-53353

Also, you can use the following steps to obtain the BladeCenter Planning and Installation Guide: 1. Go to: http://www.ibm.com/pc/support/ 2. In the Browse by product section, click Servers, and then select the brand Servers and the family BladeCenter. 3. Click Continue. 4. From the View by document type menu, click Online Publications.

2.1.3 Management Module Each BladeCenter chassis comes standard with a Management Module, as shown in Figure 2-4. You can also install a redundant Management Module by ordering the IBM KVM/Redundant Management Module Option (P/N 48P7055).

When you have two Management Modules, only one is active. The other module is in standby mode until it is switched to become the active Management Module.

Chapter 2. Hardware components 13

Figure 2-4 KVM Management Module

The Management Module provides the following three major functions: Console switching and remote graphical console for blade servers that use a keyboard, video monitor, and mouse (KVM) for their console Serial over local area network (LAN) (SoL) remote text consoles for blade servers that support SoL consoles (includes the BladeCenter JS20) Service processor for the entire BladeCenter

Important: The console switching and remote graphical console functionality of the Management Module is not used by the BladeCenter JS20 because it does not support KVM consoles. You can still use the console switching and remote graphical console functions if you have other blade servers installed in the same BladeCenter chassis that support KVM consoles.

You use the service processor functionality of the Management Module to perform many different tasks to: Initially configure BladeCenter Define and manage the users who can remotely access the Management Module Monitor the BladeCenter and reporting abnormal conditions to system managers either via e-mail or Simple Network Management Protocol (SNMP) events sent to management systems Remotely power on and off individual blade servers and I/O modules

14 The IBM Eserver BladeCenter JS20 Remotely switch the CD-ROM, diskette, and USB interfaces in the media bay of the BladeCenter chassis to a particular blade server

Maintain the firmware embedded in various BladeCenter components

The Management Module is connected to each blade server and I/O module via management buses embedded in the BladeCenter chassis midplanes. The Management Module uses these connections to provide service processor functionality in conjunction with the Blade Systems Management Processor (BSMP) on each blade server.

When using the SoL remote text console function, the Management Module also uses a virtual LAN (VLAN) provided by the LAN switch module installed in I/O module bay 1 to communicate with each blade server. This VLAN is not accessible externally.

Each Management Module is equipped with several connectors, as illustrated in Figure 2-5.

Figure 2-5 Management Module connectors

Chapter 2. Hardware components 15 You use the keyboard, video, and mouse connectors to attach a local KVM console that can be switched to any blade server that supports KVM consoles. You use the 10/100BaseT Ethernet connector to attach the Management Module to an external Ethernet network. You use this interface to access the remote graphical console, SoL consoles, and the service processor functionality of the Management Module.

You can access the various functions of the Management Module via the Ethernet interface using several different methods: Using a graphical interface through a Web browser (HTTP or HTTPS) Using a command line interface through a Telnet or SSH client Using system management software such as IBM Director or IBM Cluster Systems Management

We demonstrate these methods in Chapter 5, “Hardware setup” on page 69, and in Chapter 8, “System management scenarios” on page 139.

2.1.4 I/O modules I/O modules enable connectivity between blade servers within the same chassis, and between blade servers and the outside world. You can install up to four I/O modules in a BladeCenter chassis, one in each I/O module bay.

The available I/O modules can be used to provide connectivity to three different types of network: Local area networks Storage area networks (SANs) High-bandwidth low-latency networks used in clustering (Myrinet)

Most I/O modules support connectivity to only one type of network. The exception is the pass-through module, which can be used with all three types of networks.

Each I/O module is connected to every blade server via the BladeCenter chassis midplanes. Each single width blade server can have up to four interfaces, one to each I/O module bay. Double-width blade servers can have up to eight interfaces, two to each I/O module bay.

Each blade server must have a compatible interface to communicate with an I/O module. Every blade server that is currently available has standard Gigabit Ethernet interfaces that are connected to I/O module bays 1 and 2. Therefore, you can use only a LAN Switch or Pass-Thru I/O Module in these I/O module bays.

16 The IBM Eserver BladeCenter JS20 Restriction: You must install one of the LAN switch modules in I/O module bay 1 when using the SoL remote text console function. This function depends on a private VLAN provided by the LAN switch to function correctly.

Every single-width blade server currently available supports the installation of an optional expansion card that connects to I/O module bay 3 and 4. Double-width blade servers support two optional expansion cards. If you install an expansion card, it must be compatible with any I/O modules installed in I/O module bays 3 and 4.

You can order the following LAN Switch I/O Modules: P/N 13N0568: 4 port Gigabit Ethernet Switch Module P/N 13N2281: CISCO Systems Intelligent Gigabit Ethernet Switch Module P/N 73P9057: Nortel Networks Layer 2-7 Gigabit Ethernet Switch Module

You can order the following SAN switch I/O modules: P/N 48P7062: 2-Port Fibre Channel Switch Module P/N 26K5601: Brocade Entry SAN Switch Module P/N 90P0165: Brocade Enterprise SAN Switch Module

For the Pass-Thru I/O Module, you can order Optical Pass-thru Module (P/N 02R9080).

We describe each of these modules in more detail in the sections that follow.

IBM 4-Port Gigabit Ethernet Switch Module The IBM 4-Port Gigabit Ethernet Switch Module is a layer 2 LAN switch that provides four external 10/100/1000BaseT interfaces for connecting to LANs that are external to the BladeCenter chassis. Internally the switch module is designed to connect to a single Gigabit Ethernet interface on each installed single-width blade server, or two such interfaces on each installed double-width blade server.

The IBM 4-Port Gigabit Ethernet Switch Module is also connected via a Fast Ethernet interface to each installed Management Module. This is used to remotely access the switch module via the Management Module’s 10/100BaseT Ethernet interface. You can use this capability to support initial configuration and management of the switch module.

Major features of the 4-Port Gigabit Ethernet Switch Module include:

VLAN support based on either port or 802.1Q tagging Link aggregation of multiple external interfaces to provide increased bandwidth to external networks

Chapter 2. Hardware components 17 Note: The IBM 4-Port Gb Ethernet Switch Module for BladeCenter may be installed in your BladeCenter unit and you must replace the CD-ROM media tray in your BladeCenter unit. In this case, you must also update the microcode in the Ethernet switch module.

CISCO Systems Intelligent Gigabit Ethernet Switch Module The CISCO Systems Intelligent Gigabit Ethernet Switch Module (IGESM) is a layer 2-3 LAN switch that provides four external 10/100/1000BaseT interfaces for connecting to LANs external to the BladeCenter chassis. Internally the switch module is designed to connect to a single Gigabit Ethernet interface on each installed single width blade server, or two such interfaces on each installed double-width blade server.

The IGESM is also connected via a Fast Ethernet interface to each installed Management Module. This is used to remotely access the switch module via the Management Module’s 10/100BaseT Ethernet interface. You can use this capability to support initial configuration and management of the switch module.

The IGESM uses CISCO IOS. Use the IGESM when you require the advanced functionality of CISCO IOS or maximum compatibility with external networks that constructed using switches running CISCO IOS.

Note: To use the SoL remote text console function with the IGESM, you must manually create the required VLAN using the procedures in the CISCO Systems Intelligent Gigabit Ethernet Switch Module - Installation Guide.

Nortel Networks Layer 2-7 Gigabit Ethernet Switch Module The Nortel Networks Layer 2-7 Gigabit Ethernet Switch Module is a layer 2-7 LAN switch that provides four external 10/100/1000BaseT interfaces for connecting to LANs external to the BladeCenter chassis. Internally the switch module is designed to connect to a single Gigabit Ethernet interface on each installed single width blade server, or two such interfaces on each installed double-width blade server.

The Nortel Networks Layer 2-7 Gigabit Ethernet Switch Module is also connected via a Fast Ethernet interface to each installed Management Module. This is used to remotely access the switch module via the Management Module’s 10/100BaseT Ethernet interface. You can use this capability to support initial configuration and management of the switch module.

18 The IBM Eserver BladeCenter JS20 The Nortel Networks Layer 2-7 Gigabit Ethernet Switch Module provides several advanced networking features including:

Rich layer 2 and layer 3 switching capabilities Virtual server load balancing (layer 4 switching) Content aware load balancing (layer 7 switching) High availability clustering of switch modules in the same or different chassis

You can learn more about the Nortel Networks Layer 2-7 Gigabit Ethernet Switch Module in the IBM Redpaper IBM Eserver BladeCenter Layer 2-7 Network Switching, REDP-3755.

2-Port Fibre Channel Switch Module The 2-Port Fibre Channel Switch Module is a Fibre Channel SAN switch that provides two external 2 Gbps Fibre Channel interfaces for connecting to SANs external to the BladeCenter chassis. Internally the switch module is designed to connect to a single 2 Gbps Fibre Channel interface on each installed blade server that is equipped with a Fibre Channel expansion card.

The 2-Port Fibre Channel Switch Module is also connected via a Fast Ethernet interface to each installed Management Module. This is used to remotely access the switch module via the Management Module’s 10/100BaseT Ethernet interface. You can use this capability to support initial configuration and management of the switch module.

The 2-Port Fibre Channel Switch Module can be installed only in I/O module bays 3 or 4. If you install a 2-Port Fibre Channel Switch Module, any blade servers that have expansion cards must use a Fibre Channel expansion card.

Brocade Enterprise SAN Switch Module The Brocade Enterprise SAN Switch Module is a Fibre Channel SAN switch that provides two external 1 Gbps or 2 Gbps (autosense) Fibre Channel interfaces for connecting to SANs external to the BladeCenter chassis. Internally the switch module is designed to connect to a single 2 Gbps Fibre Channel interface on each installed blade server that is equipped with a Fibre Channel expansion card.

The Brocade Enterprise SAN Switch Module is also connected via a Fast Ethernet interface to each installed Management Module. This is used to remotely access the switch module via the Management Module’s 10/100BaseT Ethernet interface. You can use this capability to support initial configuration and management of the switch module.

Chapter 2. Hardware components 19 The Brocade Enterprise SAN Module can be installed only in I/O module bays 3 or 4. If you install a 2-Port Fibre Channel Switch Module, any blade servers that have expansion cards must use a Fibre Channel expansion card.

Use the Brocade Enterprise SAN Switch Module when you require maximum

compatibility with external SAN fabrics constructed using Brocade-based switches. The Brocade Enterprise SAN Switch Module is functionality equivalent to the Brocade Silkworm 3900 SAN switch.

Brocade Entry SAN Switch Module The Brocade Entry SAN Switch Module is almost identical to the Brocade Enterprise SAN Switch Module described earlier. However, it is limited to participating in SAN fabrics that contain at most two SAN switches.

Optical Pass-thru Module The Optical Pass-thru Module provides the capability to access one of the network interfaces on each blade server via an externally accessible optical interface. You can use this optical interface to connect the blade servers to external switches that support compatible optical interfaces, including: LAN switches SAN switches Myrinet switches

Each Optical Pass-thru Module has four multi-port optical connectors. The connectors are designed for special break-out cables that provide either four SC connectors (P/N 73P5992) or four LC connectors (P/N 73P6033) per multi-port optical connector. You can connect to a maximum of 14 blade server network interfaces using four of these cables (two of the connectors are unused on the fourth cable).

The type of external switch that you connect to the Optical Pass-thru Module must be compatible with the network interface on the blade servers that are connected to the I/O module bay where the Optical Pass-thru Module is installed.

If you install an Optical Pass-thru Module in I/O module bay 1 or 2, you must connect the optical interfaces to a Gigabit Ethernet switch with 1000Base-SX interfaces.

Restriction: When a BladeCenter chassis contains a BladeCenter JS20, always use a LAN switch module in I/O module bay 1 to support the SoL remote text console function. This is because there is no other mechanism for providing a console on a BladeCenter JS20.

20 The IBM Eserver BladeCenter JS20 If you install an Optical Pass-thru Module in I/O module bay 3 or 4, you must connect the optical interfaces to an external switch that is compatible with the type of expansion cards installed on blade servers.

If you install a Myrinet expansion card on any blade server, you must install an

Optical Pass-thru Module in I/O module bay 4 to enable the Myrinet interface on the expansion card to be connected to an external Myrinet switch. This is because currently no Myrinet switch modules are available for the BladeCenter.

2.1.5 Blade servers IBM offers a variety of blade servers that you can install in a BladeCenter chassis. IBM ^ BladeCenter JS20 Type 8842 Blade Server IBM ^ BladeCenter HS20 Type 8678 Blade Server IBM ^ BladeCenter HS20 Type 8832 Blade Server IBM ^ BladeCenter HS40 Type 8839 Blade Server

Because the BladeCenter JS20 is the subject of this book, we describe it in greater detail in 2.2, “BladeCenter JS20” on page 22.

The following sections briefly introduce the other blade servers. You can install any combination of blade servers in the same BladeCenter chassis subject to available power supply module capacity.

HS20 Blade Server The HS20 Blade Server is a single-width blade server that contains either one or two Intel® Xeon DP processors and up to 8 GB of memory. Up to 14 HS20 Blade Servers can be installed in each BladeCenter chassis.

The original HS20 Type 8678 Blade Server used a 400 MHz front side bus (FSB) and supports a range of processor speeds from 2.0 GHz to 2.8 GHz. The HS20 Type 8678 Blade Server is now withdrawn from marketing and has been replaced by the HS20 Type 8832 Blade Server.

The HS20 Type 8832 Blade Server uses a 533 MHz FSB and supports a range of processor speeds from 2.4 Ghz to 3.2 GHz.

HS40 Blade Server The HS40 Blade Server is a double-width blade server that contains from one to four Intel Xeon MP processors and up to 16 GB of memory. Up to seven HS40 Blade Servers can be installed in each BladeCenter chassis.

Chapter 2. Hardware components 21 2.2 BladeCenter JS20

The BladeCenter JS20 is a single-width blade server that offers these features:

Two PowerPC 970 or PowerPC 970FX microprocessors Up to 4 GB of RAM Up to two integrated development environment (IDE) hard disk drives mounted on the blade server Two integrated Gigabit Ethernet interfaces Expansion option connectors for an expansion card option to provide either: – Two additional Gigabit Ethernet interfaces – Two 2 Gbps Fibre Channel interfaces – One Myrinet interface Blade Systems Management Processor

Figure 2-6 illustrates the physical layout of the BladeCenter JS20.

I/O expansion option I/O expansion option Microprocessor 0 connector (J18) and heat sink (U87) connector (J22) Four Memory DIMMS

Secondary IDE (J2) Primary IDE (J1) Battery Microprocessor 1 (shown with attached disk) and heat sink (U41)

Figure 2-6 BladeCenter JS20 physical layout

22 The IBM Eserver BladeCenter JS20 The architecture of the BladeCenter JS20 is illustrated by the diagram in Figure 2-7.

PowerPC PowerPC 970 970 BSMP or or 970FX 970FX

6.4 GBps 6.4 GBps or or 8.8 GBps 8.8 GBps 5.3 GBps Memory Controller and Host I/O Bridge PC2700 ECC (Northbridge) Memory Modules

1.6 GBps

Chassis 1.06 GBps Midplanes Gigabit Ethernet Controller HyperTransport PCI-X Tunnel 1.06 GBps Expansion Card

800 MBps

USB HyperTransport I/O Hub Controllers

IDE Controller

Boot FLASH

Super I/O

IDE IDE Real Time NVRAM UART Disk Disk Clock

Figure 2-7 BladeCenter JS20 block diagram

The BladeCenter JS20 includes both base features and a range of optional features. We describe each of these in the sections that follow.

Chapter 2. Hardware components 23 2.2.1 Base features

There are two models of the BladeCenter JS20. Both models have the same base features with the exception of the microprocessors used in the processor subsystem. The JS20 Type 8842-21X Blade Server uses the PowerPC 970

microprocessor. The JS20 Type 8842-41X Blade Server uses the PowerPC 970FX microprocessor.

Processor subsystem The processor subsystem of the BladeCenter JS20 is comprised of either two PowerPC 970 or two PowerPC 970FX microprocessors. They operate at internal clock frequencies of either 1.6 GHz or 2.2 GHz, respectively.

Each processor includes a 32 KB level 1 data cache, a 64 KB level 1 instruction cache, and a 512 KB ECC level 2 cache.

Each processor is also interconnected with the rest of the system via a dedicated interface to the memory controller and host I/O bridge (Northbridge). The interface to each processor supports a maximum aggregate bandwidth of 6.4 GBps (for processors operating at 1.6 GHz) or 8.8 GBps (for processors operating at 2.2 GHz).

You can learn more about the features and architecture of the processors in 2.3, “PowerPC 970 and PowerPC 970FX Microprocessors” on page 29.

Memory subsystem The memory subsystem is comprised of the memory controller and up to four DDR1 333 MHz (PC2700) ECC memory modules. The memory controller uses two-way interleaving to increase the available peak memory bandwidth to 5.3 GBps (333 MHz x 16 bytes).

The BladeCenter JS20 includes two 256 MB memory modules that provide a total memory capacity of 512 MB. These memory modules are installed in slots 3 and 4. The remaining memory module slots (1 and 2) are empty in the base BladeCenter JS20.

You can expand the available memory by installing optional memory modules. The available memory modules are described in “Memory modules” on page 27.

I/O subsystem The I/O subsystem is connected to the processor and memory subsystems via a hyper transport link that supports an aggregate bandwidth of 1.6 GBps. Downstream of this link is a range of I/O devices and expansion interfaces as explained in the following sections.

24 The IBM Eserver BladeCenter JS20 Gigabit Ethernet controller Each BladeCenter JS20 includes a Broadcom BCM5704s Gigabit Ethernet controller, which provides the primary mechanism for connecting the BladeCenter JS20 to other systems. This controller provides two independent Gigabit Ethernet interfaces.

Each Gigabit Ethernet interface is connected to a separate BladeCenter I/O module bay via the BladeCenter midplanes. The first interface is connected to I/O module bay 1, and the second interface is connected to I/O module bay 2.

To use a Gigabit Ethernet interface, install a compatible I/O module in the corresponding I/O module bay. Install a LAN Switch I/O Module in I/O module bay 1 to support remote access to the BladeCenter JS20 text console via SoL.

I/O expansion option connectors Each BladeCenter JS20 contains I/O expansion option connectors, which you can use to attach one of several different expansion cards. This capability enables you to expand the external I/O connectivity of the BladeCenter JS20 to include Fibre Channel SANs, Myrinet high-bandwidth low-latency networks, and additional Gigabit Ethernet LAN interfaces.

The expansion card installed in the I/O expansion option connectors can access I/O modules installed in BladeCenter I/O module slots 3 and 4 via the BladeCenter midplanes.

Note: The main I/O expansion option connector provides a PCI-X electrical interface. However, a special form factor is required to enable installation within the confined dimensions of a blade server. Therefore, standard PCI-X adapter cards cannot be used with the BladeCenter JS20.

You can learn more about these expansion cards in “Expansion cards” on page 28.

IDE controller Each BladeCenter JS20 includes a dual-channel IDE controller and two connectors for 2.5-inch IDE hard disk drives. You can use this capability to install one or two optional 2.5-inch IDE hard disk drives on the BladeCenter JS20.

Note: You must install at least one hard disk drive for use as a boot device unless you plan to install a Fibre Channel Expansion Card and configure the BladeCenter JS20 to boot from a SAN. You can also boot from a network interface to support the network installation of an operating system.

Chapter 2. Hardware components 25 You can learn more about the available hard disk drives in “Expansion cards” on page 28.

USB controllers Each BladeCenter JS20 includes two USB 1.1 host controllers. They are connected via the BladeCenter midplanes to the USB peripherals located in the media bay of the BladeCenter chassis.

You use these controllers when accessing either the CD-ROM or diskette drive that are located in the media bay of the BladeCenter chassis. You can also access other USB devices that may be attached to the USB connector located in the media bay of the BladeCenter chassis.

Note: Verify that the USB device is supported by the operating system installed on the blade server before you attach it to the USB connector located in the media bay of the BladeCenter chassis.

You can assign only the USB peripherals to one blade server at a time. Either use a button on the front of the blade server or one of the remote interfaces to the Management Module to switch the USB peripherals to a specific blade server.

Miscellaneous I/O Each BladeCenter JS20 incorporates various miscellaneous I/O devices that are necessary to implement a functional server. These are illustrated in the BladeCenter JS20 diagram in Figure 2-7 on page 23 and include: A real-time clock (RTC) for maintaining the date and time Non-volatile memory (NVRAM) for storing configuration settings Boot FLASH memory for storing the firmware (BIOS) for the blade server A serial port (UART)

The serial port is connected within the blade server to the BSMP. The BSMP participates in the implementation of the SoL remote text console function that provides remote connectivity to the serial port via the BladeCenter Management Module.

Blade Systems Management Processor Each BladeCenter JS20 has a BSMP. The BSMP provides the interface that enables the BladeCenter Management Module to manage the BladeCenter JS20 via the BladeCenter chassis midplanes.

The BSMP is responsible for powering up the BladeCenter JS20, configuring the various hardware subsystems, monitoring temperatures, and driving the status indicators on the blade server front panel.

26 The IBM Eserver BladeCenter JS20 The BSMP also participates in the implementation of the SoL remote text console function on the BladeCenter JS20.

2.2.2 Optional features

You can install several optional features on a BladeCenter JS20.

Memory modules Each BladeCenter JS20 is equipped with 512 MB of memory. This is provided using a pair of 256 MB memory modules.

You can also install the following optional memory modules on a BladeCenter JS20: P/N 73P2275: IBM 512 MB 333 MHz PC2700 CL2.5 ECC DDR RDIMM P/N 73P2276: IBM 1 GB 333 MHz PC2700 CL2.5 ECC DDR RDIMM

You must install the optional memory module features in identical pairs. This is necessary because the memory controller on the BladeCenter JS20 uses two-way interleaving.

Two spare memory module slots (1 and 2) are available on the BladeCenter JS20. You can expand the available memory to either 1.5 GB or 2.5 GB, using either a pair of 512 MB or 1 GB memory modules respectively.

You can also remove the standard pair of 256 MB memory modules and replace them by a pair of either 512 MB or 1 GB memory modules if further memory expansion is required. To obtain the maximum supported memory configuration of 4 GB, remove both of the base 256 MB memory modules and install four 1 GB memory module features.

Hard disk drives The base BladeCenter JS20 is not equipped with any hard disk drives. You can install one or two of the 2.5-inch IDE hard disk drive option 40 GB 5400 rpm ATA-100 Hard Disk Drive (P/N 48P7063).

When you install two hard disk drives, each hard disk drive is connected to its own dedicated IDE bus.

Restriction: If you install an expansion card on a BladeCenter JS20, only one hard disk drive can be installed on the same blade server.

You do not have to install a hard disk drive if you have installed a Fibre Channel Expansion Card and have configured the BladeCenter JS20 to boot from a SAN.

Chapter 2. Hardware components 27 Expansion cards You can install any one of the following expansion cards in the I/O expansion option connectors of the BladeCenter JS20: P/N 73P9030: Gigabit Ethernet Expansion Card P/N 13N2203: Fibre Channel Expansion Card P/N 73P6000: Myrinet Cluster Expansion Card

We describe each of these expansion cards in the sections that follow.

Fibre Channel Expansion Card You can use the Fibre Channel Expansion Card to connect the BladeCenter JS20 to SANs. The Fibre Channel Expansion Card provides two 2 Gbps Fibre Channel interfaces that are connected to I/O module bays 3 and 4. You must install either a SAN switch I/O module or the Optical Pass-Thru I/O Module in one or both of these I/O module bays to connect the blade server Fibre Channel interfaces to an external SAN.

Note: There have been two different versions of the Fibre Channel Expansion Card. Both versions are functionally equivalent. The earlier version (P/N 48P7061) is now withdrawn from marketing. However, you may still find it in existing BladeCenter installations.

Gigabit Ethernet Expansion Card You can use the Gigabit Ethernet Expansion Card to increase the number of Gigabit Ethernet network interfaces on the BladeCenter JS20 from two to four.

The Gigabit Ethernet Expansion Card provides two Gigabit Ethernet interfaces that are connected to I/O module bays 3 and 4. You must install either a LAN Switch I/O Module or the Optical Pass-Thru I/O Module in one or both of these I/O module bays to connect the Gigabit Ethernet interfaces on the expansion card to an external LAN.

Myrinet Cluster Expansion Card You can use the Myrinet Cluster Expansion Card to connect the BladeCenter JS20 to a Myrinet network. Myrinet was developed by Myricom Inc., and conforms to an ANSI standard (ANSI/VITA 26-1998).

Myrinet networks are typically used to support certain types of high performance computing applications that distribute computation across a cluster of multiple

servers. Such applications are said to exploit distributed memory parallelism.

Distributed memory parallel applications often require the servers in the cluster to exchange messages with each other through some form of interconnection

28 The IBM Eserver BladeCenter JS20 network. The performance of such applications often depends on how quickly these messages can be exchanged. The highest performance is usually obtained by using a high-bandwidth, low-latency interconnection network.

Myrinet provides a high-bandwidth, low-latency interconnection network that can

be used to optimize the performance of distributed memory parallel applications. Consider using Myrinet if your application requires higher performance than is achievable using Gigabit Ethernet technology.

You can learn more about Myrinet technology at the Myrinet Web site at: http://www.myri.com

Note: The Myrinet Cluster Expansion Card is a repackaging of the Myricom M3F-PCIXD-2 PCI-X adapter to fit the special form factor required for blade server expansion cards.

When you install a Myrinet Cluster Expansion Card on a blade server, you must install an Optical Pass-thru Module in I/O module bay 4 of the BladeCenter chassis. You must also connect the external interfaces of the Optical Pass-thru Module to an external Myrinet switch via appropriate optical fiber cables.

We elaborate on the planning implications associated with using the Myrinet Cluster Expansion Card in 4.1.2, “High-performance, low-latency network requirements” on page 58.

You can order Myrinet switches from IBM as a component of the IBM ^ Cluster 1350 product. You can learn more about Cluster 1350 at: http://www.ibm.com/eserver/cluster

2.3 PowerPC 970 and PowerPC 970FX Microprocessors

This section provides more detailed information about the IBM PowerPC 970 Microprocessor and IBM PowerPC 970FX RISC Microprocessor that are used for the main processors on the BladeCenter JS20.

This section reviews the POWER™ and PowerPC Architecture™ upon which the PowerPC 970 and PowerPC 970FX microprocessors are based. It also introduces the Vector/SIMD Multimedia eXtension (VMX) to the PowerPC Architecture. And it describes the major features of the PowerPC 970 and

PowerPC 970FX microprocessors.

Chapter 2. Hardware components 29 2.3.1 Review of POWER and PowerPC Architecture

In 1990, IBM announced a new processor architecture called performance optimization with enhanced RISC (POWER). This architecture was first implemented in the IBM RS/6000®.

The POWER architecture was based on earlier IBM Research efforts to develop reduced instruction set computer (RISC) technology. These efforts began in the 1970s and continued throughout the 1980s.

RISC and CISC concepts In the 1960s and 1970s, most computers were of a design now known as a complex instruction set computer (CISC). This type of design featured a large and highly functional instruction set. Each instruction in a CISC processor design typically needs several processor clock cycles to execute. An example of a CISC design is the IBM System/360™ and its successors.

The need for complex and highly functional instructions primarily existed because computers were equipped with small quantities of slow memory. Complex instructions resulted in fewer instructions per program, so less memory was needed for program storage. Complex instructions were also easier for human programmers to use when developing applications in assembly language as single instructions could be rich in functionality.

However, studies showed that only a small percentage of CISC instructions were commonly used by programs. Also, the benefit of the CISC approach diminished over time as memory became larger and faster, and high-level languages replaced assembly language.

The RISC concept was originally developed by IBM Fellow John Cocke in the early 1970s and includes the following characteristics: A simple architecture with an optimized set of machine instructions The instructions in a RISC architecture are typically simple in function and regular in size. This simplifies the design of the instruction decode hardware. The large number of infrequently used instructions typically found in CISC architectures are eliminated and replaced by sequences of simple instructions in RISC architectures. A high instruction execution rate A common objective of early RISC processor designs was to execute an average of one instruction per processor clock cycle. This has subsequently been improved upon in superscalar RISC processor designs that can execute multiple instructions per processor clock cycle.

30 The IBM Eserver BladeCenter JS20 Optimizing compiler availability

RISC architectures assume the availability of good optimizing compilers, and that most users will be developing applications in high-level languages. This has simplified the hardware design by moving some of the complexity that was previously in the hardware into the compiler. However, the simplified instruction set of RISC architectures often makes it easier for compilers to choose optimal instruction sequences. Load and store architecture RISC architectures typically incorporate large numbers of processor registers, and computational instructions manipulate values only in these registers. Separate instructions are provided for loading data from memory and storing data to memory. Compilers can take advantage of this clean separation and schedule instructions in a way that keeps the processor busy despite the long latency associated with memory access.

Today the difference between CISC and RISC is less clear. Many CISC architectures have adopted many RISC concepts. Meanwhile, the instruction sets of many RISC architectures have grown to include hundreds of instructions.

The original POWER and POWER2™ processors The first implementation of POWER architecture released by IBM was the POWER processor as used in the first IBM RS/6000 servers and workstations. This processor was a multiple-chip implementation of the POWER architecture comprised of either seven or nine separate chips. It featured an 8 KB instruction cache and either a 32 KB or 64 KB data cache. Over the life of the POWER processor, it was shipped with clock speeds ranging from 20 MHz to 62.5 MHz.

The original POWER processor introduced the concept of a superscalar RISC processor that could perform multiple operations in a single clock cycle by exploiting multiple execution units that operated concurrently. At the time, this represented a significant improvement over contemporary processor implementations available from other manufacturers.

The floating-point performance of the POWER processor was further enhanced by the presence of the fused multiply-add (FMA) instruction. It enabled two floating-point operations to be executed by the floating-point execution unit each processor clock cycle. The peak floating-point performance of the POWER processor was therefore twice the processor clock rate. For example, a POWER processor operating at 25 MHz had a peak floating-point performance of 50 MFLOPS.

In subsequent years, IBM introduced a single chip implementation of the POWER processor called the RISC Single Chip (RSC), which was used in the IBM RS/6000® Model 220 and Model 230.

Chapter 2. Hardware components 31 In 1993, IBM announced the POWER2 processor, which expanded upon the POWER processor superscalar concept by adding additional execution units to increase the number of operations that could be performed concurrently in a single clock cycle. This processor was a multiple chip implementation of a super-set of the original POWER architecture. The POWER2 processor had two floating-point execution units, each able to execute a FMA instruction every clock cycle. Therefore, the peak floating-point performance of the POWER2 processor was four times the processor clock rate. For example, a POWER2 processor operating at 55 MHz had a peak floating-point performance of 220 MFLOPS.

The memory subsystem of the POWER2 was also enhanced in order to keep up with the increased level of processor performance. This was achieved using larger caches and wider paths to memory.

As a super-set of the POWER architecture, the POWER2 supported several instructions that are not present on any other POWER or PowerPC Architecture processors.

IBM subsequently delivered a single chip implementation of the POWER2 called the POWER2 Super Chip (P2SC), which was offered at processor clock speeds as high as 160 MHz, and delivered peak floating-point performance of 640 MFLOPS.

PowerPC Architecture In 1991, IBM teamed with Apple Computer and Motorola to define the PowerPC Architecture. The goals of this new architecture were to: Permit a broad range of implementations, from low-cost controllers to high-performance processors Be sufficiently simple to permit the design of processors that have a very short cycle time Minimize the effects that hinder the design of aggressive superscalar implementations. Include multiprocessor features. Define a 64-bit architecture that is a super-set of the 32-bit POWER architecture, providing application binary compatibility for 32-bit applications

While based on the POWER architecture, the PowerPC Architecture incorporated several modifications to enable it to be more widely applied in a variety of application scenarios. This vision has been subsequently realized with processors implementing PowerPC Architecture now having been installed in

32 The IBM Eserver BladeCenter JS20 desktop, server, and embedded systems across commercial, consumer, industrial, and scientific settings.

To achieve the design goals for the PowerPC Architecture, some features of the original POWER architecture were removed. These were mostly features that were infrequently used. None of the POWER2 extensions to the original POWER architecture were included in the PowerPC Architecture.

The PowerPC Architecture defines both 32-bit and 64-bit modes of operation. The primary differences in these two modes of operation are in the effective length of addresses used by the processor, and the availability of extra capabilities to manipulate double word (64-bit) fixed-point operands in 64-bit mode. Floating-point capabilities are the same in both 32-bit and 64-bit modes.

The 32-bit PowerPC Architecture implementations only support the 32-bit mode of operation. The 64-bit PowerPC Architecture implementations support both the 32-bit and 64-bit modes of operation. This enables the 64-bit PowerPC Architecture implementations to support the full-speed execution of existing 32-bit applications, alongside 64-bit applications, in the same operating environment.

The PowerPC Architecture is documented in a series of books. You can download the most current version of the PowerPC Architecture books from the IBM developerWorks® Web site at: http://www.ibm.com/developerworks/eserver/articles/archguide.html

The architecture is divided into three parts, each one documented in a separate book: Book I: PowerPC User Instruction Set Architecture Book II: PowerPC Virtual Environment Architecture Book III: PowerPC Operating Environment Architecture

Book IV is also available for each specific PowerPC processor implementation. It describes extensions to the architecture that are specific to a particular implementation.

We briefly review the PowerPC User Instruction Set Architecture, since this is the part of the architecture that is of most interest to application programmers.

PowerPC User Instruction Set Architecture The logical components of a PowerPC processor from the perspective of a application program are: Fixed-point processor (FXU) Floating-point processor (FPU) Branch processor (BPU)

Chapter 2. Hardware components 33 These components all interact with storage as illustrated in Figure 2-8.

Instructions from Storage Branch Processing (BPU)

Fixed-Point Floating-Point Instructions Instructions Floating- Fixed-Point Point Processing Processing (FXU) (FPU) Data to/from Storage

Storage

Figure 2-8 PowerPC logical processing model

A PowerPC Architecture groups the instructions that an application program uses on a PowerPC processor along the same lines into: Fixed-point instructions Floating-point instructions Branch instructions

All instructions in PowerPC Architecture are four bytes (one word) in size and are aligned on word boundaries. This simplifies the decoding of instructions and is characteristic of a RISC architecture.

The fixed-point instructions operate on a set of 32 general purpose registers (GPRs), that are each 8 bytes (64 bits) in size on 64-bit PowerPC Architecture processors. There is also a fixed-point exception register (XER). In 32-bit mode, only the lower 32 bits of the GPRs are considered significant, and the double word load and store instructions are not available.

The floating-point instructions operate on a set of 32 floating-point registers (FPRs), that are each 8 bytes (64 bits) in size. There is also a floating-point

status and control register (FPSCR). The floating point capabilities are the same in both 32-bit and 64-bit mode. The large number of GPRs and FPRs is also typical of a RISC architecture.

34 The IBM Eserver BladeCenter JS20 The architecture also features a condition register that is set by the result of fixed or floating point computational instructions and that can be used to control branch processing. Branch instructions also make use of a count and a link register.

Figure 2-9 illustrates the complete set of architected PowerPC registers used by application programs.

CR Condition Register 031

LR Link Register 063

CTR Count Register 063

GPR0

GPR1 General Purpose Registers ...

GPR31 063

XER Fixed-Point Exception Register 063

FPR0

FPR1 Floating-Point Registers ...

FPR31 063

Floating-Point Status FPSCR and Control Register 031 Figure 2-9 PowerPC registers used by application programs

In the PowerPC Architecture, fixed and floating-point computational instructions can operate on operands stored in registers. Data must be loaded from memory into a register using a fixed or floating-point load instruction before computations can be performed. Similarly, data must be stored from a register into memory using an explicit fixed or floating-point store instruction once computation is completed. This behavior is also characteristic of a RISC architecture.

Chapter 2. Hardware components 35 IBM processors implementing PowerPC Architecture Since IBM teamed with Motorola and Apple Computer to define PowerPC Architecture, IBM has developed many different processors that implement this architecture. These processors fall into two main categories: Processors that are used as the main processor within IBM workstations and servers Processors that IBM produces for embedding in a wide variety of computing and non-computing devices

We focus our review of IBM processors implementing PowerPC Architecture on those processors that are used as the main processors within IBM workstations and servers.

32-bit PowerPC processors In the early 1990s, IBM jointly designed with Motorola a series of 32-bit PowerPC processors with the designation 60X that were used in IBM RS/6000 workstations and servers. They were also used in computer systems from other computer manufacturers such as Apple Computer.

The first of these was the PowerPC 601®. It debuted in the IBM RS/6000 Model 250 in 1993. The PowerPC 601 was an evolution of the RISC Single Chip implementation of the POWER processor and did not implement the full PowerPC Architecture. It represented a transition between the earlier POWER architecture and the PowerPC Architecture.

The PowerPC 601 was subsequently followed by the PowerPC 603™ and PowerPC 604™. These represented full implementations of the 32-bit PowerPC Architecture. Enhanced versions of these processors known as the PowerPC 603e and PowerPC 604e were subsequently introduced.

The PowerPC 603 was an entry-level processor. The PowerPC 604 was a high-end processor and introduced support for symmetrical multi-processor (SMP) configurations.

RS64 processors In 1997, IBM introduced its first servers that incorporated processors based on the 64-bit PowerPC Architecture. These processors were called the RS64. The RS64 was subsequently enhanced with the introduction of the RS64-II, RS64-III, and RS64-IV in later years. The RS64-IV operated at processor clock speeds as high as 750 MHz. The RS64 processor and its successors were optimized for use in commercial applications and de-emphasized the floating point capabilities of the earlier POWER2 processors. The RS64 processors were used in both IBM RS/6000

36 The IBM Eserver BladeCenter JS20 (later IBM Eserver pSeries®) and IBM AS/400® (later IBM Eserver iSeries™) servers, most frequently in SMP configurations, with some as large as 24 processors.

POWER3™ and POWER3-II processors The POWER3 processor was first used in IBM RS/6000 servers and workstations introduced in 1998. The POWER3 processor continued the evolution of the excellent floating-point performance provided by the POWER2 Super Chip (P2SC) processor, while implementing 64-bit PowerPC Architecture and supporting SMP configurations of up to 16 processors.

Upon introduction, the POWER3 processor operated at 200 MHz, and delivered peak floating-point performance of 800 MFLOPS per processor. An improved version of the POWER3 processor, known as the POWER3-II, was subsequently introduced. The POWER3-II featured processor clock speeds as high as 450 MHz, and delivered peak floating point performance of 1.8 GFLOPS per processor.

You can learn more about the POWER3 processor in the IBM white paper POWER3: Next Generation 64-bit PowerPC Processor Design, which is available on the Web at: http://www.ibm.com/servers/eserver/pseries/hardware/whitepapers/power3wp.html

POWER4™ and POWER4+™ processors The POWER4 processor combined the best attributes of both the RS64 and POWER3 processors. It delivers a 64-bit PowerPC Architecture processor that is capable of delivering high levels of performance in both commercial and scientific applications.

The POWER4 processor is the first PowerPC Architecture processor to feature multiple processor cores on a single chip. Each POWER4 processor chip has two independent processor cores and a shared level-2 cache.

The original POWER4 processor operated at processor clock speeds as high as 1.3 GHz and has featured in SMP configurations as large as 32 processor cores.

An enhanced version of the POWER4 processor, known as the POWER4+, was subsequently introduced and installed at processor clock speeds as high as 1.9 GHz in the 32 processor core IBM Eserver pSeries Model 690.

You can learn more about the POWER4 processor in the IBM white paper entitled POWER4 System Microarchitecture, which is available on the Web at: http://www.ibm.com/servers/eserver/pseries/hardware/whitepapers/power4.html

Chapter 2. Hardware components 37 PowerPC 970 and PowerPC 970FX processors These processors were first used by IBM in the BladeCenter JS20 that is the subject of this book. We explore these processors in more detail in 2.3, “PowerPC 970 and PowerPC 970FX Microprocessors” on page 29.

POWER5™ processor IBM recently introduced the POWER5 processor in the IBM Eserver i5 and p5 families of servers.

The POWER5 processor represents an evolution of the POWER4 processor. Like the POWER4, the POWER5 processor implements two processor cores on a single chip with a shared level-2 cache.

The major innovation in the POWER5 processor is the introduction of symmetrical multi-threading (SMT) capabilities. SMT enables two threads of execution (contexts) to execute simultaneously on the same processor core. This can improve the performance of many applications through the exploitation of otherwise idle execution units within the processor core.

The other significant innovation in the POWER5 processor design is the integration of a memory controller onto the processor chip to decrease the latency to memory and improve system reliability.

2.3.2 Vector/SIMD Multimedia eXtension The Vector/SIMD Multimedia eXtension (VMX) is an extension to the PowerPC Architecture. It defines additional registers and instructions to support single-instruction multiple-data (SIMD) operations that accelerate data-intensive tasks.

The VMX extensions to the PowerPC Architecture were developed jointly by Apple Computer, IBM, and Motorola. Apple Computer and Motorola use different terminology to refer to the VMX extensions of the PowerPC Architecture. Specifically, Motorola uses the term Altivec, and Apple uses the term Velocity Engine.

A short vector processing history The basic concept behind vector processing is to enhance the performance of data-intensive applications by providing hardware support for operations that can manipulate an entire vector (or array) of data in a single operation. The number of data elements operated upon at a time is called the vector length.

38 The IBM Eserver BladeCenter JS20 Scalar processors perform operations that manipulate single data elements such as fixed-point or floating-point numbers. For example, scalar processors usually have an instruction that adds two integers to produce a single-integer result.

Vector processors perform operations on multiple data elements arranged in groups called vectors (or arrays). For example, a vector add operation to add two vectors performs a pairwise addition of each element of one source vector with the corresponding element of the other source vector. It places the result in the corresponding element of the destination vector. Typically a single vector operation on vectors of length n is equivalent to performing n scalar operations.

Figure 2-10 illustrates the difference between scalar and vector operations.

Vector Add Operation

7 5 12 Scalar Add Operation 1 6 7 6 11 17 4 + 7 11 + 3 4 7 3 5 8 10 2 12

Figure 2-10 Scalar and vector operations

Processor designers are continually looking for ways to improve application performance. The addition of vector operations to a processor architecture is one method that a processor designer can use to make it easier to improve the peak performance of a processor. However, the actual performance improvements that can be obtained for a specific application depend on how well the application can exploit vector operations.

The concept of vector processing has existed since the 1950s. Early implementations of vector processing (known as array processing) were installed in the 1960s. They used special purpose peripherals attached to general purpose computers. An example is the IBM 2938 Array Processor, which could be attached to some models of the IBM System/360. This was followed by the IBM 3838 Array Processor in later years.

By the mid-1970s, vector processing became an integral part of the main processor in large supercomputers manufactured by companies such as Cray Research. By the mid-1980s, vector processing became available as an optional feature on large general-purpose computers such as the IBM 3090™.

In the 1990s, developers of microprocessors used in desktop computers adapted the concept of vector processing to enhance the capability of their microprocessors when running desktop multimedia applications. These

Chapter 2. Hardware components 39 capabilities were usually referred to as Single Instruction Multiple Data (SIMD) extensions and operated on short vectors. Examples of SIMD extensions in widespread use today include:

Intel Multimedia Extensions (MMX™) Intel Streaming SIMD Extensions (SSE) AMD 3DNow! Motorola Altivec and IBM VMX

The SIMD extensions found in microprocessors used in desktop computers operate on short vectors of length 2, 4, 8, or 16. This is in contrast to the classic vector supercomputers that can often exploit long vectors of length 64 or more.

VMX extensions to PowerPC Architecture The VMX extensions to PowerPC Architecture add a vector processor (VXU) to the PowerPC logical processing model that was described earlier in “PowerPC User Instruction Set Architecture” on page 33. This is illustrated in Figure 2-11.

Instructions Branch from Storage Processing (BPU)

Fixed-Point Floating-Point Vector Instructions Instructions Instructions Floating- Fixed-Point Vector Point Processing Processing Processing (FXU) (VXU) (FPU)

Data to/from Storage

Storage

Figure 2-11 PowerPC with VMX logical processing model

The VXU operates on vectors that are a total of 128-bits long. These can be interpreted by the VXU as either: A vector of sixteen 8-bit bytes A vector of eight 16-bit half words A vector of four 32-bit words

40 The IBM Eserver BladeCenter JS20 The VMX extensions to PowerPC Architecture define 32 vector registers that form the vector register file (VRF). The VRF is architecturally distinct from the standard PowerPC FPRs and GPRs.

The VMX extensions to PowerPC also define two additional registers:

The VMX status and control register (VSCR), which is used to control the operation of the VXU and report the status of some VMX operations The VRSAVE register, which can be used to assist the operating system save state across context switches by providing a mechanism for software to indicate what vector registers are in use

The additional registers provided by VMX are illustrated in Figure 2-12.

VRSAVE VRSAVE Register 031

VSCR Vector Status and Control Register 031

Vector Register File (VRF)

VR0

VR1

...

VR31 0 127

Figure 2-12 VMX registers

The VMX extensions to PowerPC Architecture define new instructions that use the VXU to manipulate vectors stored in the VRF. These instructions fall into these categories: Vector integer arithmetic instructions (on 8-bit, 16-bit, or 32-bit integers) Vector floating-point arithmetic instructions (32-bit only) Vector load and store instructions Vector permutation and formatting instructions

Processor control instructions used to read and write from the VMX status and control register Memory control instructions used to manage caches

Chapter 2. Hardware components 41 For additional information about the topics presented in this chapter, we recommend that you read the PowerPC Microprocessor Family: AltiVec Technology Programming Environments Manual. You can find this manual on the Web at: http://www-306.ibm.com/chips/techlib/techlib.nsf/techdocs/FBFA164F824370F987256 D6A006F424D

2.3.3 Features of the PowerPC 970 and PowerPC 970FX The IBM PowerPC 970 Microprocessor is a recent implementation of 64-bit PowerPC Architecture designed for use in workstations and small servers requiring from one to four processors. The PowerPC 970 is based on the processor core used in the POWER4 with the addition of a VXU to support the VMX extensions to the PowerPC Architecture.

The IBM PowerPC 970FX RISC Microprocessor is an enhanced version of the PowerPC 970 that takes advantage of improvements in semiconductor manufacturing processes to deliver higher performance and lower power consumption. In most respects, the PowerPC 970FX is identical to the PowerPC 970.

Table 2-3 lists the physical attributes of the PowerPC 970 and PowerPC 970FX.

Table 2-3 Physical attributes of PowerPC 970 and PowerPC 970FX Physical attribute PowerPC 970 PowerPC 970FX

Process geometry 130 nm 90 nm

Physical size 125 mm2 66 mm2

Transistor count 58 million 58 million

Package pins 576 576

Unless otherwise noted, the following discussion applies to both the PowerPC 970 and PowerPC 970FX. References to PowerPC 970 also refer to the PowerPC 970FX.

Figure 2-13 illustrates the internal structure of the PowerPC 970.

42 The IBM Eserver BladeCenter JS20

Figure 2-13 PowerPC 970 internal organization

Chapter 2. Hardware components 43 The major features of the PowerPC 970 include:

Speculative, superscalar RISC processor core with vector extensions On-chip level-1 and level-2 caches Deeply pipelined design up to 25 stages long Aggressive branch prediction In order dispatch of up to five operations per cycle Out of order issue of up to ten operations per cycle In order completion of up to five operations per cycle Register renaming Up to 215 instructions in-flight Fast selective flush of incorrect speculative instructions and results Hardware prefetching of up to eight streams including four VMX streams

We elaborate on some of these features in the following sections.

Cache structure The PowerPC 970 includes several on-chip caches to reduce memory latency when fetching instructions and performing data load and store operations. Understanding the structure of these caches is sometimes useful when doing low-level assembly language programming.

The on-chip caches include: 64 KB, direct-mapped instruction cache 32 KB, 2-way set associative data cache 128-entry, instruction effective to real address translation (ERAT) cache instructions 128-entry, data ERAT cache 64-entry, fully associative segment lookaside buffer (SLB) 1024-entry, 4-way set associative translation lookaside buffer (TLB) 512 KB, 8-way set associative, level-2 cache

Speculative superscalar features The design of the PowerPC 970 uses a variety of techniques to enable superscalar operation, where multiple instructions are executed during each processor clock cycle. This capability is enabled by the exploitation of multiple execution units within the processor core.

44 The IBM Eserver BladeCenter JS20 The speculative superscalar techniques used in the PowerPC 970 include:

Aggressive branch prediction – Prediction for up to two branches per cycle – Support for up to 16 predicted branches in flight

– Prediction support for both branch direction and branch addresses In order dispatch of up to five operations into a distributed issue queue structure per cycle Out of order issue of up to ten operations into ten execution pipelines per cycle, comprised of: – Two load or store operations – Two fixed-point register-to-register operations – Two floating-point operations – One branch operation – One condition register operation – One VMX permute operation – One VMX arithmetic operation Register renaming on most architected PowerPC registers including the GPRs, FPRs, VRFs, condition register fields, FPSCR, VSCR, the link register (LR), and the count register (CTR)

The total number of in-flight instructions within the processor core at any point in time can be as high as 215.

The VMX execution units operate on multiple data elements concurrently. Therefore, they provide a performance equivalent to multiple scalar execution units.

Bus Interface Unit The interface between the PowerPC 970 and external devices is provided by the Bus Interface Unit (BIU). The interface is comprised of two unidirectional paths (one in, one out) that are each four bytes wide.

The internal processor clock operates at an integral multiple of the interface speed. On the BladeCenter JS20, the processor clock operates at double the interface speed.

The total aggregate transfer rate across the interface in bytes per second is four times of the processor clock speed (1/2 x 2 x 4). This equates to 6.4 GBps for processors operating at 1.6 GHz, and 8.8 GBps for processors operating at 2.2 GHz.

The interface is multiplexed, with both addresses and data being transferred across the same wires. Therefore the peak theoretical data throughput across

Chapter 2. Hardware components 45 the bus interface is slightly lower than the total aggregate bus transfer rate derived above.

For more detailed information about the PowerPC 970 and PowerPC 970FX microprocessors, refer to the IBM Microelectronics Division Web site at:

http://www.chips.ibm.com

To access the documentation, follow these steps: 1. Point your Internet browser to: http://www.chips.ibm.com 2. Click Support. 3. Click Documentation. 4. Under Browse a product family, click PowerPC. 5. Under Categories, click PowerPC 9XX Microprocessors. 6. Under Categories, click PowerPC 970 and 970FX Microprocessors.

Various documents describe the PowerPC Architecture, programming, and 64-bit computing.

46 The IBM Eserver BladeCenter JS20

Chapter 3. Software environment

This chapter discusses the software environments that are available for the IBM Eserver BladeCenter JS20. It covers the operating systems and IBM middleware supported on the BladeCenter JS20, as well as the system management tools for the JS20.

The BladeCenter JS20 is supported by the AIX and Linux operating systems. At the time of this writing, AIX 5.2H, AIX 5.3, Red Hat Enterprise Linux 3.0, and SUSE LINUX Enterprise Server (SLES) are the supported operating systems on this platform.

Both AIX and Linux operating systems work with a 64-bit kernel and mostly 32-bit userland programs. Some of the included programs were compiled for 64 bit when the code fully exploited the 64-bit address space.

3.1.1 AIX AIX has evolved from its beginnings on the IBM RT to becoming the operating system of choice for IBM’s largest UNIX servers. AIX is an enterprise operating system that scales from workstations all the way up to massively parallel super computers.

Certified as C2 security compliant by the US government, AIX also supports industry-standard security features such as Pluggable Authentication Modules (PAM), authenication by x.509 certificates, OpenSSH, Kerberos, and LDAP. AIX fully exploits the features of the JS20 with its 64-bit kernel and support for over 16 TB of disk space.

With AIX installed, the JS20 can run thousands of software titles that were written for the AIX platform, taking full advantage of AIX’s rich capabilities. At the time of the writing of this book, AIX Version 5.2H and AIX 5.3 support the BladeCenter JS20.

3.1.2 Red Hat Enterprise Linux Red Hat has made its distribution of Linux available since 1994. Originally, Red Hat offered its distribution for free download, and sold support contracts. In 2002, Red Hat began marketing Red Hat Enterprise Linux (RHEL).

Unlike the original Red Hat distribution, RHEL was available only with a maintenance contract from Red Hat. RHEL runs on x86_64, i386, ia64, ppc/64, s390, and s390x platforms.

By early 2003, Red Hat stopped developing its non-enterprise distribution and focused its efforts on RHEL. Red Hat elected to turn development of its free software over to the community. This free community-supported distribution became known as Fedora.

48 The IBM Eserver BladeCenter JS20 Red Hat Enterprise Linux 3 - Update 2 and later is supported on the JS20. Although VMX exploitation was not supported at the time of this book’s development in the version of GNU C Compiler (gcc) that ships with RHEL 3, kernel support is available for programs that have been written to exploit VMX instructions.

3.1.3 SUSE LINUX Enterprise Server SuSE has been a Linux distributor since 1992 and was acquired by Novell in early 2004. Originally, SuSE provided a free downloadable distribution, for which support contracts could be purchased as desired. Eventually, their product line was split. SuSE now markets SUSE LINUX Desktop, as well as SUSE LINUX Enterprise Server.

SUSE LINUX Desktop is free to download, where SLES is available only with the purchase of a maintenance contract. The desktop product is also available only for x86_64 and IA32, where SLES supports IA32, ia64, ppc/64, s390, s390x, and x86_64.

SLES Version 8 and Version 9 are supported on the BladeCenter JS20. SLES 9, the latest version at the time of writing, ships with the 2.6 Linux kernel. This brings significant gains to performance and scalability.

Both versions of SLES are VMX savvy and have versions of gcc that allow manual exploitation of the vector engine.

3.2 System management tools

IBM has devised a strategy for systems management called the Universal Manageability Initiative. There are several technologies that have been incorporated under this umbrella. They all work toward the same goal: simple, effective solutions for managing heterogeneous environments.

One of the main building blocks in this strategy is the use of simple yet powerful software to oversee the management of many different systems. This section investigates some of the possible options that are available for managing such systems. It reviews the BladeCenter Web interfaces, IBM Director, Cluster Systems Management, and xCAT.

3.2.1 BladeCenter Web interfaces The BladeCenter Web interface allows system administrators to easily and effectively manage up to 14 blades from an integrated interface. From trivial tasks such and powering blades on or off, to more complex tasks such as firmware

Chapter 3. Software environment 49 management, the Web interface allows powerful control over all blades and input/output (I/O) modules that are attached to the BladeCenter chassis.

Management of other BladeCenter resources, such as I/O modules, is also possible from here, as well as the retrieval of system health information.

BladeCenter-specific features are also configured from here such as the Serial over LAN (SoL).

3.2.2 IBM Director IBM Director assists with the remote management of many IBM and non-IBM machines including the BladeCenter JS20. The IBM Director console allows system administrators to manage multiple BladeCenter chassis from a common interface. It is the ideal solution for administering BladeCenter chassis in heterogeneous environments or environments where a Director infrastructure exists.

From the IBM Director console, system administrators can monitor resource utilization. This key feature can be used for performance and capacity planning, as well as to alert support staff of critical errors such as hardware failure.

IBM Director also allows the remote management of software. You can remotely reconcile and install software from the console interface. This can be useful in large environments for such resource-intensive tasks as patch management. By using the Software Distribution Premium Edition, you can remotely install patches to several servers at the same time with the click of a button.

You can learn more about IBM Director in 8.1, “IBM Director” on page 140.

3.2.3 IBM Cluster Systems Management IBM Cluster Systems Management (CSM) provides many useful functions to manage a cluster from a single point of control. These include resource monitoring, automated monitoring and operation, remote hardware control, remote command execution, security, configuration file management, parallel network installation, and diagnostics.

By consolidating these capabilities, CSM helps to increase efficiency of an administrator’s time and reduce their expenses. It helps administrators install their clusters rapidly by automating many configuration tasks and by leveraging existing open source products. CSM also provides efficient monitoring of cluster resources without overwhelming network bandwidth. The automated error detection CSM provides helps catch problems before they impact the environment, and assists with rapid

50 The IBM Eserver BladeCenter JS20 resolution and recovery of problems that occur. CSM’s architecture and modular construction maximizes flexibility so cluster solutions can evolve and grow as needs change.

CSM is a collection of components that have been integrated to provide the basis to construct and manage a cluster. Each of these components provides specific capabilities related to the management of the cluster. This component-based architecture provides flexibility for future expansion of the capabilities provided by CSM. Each of the CSM components can be easily personalized to help meet specific needs. For example, a cluster administrator can set up monitoring of application processes and take actions if those processes disappear.

In short, CSM has been designed for large-scale cluster environments that require simple control over multiple homonymous servers.

This book covers CSM in more detail in 8.2, “IBM Cluster Systems Management” on page 147.

Chapter 3. Software environment 51

52 The IBM Eserver BladeCenter JS20

Chapter 4. Planning considerations

This chapter discusses planning considerations associated with the implementation of the IBM Eserver BladeCenter JS20. Specifically it covers network planning, operating system installation, and systems management.

Successful installation of the BladeCenter JS20 requires that you have a clear plan for how you will use the various networking capabilities of the BladeCenter infrastructure. This plan should address the following questions: What network connectivity is needed for the blade servers to support the applications installed on them? What network connectivity is needed to manage the BladeCenter, input/output (I/O) modules, and blade servers? What Virtual local area networks (VLANs) are required for each LAN Switch I/O Module to provide the network connectivity established previously? What IP subnet will be used for each VLAN and how will IP addresses be allocated to each device on the VLAN? Will IP addresses be assigned statically or dynamically using DHCP? What host names will be used for each network interface? How will host names be resolved to IP addresses? Are there requirements for a high-performance, low-latency interconnection network? Where multiple BladeCenter chassis are installed, how will they be interconnected?

The following sections explore common network planning requirements that we expect to arise in the installation of BladeCenter JS20s.

4.1.1 Minimal network requirements At a minimum, most BladeCenter JS20 environments have the following network requirements: A dedicated hardware management subnet: It is used to access both the Management Module and management interfaces of I/O modules that are installed in each BladeCenter chassis. A Serial over LAN (SoL) subnet internal to each BladeCenter chassis that supports the SoL remote text console function: This is always implemented using a LAN Switch I/O Module installed in I/O module bay 1. A subnet connected to each BladeCenter JS20: It is used to install and

manage the operating system on the blade server. Where you run AIX on some blade servers, and Linux on other blade servers, you may need to use multiple subnets for this purpose.

54 The IBM Eserver BladeCenter JS20 One or more subnets connected to each BladeCenter JS20: They are used by applications installed on the blade server to communicate other systems.

Figure 4-1 illustrates how these requirements can be provided in a logical network view.

Hardware Management Subnet

I/O ModulesI/O Management ModuleI/O Module ModuleI/O Module

SOL Subnet

Console Console Console Console Console Console

JS20 JS20 JS20 JS20 JS20 JS20 Blade Blade Blade Blade Blade Blade Server Server Server Server Server Server

eth0 eth1 eth0 eth1 eth0 eth1 eth0 eth1 eth0 eth1 eth0 eth1

OS Application Management Subnet Subnet

Figure 4-1 Network logical view

The sections that follow describe each of the logical networks illustrated in Figure 4-1.

Chapter 4. Planning considerations 55 Hardware management subnet We recommend that you establish a dedicated hardware management subnet. The 10/100BaseT Ethernet interface on the Management Modules installed in each BladeCenter chassis provides the gateway to this subnet for each BladeCenter chassis. You need to install external LAN switches or hubs to interconnect the 10/100BaseT Ethernet interface on the Management Modules of each BladeCenter chassis with external management systems.

You use this subnet to access the Management Module Web interface and command line interface (CLI). You also use this subnet to access the Web interface and CLI of I/O modules.

System management applications such as IBM Director or IBM Cluster Systems Management (CSM) also use this subnet to communicate with the hardware management functions of the BladeCenter infrastructure.

Restrict access to this subnet to those management systems, system administrators, and operators who have responsibility for managing the BladeCenter infrastructure.

You need to allocate multiple IP addresses to each BladeCenter chassis on this subnet, including: One IP address for the external interface of the Management Module in each BladeCenter chassis One IP address for the internal interface of the Management Module in each BladeCenter chassis One IP address for the management interface of each I/O module in each BladeCenter chassis

Note: Although the logical network view illustrated in Figure 4-1 shows the I/O Management Module interfaces connecting directly to the hardware management subnet, they are physically connected via the Management Module, which acts as gateway to those interfaces. The Management Module performs a proxy Address Resolution Protocol (ARP) function to make it appear as though the I/O module management interfaces are attached to the hardware management subnet.

It is possible to use a different subnet for the Management Module internal network interface and I/O module management interfaces. However, we do not recommend this configuration.

Learn how to configure these IP addresses in Chapter 5, “Hardware setup” on page 69.

56 The IBM Eserver BladeCenter JS20 Serial over LAN subnet The SoL remote text console function requires a subnet and underlying VLAN that is implemented by a LAN Switch I/O Module installed in I/O module bay 1 of the BladeCenter chassis (see Figure 2-2 on page 9). This subnet and VLAN is entirely internal to each BladeCenter chassis and should not be externally accessible.

If you use the 4-Port Gigabit Ethernet Switch Module or the Nortel Networks Layer 2-7 Gigabit Ethernet Switch Module, the VLAN uses VLAN ID 4095. If you use the Cisco Systems Intelligent Gigabit Ethernet Switch Module, you can choose the VLAN ID.

You need to assign a unique range of IP addresses to this subnet for use by the SoL remote text console function.

Important: One IP address is required for each blade server.

You need to specify only the starting IP address within the range of IP addresses that you assign into the Management Module. The Management Module then automatically assigns consecutive IP addresses from the starting address that you provide to each blade server that you have installed.

You can learn how to configure the SoL subnet and VLAN in 5.5.1, “Configuring Serial over LAN” on page 89.

Operating system management subnet We expect most environments that use the BladeCenter JS20 to rely on the network installation procedure to install the operating systems. We discuss this further in 4.2, “Operating system installation” on page 60.

The operating system management subnet is used to support both the initial installation and subsequent management of the operating systems installed on BladeCenter JS20s. This subnet is implemented using a VLAN provided by the LAN Switch I/O Modules installed in I/O module bay 2 of each BladeCenter chassis. Alternatively you can implement it using a Pass-Thru I/O Module installed in I/O module bay 2 that is connected to an external LAN switch.

You may want to install both AIX and Linux operating systems on different BladeCenter JS20s in the same environment. In this case, you may need to set up multiple operating system management subnets and underlying VLANs, one for blade servers running AIX and the other for blade servers running Linux.

Chapter 4. Planning considerations 57 Application subnet The primary reason you install BladeCenter JS20s is to support applications. Many applications have requirements to communicate with other systems. You use one or more application subnets for this purpose.

The primary application subnet is implemented using a VLAN provided by the LAN Switch I/O Modules installed in I/O module bay 1 of each BladeCenter chassis. The same LAN Switch I/O Module is used to support the SoL subnet and VLAN.

Where different BladeCenter JS20s participate in different applications and there is a requirement to separate application traffic, you may need to define multiple application subnets and VLANs. Each BladeCenter JS20 is connected to the appropriate application subnet based on the applications that are installed on the blade server.

If application communication requirements with other systems are complex, you can install an additional pair of Gigabit Ethernet interfaces on each BladeCenter JS20. You do this using the Gigabit Ethernet Expansion Card in conjunction with compatible I/O modules installed in I/O module bays 3 and 4.

4.1.2 High-performance, low-latency network requirements Distributed memory parallel applications may require the installation of a high-performance, low-latency interconnection network between BladeCenter JS20s. This requirement can be supported through the use of an optional Myrinet network.

To install a Myrinet network, you must install a Myrinet Expansion Card on each BladeCenter JS20 that requires connectivity to the high-performance, low-latency interconnection network. You must also install an Optical Pass-Thru I/O Module in I/O module bay 4 of each BladeCenter chassis that contains blade servers equipped with Myrinet Expansion Cards. Then connect the Optical Pass-Thru I/O Module to external Myrinet switches to complete the Myrinet network infrastructure. This is illustrated in Figure 4-2.

58 The IBM Eserver BladeCenter JS20

Blade Blade Blade Blade Blade Blade Server Server Server Server Server Server ... Myrinet Myrinet Myrinet Myrinet Myrinet Myrinet Expansion Expansion Expansion Expansion Expansion Expansion Card Card Card Card Card Card

Optical Pass-Thru Module I/O Module Bay 4

Break-out Cables

External Myrinet Switch

Figure 4-2 Myrinet network infrastructure

The Myrinet network infrastructure can be used to support application programming interfaces such as the message passing interface (MPI) that are often used by distributed memory parallel applications.

You can define IP addresses for each Myrinet network interface. You can also use the Myrinet network infrastructure to support any application communication based on IP protocols. For example, you can use this capability to support a clustered file system such as IBM General Parallel File System (GPFS).

You can define a dedicated IP subnet for use with the Myrinet network infrastructure that is distinct from the other IP subnets that are identified in 4.1.1, “Minimal network requirements” on page 54.

4.1.3 Multiple BladeCenter environments

Large configurations of BladeCenter JS20s require the installation of multiple BladeCenter chassis. To support the network requirements identified previously, you need to install external LAN switches to interconnect the BladeCenter chassis with each other.

Chapter 4. Planning considerations 59 4.2 Operating system installation

There are two methods to install the JS20 in a BladeCenter. The first is an

attended method, using CD media. The second is a remote method through a network installation. Using the CD installation media is similar to other pSeries systems. You assign the CD-ROM to a blade, follow the installation steps answering any prompts, and then repeat the process on the next blade.

Refer to the following Web site for instructions to install SuSE from CDs: http://www.ibm.com/pc/support/site.wss/document.do?sitestyle=ibm&lndocid= MIGR-54819

With network installation, you can perform several installations at the same time. The method is designed to reduce the installation time required when a large number of blades requires operating system installation. This method was chosen as the focus for the following sections.

4.2.1 Network installation planning You can use two approaches to set up a network installation environment for the BladeCenter JS20: Establish one or more network installation servers and manually initiate network installation tasks. Use a systems management tool, such as IBM Cluster Systems Management or xCAT, which can be used to automate much of the setup of network installation servers and initiation of network installation tasks.

This book provides examples of both approaches. The remainder of this section focuses on the planning considerations for the first approach. You can find planning considerations for the second approach in 4.3, “Systems management” on page 63.

Required network services Network installation depends on several different network services: A bootstrap protocol (BOOTP) server or dynamic host configuration protocol (DHCP) server A Trivial File Transfer Protocol (TFTP) server

A Network File System (NFS) server You can use one physical server to provide all three of these network services. If you want to install AIX on BladeCenter JS20s, this server must run AIX. If you want to install Linux on the BladeCenter JS20s, we recommend that you use the

60 The IBM Eserver BladeCenter JS20 same Linux distribution in the network installation server that you are planning to install on the BladeCenter JS20s.

In some situations, you may want to install AIX on some BladeCenter JS20s and Linux on other BladeCenter JS20s. While it is possible to use a single AIX server to do this, we recommend that you establish two network install servers, one for installing AIX and the other for installing your chosen Linux distribution.

Important: A potential problem may arise when you have multiple servers acting as BOOTP or DHCP servers. Be careful when planning your network so that they do not interfere with one another.

One approach is to place the BladeCenter JS20s running AIX and the AIX network install server on a different VLAN from the BladeCenter JS20s running Linux and the Linux install server. If you use this approach, you must also disable the relay of BOOTP or DHCP requests in your network routers.

To learn how to set up AIX network install servers, see Chapter 7, “Installing AIX on the JS20” on page 123. Or to set up Linux network install servers, see Chapter 6, “Installing Linux” on page 101.

Setting up network installation The BladeCenter JS20 firmware has the capability to boot an operating system over a network using the BOOTP. You use this capability to initiate network installation of the BladeCenter JS20.

The BOOTP protocol is a client-server protocol. When initiating a network installation on a BladeCenter JS20, it behaves as a BOOTP client. Therefore, you need to set up a BOOTP server to support initiating a network installation.

When you instruct the BladeCenter JS20 firmware to boot over a network using BOOTP, it sends a request to the BOOTP server. The BOOTP server should generate a response that contains the following information: The IP address, subnet mask, and default gateway that the BladeCenter JS20 should use for the network interface that sent the BOOTP request The IP address of a TFTP server that the BladeCenter JS20 should contact, and the name of the file on that server that the BladeCenter JS20 should request, to get the operating system installation boot image

The BOOTP protocol has been around since the mid1980s. In many environments, BOOTP has now been replaced by the newer DHCP protocol. DHCP was designed to interoperate with BOOTP, and most DHCP servers can serve BOOTP clients.

Chapter 4. Planning considerations 61 When you perform network installations of AIX, you use the AIX BOOTP server. When you perform network installations of Linux, you use a DHCP server that is configured to support BOOTP clients.

When the BladeCenter JS20 firmware has received a response from a BOOTP

server, it contacts the TFTP server specified in the BOOTP response to load the operating system’s installation boot image.

The operating system’s installation boot image then starts running on the BladeCenter JS20. Eventually it contacts the Network File System (NFS) server to obtain the files that it needs to perform the operating system installation.

Learn the details about the actual process of setting up network installation in 5.5.4, “Specifying IP parameters to Open Firmware” on page 96, and 7.1, “Minimal NIM installation” on page 124.

Network interface selection The BladeCenter JS20 has two standard Gigabit Ethernet interfaces. The first is connected to I/O module bay 1, and the second is connected to I/O module bay 2. Refer to Figure 2-2 on page 9.

Restriction: The BladeCenter JS20 does not support the keyboard, video and mouse (KVM) console supported by other blade servers. Instead the BladeCenter JS20 only supports SoL remote text consoles via the BladeCenter Management Module.

The SoL remote text console function uses the first Gigabit Ethernet interface on each BladeCenter JS20 for communication between the Management Module and the service processor found in each BladeCenter JS20. This interface is known as eth0 under Linux and en0 under AIX.

Under normal conditions, using the Gigabit Ethernet interface by the SoL remote text console function is entirely transparent. It does not impact usage of the Ethernet interface by the operating system running on the BladeCenter JS20.

Unfortunately, the initiation of network installation using the BOOTP protocol from the BladeCenter JS20 firmware temporarily disrupts the operation of the SoL remote text console when it occurs on the same physical Gigabit Ethernet interface. This disruption makes it difficult to diagnose problems with the network installation process. Therefore, we recommend that you use the second Gigabit Ethernet interface on each BladeCenter JS20 during network installation of the operating system. The second interface is known as eth1 under Linux and en1 under AIX.

62 The IBM Eserver BladeCenter JS20 To use the second interface, you must have a LAN Switch I/O Module, or Pass-Thru I/O Module connected to an external LAN switch, installed in I/O module bay 2. You can learn more about why you should use the second Gigabit Ethernet interface for network installation in the RETAIN® tip H181655 on the Web at: http://www-307.ibm.com/pc/support/site.wss/document.do?lndocid=MIGR-55282

4.3 Systems management

The BladeCenter JS20 supports a rich set of systems management capabilities. This section discusses planning considerations associated with exploiting some of those systems management capabilities.

4.3.1 IBM Director IBM Director is a systems management tool. It was originally developed to provide a comprehensive management solution for servers and workstations based on Intel microprocessors. Recently its capabilities have been expanded to support management of other platforms such as the BladeCenter JS20. We introduce IBM Director in 3.2.2, “IBM Director” on page 50.

This section reviews the major planning considerations associated with the installation of IBM Director. For a comprehensive treatment of planning for IBM Director, refer to the IBM Redbook Implementing System Management Solutions using IBM Director, SG24-6188.

IBM Director has three components: IBM Director Server: This is the core of the IBM Director system management solution. IBM Director Console: This provides the user interface that system administrators and operators use to interact with IBM Director. IBM Director Agents: These provide the mechanism by which the IBM Director can monitor and control a specific server and operating system environment.

You can only install the IBM Director Agent on the BladeCenter JS20.

Chapter 4. Planning considerations 63 IBM Director Agent for the BladeCenter JS20 Support for the BladeCenter JS20 was introduced in IBM Director Version 4.12. You can learn how to install the IBM Director Agent in 8.1.2, “Installing an IBM Director Agent” on page 145.

We recommend that you install the IBM Director Server on an xSeries server.

For maximum functionality, this server needs IP connectivity to the Management Module in each BladeCenter chassis and to every blade server. This connectivity can be provided by connecting the IBM Director Server to both the hardware management subnet and operating system management subnets described in 4.1.1, “Minimal network requirements” on page 54.

If your environment is primarily comprised of servers running Linux, consider installation of the IBM Director Server under Linux.

4.3.2 IBM Cluster Systems Management IBM Cluster Systems Management is a systems management tool. It simplifies the management of clusters of servers running the AIX and Linux operating systems.

This section focuses on the planning considerations that are associated with using CSM to manage BladeCenter JS20s. For more planning information, refer to the IBM Cluster Systems Management for Linux Planning and Installation Guide, SA22-7873, and the IBM Cluster Systems Management for AIX5L Planning and Installation Guide, SA22-7919.

Supported CSM and operating system releases Preliminary support for the BladeCenter JS20 was introduced in CSM for Linux Version 1.3.3.1.

CSM requires specific support for each operating system version that is running on servers that it manages. Currently (at the time of publication) CSM Version 1.4 supports the following operating systems on the BladeCenter JS20: Red Hat Enterprise Linux Version 3 (QU3) AIX 5L™ Version 5.2 and 5.3 SUSE LINUX Enterprise Server (SLES) Version 8.1and 9

To check for updates to this list of supported operating systems, refer to the CSM publication site at: http://publib.boulder.ibm.com/clresctr/windows/public/clusterbooks.html

64 The IBM Eserver BladeCenter JS20 Management node selection Each CSM cluster requires a management node, which provides the central point of administration and control for the entire cluster. CSM has complex rules concerning the compatibility of management nodes, non-management nodes (cluster nodes), and the operating system they run. We attempt to simplify these rules by focusing exclusively on the requirements for a management node that only manages BladeCenter JS20s. If your environment is more complex, refer to the CSM planning documents.

You can use a CSM management node to both install and manage BladeCenter JS20s, or you can use a management node to manage BladeCenter JS20s that were installed using other mechanisms. Most environments use CSM to both install and manage BladeCenter JS20s, so we focus on that scenario.

If you use a management node to both install and manage BladeCenter JS20s running Linux, you must use the same type of Linux distribution on both the management node and the BladeCenter JS20s. However, the management node does not need the same processor architecture that is used in the BladeCenter JS20.

For example, if you plan to install SLES on BladeCenter JS20s, you can use the following types of management node and operating system combinations to both install and manage the BladeCenter JS20s: A supported xSeries server running SLES A supported pSeries server running SLES A IBM HS20 Blade Server running SLES A IBM BladeCenter JS20 running SLES

In large installations, we recommend that you use either an xSeries or pSeries server as a management node. If your environment also contains non-BladeCenter JS20s, such as the HS20 Blade Server, or xSeries cluster nodes, we recommend that you use an xSeries management node such as the xSeries 345.

We performed our testing of CSM using an xSeries management node, since many users of the BladeCenter JS20 may have environments that includes HS20 Blade Servers.

Chapter 4. Planning considerations 65 Network considerations and summary IBM CSM is designed to work in the type of network environment that is outlined in 4.1, “Network planning” on page 54, with multiple subnets that segregate different types of network traffic. The CSM management node is normally equipped with multiple Ethernet network interfaces that are connected as follows: One interface connects to the hardware management subnet to support CSM’s hardware management functions. One interface connects to the operating system management subnet to support CSM’s capabilities to install operating systems, maintain software and configuration files, and provide ongoing monitoring of cluster nodes. One interface connects to an external network to enable remote access to the management node by system administrators and operators.

For further information about the IBM Cluster Systems Management facility, refer to the online documentation that is available.

66 The IBM Eserver BladeCenter JS20

Part 2

Part 2 Implementing the BladeCenter JS20

This part discusses implementation of the IBM Eserver BladeCenter JS20. It covers planning considerations and setting up hardware. It explains how to install AIX 5L and Linux. Plus it presents systems management scenarios.

68 The IBM Eserver BladeCenter JS20

Chapter 5. Hardware setup

This chapter explains the various activities that you must perform before you can install an operating system on the BladeCenter JS20. Such activities include configuring the Management Module and LAN switch modules and updating the firmware. They also include configuring blade servers and the Serial over LAN (SoL) remote text console function and using the Open Firmware interfaces.

This section describes the steps to set up the BladeCenter Management Module so that you can work with BladeCenter JS20s. Advanced configuration of the Management Module is beyond the scope of this book.

The primary setup task for the Management Module is to assign IP addresses, which are necessary to communicate with the Management Module Web and command line interfaces. To learn more about selecting these IP addresses, see Chapter 4, “Planning considerations” on page 53.

The Management Module has two network interfaces, an external interface (eth0) and an internal interface (eth1). The external interface is accessible via the 10/100BaseT connector on the Management Module. The internal interface is connected to the management interfaces of all the installed I/O modules that support such interfaces. This includes all switch I/O modules.

The default behavior of a new Management Module is to request an IP address for the external interface via DHCP. If the Management Module does not receive a valid response from a DHCP server within two minutes of powering on, it uses a default static IP address of 192.168.70.125 with the default subnet mask of 255.255.255.0.

Note: The default host name is MMxxxxxxxxxxxx, where xxxxxxxxxxxx is the burned-in medium access control (MAC) address.

The default IP address for the internal interface is statically assigned to be 192.168.70.126 with a subnet mask of 255.255.255.0.

You can reset the IP addresses of a Management Module that was previously configured back to the factory defaults by using the IP reset button shown in Figure 5-1 on the Management Module. You can find the procedure for doing this in the IBM Eserver BladeCenter Management Module User’s Guide.

70 The IBM Eserver BladeCenter JS20

Figure 5-1 Management Module controls and indicators

Configuring the Management Module Use the following procedure to set the IP addresses for the Management Module external and internal interfaces: 1. Connect the Management Module to an isolated private Ethernet network. Also connect a workstation that has a Web browser to the same isolated private Ethernet network. 2. Configure a static IP address for the workstation Ethernet interface that is in the same subnet as the Management Module default IP addresses. For example, we used a static address of 192.168.70.100 with a subnet mask of 255.255.255.0 for the workstation Ethernet interface. Do not use addresses in the range of 192.168.70.125 through 192.168.70.130. These addresses conflict with the default addresses assigned by the Management Module. 3. Connect to the Management Module Web interface by pointing your Web browser on the workstation to:

http://192.168.70.125 4. Enter a valid user ID and password. The factory default configuration of a Management Module defines a user ID named USERID with a password of

Chapter 5. Hardware setup 71 PASSW0RD. The 0 in the password is a zero. In production environments, consider changing these defaults.

Note: The default user ID is USERID, and the default password is PASSW0RD. The number zero is between the W and the R.

5. From the Management Module Web interface, select MM Control → Network Interfaces. 6. The form window shown in Figure 5-2 is displayed. Enter the desired external and internal IP addresses, subnet masks, and default gateway for the Management Module. Then click Save to store the new IP addresses.

Figure 5-2 IP address configuration

72 The IBM Eserver BladeCenter JS20 7. Restart the Management Module. In the Management Module Web interface, select MM Control → Restart MM. Then click the Restart button on the displayed form. You are prompted to confirm the restart before it occurs.

8. Remove the Management Module from the isolated private Ethernet network and connect it to the network you will use to manage the BladeCenter.

You can now connect to the Management Module Web and command line interfaces using the IP address that you assigned to the Management Module external network interface.

Now consider performing other Management Module setup tasks, such as: Setting the Management Module date and time so that log entries have useful timestamps Defining user IDs and passwords for the system administrators and operators who will manage the BladeCenter Alternatively, you can configure the Management Module to use a Lightweight Directory Access Protocol (LDAP) directory for this purpose. Configuring the Management Module to send alerts to management systems via SNMP, or system administrators using e-mail via Simple Mail Transfer Protocol (SMTP) Enabling the use of Secure Sockets Layer (SSL) to securely access the Management Module Web interface Enabling the use of Secure Shell (SSH) to securely access the Management Module command line interface (CLI)

For additional information about how to perform these tasks, refer to the IBM Eserver BladeCenter Management Module User’s Guide.

5.2 LAN Switch I/O Module configuration

This section explains the basic set up LAN Switch I/O Modules so you can work with BladeCenter JS20s. Advanced configuration of LAN Switch I/O Modules is beyond the scope of this book.

The primary setup tasks for LAN Switch I/O Modules are to: Assign IP addresses to the LAN Switch I/O Module management interfaces Enable the LAN Switch I/O Module external Ethernet ports

You can learn about the selection of IP addresses for the LAN Switch I/O Module management interfaces in Chapter 4, “Planning considerations” on page 53.

Chapter 5. Hardware setup 73 When a new LAN Switch I/O Module is first installed, the Management Module assigns a default IP address to the management interface of the LAN Switch I/O Module. The default IP address is chosen based on the I/O module bay where the LAN Switch I/O Module is installed. Those I/O modules installed in I/O module bays 1, 2, 3, and 4 are assigned IP addresses 192.168.70.127, 192.168.70.128, 192.168.70.129, and 192.168.70.130, respectively.

Set the IP address of each LAN Switch I/O Module management interface and enable the external ports: 1. Connect to the Management Module using the Web browser interface as explained in “Configuring the Management Module” on page 71. 2. From the Management Module Web interface, select I/O Module Tasks → Management. 3. In the form shown in Figure 5-3, scroll down to the entry for the LAN Switch I/O Module that you want to configure. Enter the IP address, subnet mask, and gateway that you want to assign to the LAN Switch I/O Module management interface. Click the Save button to activate the new IP address.

Figure 5-3 LAN Switch I/O Module IP address

74 The IBM Eserver BladeCenter JS20 4. You are prompted to confirm that you want to change the IP address.

5. Select the Advanced Management link for the LAN Switch I/O Module. 6. Scroll down to the Advanced Setup section of the displayed form. This is illustrated in Figure 5-4. For the External Ports field, select Enabled from the

drop-down list and then click the Save button.

Figure 5-4 LAN Switch I/O Module setup

Chapter 5. Hardware setup 75 At this point, the LAN Switch I/O Module management interface has an IP address. Also, the external ports on the LAN switch module are enabled so that they can be used to communicate with blade servers.

The SoL remote text console function of the Management Module depends on a

VLAN provided by a LAN Switch I/O Module installed in I/O module bay 1. This VLAN is automatically provided by the 4-Port Gigabit Ethernet Switch Module and the Nortel Networks Layer 2-7 Gigabit Ethernet Switch Module using VLAN 4095.

If you are using a Cisco Systems Intelligent Gigabit Ethernet Switch Module in I/O module bay 1, you must manually configure this VLAN. A procedure is documented in the Cisco Systems Intelligent Gigabit Ethernet Switch Module for the IBM Eserver BladeCenter Installation Guide. This Cisco guide describes how to manually set up the VLAN that is needed to support the SoL remote text console function.

5.3 Blade server configuration

Minimal configuration is required for each BladeCenter JS20 prior to installing an operating system. The two main configuration tasks are: Assigning a name to the blade server Setting the blade server boot sequence

The easiest way to perform these tasks is through the Management Module Web interface.

76 The IBM Eserver BladeCenter JS20 5.3.1 Assigning names to blade servers

Set the name of each blade server in a BladeCenter chassis by using these steps:

1. From the Management Module Web interface, select Blade Tasks → Configuration. 2. In the Blade Information section of the form, as shown in Figure 5-5, type the name that you want to assign to each blade server. 3. Scroll down the form and select the Save button to save the names that you have assigned to each blade server.

Figure 5-5 Blade server naming

Chapter 5. Hardware setup 77 5.3.2 Setting the boot sequence

Set the boot sequence of each blade server by using this procedure:

1. From the Management Module Web interface, select Blade Tasks → Configuration. 2. Scroll down the form to the Boot Sequence section to see the current boot sequence for all the blade servers. 3. If a blade server does not have the correct boot sequence, you can change the boot sequence by selecting the blade server name, which displays the form shown in Figure 5-6. 4. Set the desired boot sequence using the drop-down lists and click the Save button to set the new boot sequence.

The correct boot sequence depends on the method that you plan to use to install an operating system on the blade server.

Figure 5-6 Boot sequence of blades

78 The IBM Eserver BladeCenter JS20 5.4 Firmware

You may need to upgrade various firmware components before you install an

operating system on a BladeCenter JS20. The minimum firmware levels required and the actual firmware levels we used during the development of this book are listed in Table 5-1.

To check the level of firmware components, use the following two steps: 1. Find the latest firmware levels and update packages distributed on the IBM support Web site. a. Go to the IBM support Web site: http://www.ibm.com/pc/support b. Under Download, select Driver matrices. c. On the Personal computing drivers Web page, select Servers. d. On the Software and device drivers - Servers Web page, select eServer™ BladeCenter. e. On the “Software and device drivers - IBM eServer BladeCenter (Type 8677)” Web page, you see the firmware levels for the BladeCenter. f. You can also select IBM eServer BladeCenter JS20 - Software and device drivers to see JS20-specific firmware levels. 2. Identify the current build levels and levels of firmware that are currently installed by displaying the firmware vital product data (VPD). You can view the firmware VPD for most firmware components, as shown in Table 5-1, through the Management Module Web interface. This is illustrated in Figure 5-7.

Table 5-1 BladeCenter and BladeCenter JS20 firmware components Firmware component Minimum level Level we used

Management Module V1.09 (BRET58A) V1.14 (BRET67I)

4-Port Gigabit Ethernet V1.04 (ibmrun.081) V1.07 (ibmrun.095) Switch Module

BladeCenter JS20 BIOS FW04310120 FW04310120

BladeCenter JS20 BSMP V1.00 (BQ8T16A) V1.0 (BQ8T20A)

Note: FW04310120 for the JS20 BIOS is the minimum required supported level to support AIX 5L releases like V5.2 (ML4).

Chapter 5. Hardware setup 79

Figure 5-7 Viewing firmware levels on BladeCenter

The following sections explain how to update the various firmware components associated with the BladeCenter JS20.

5.4.1 Management Module firmware The easiest way to upgrade the firmware of the Management Module is through the Management Module Web interface.

1. Download the firmware update package from the IBM support Web site. This is usually a ZIP file that contains several files with a PKT extension.

80 The IBM Eserver BladeCenter JS20 2. Unzip the firmware update package into a directory on the workstation where you are running the Web browser that you are using to connect the Management Module Web interface.

3. Update each PKT file to the Management Module as explained in the following steps. a. In the navigation panel of the Management Module Web interface (Figure 5-8), select MM Control → Firmware Update. b. In the Update MM Firmware pane (right side of Figure 5-8), select the Browse button and locate the PKT file in the directory where you unpacked it. c. Click the Update button to initiate the download of the PKT file to the Management Module. This step may take some time to complete.

Figure 5-8 Updating the Management Module firmware

d. A window opens that shows you the progress of the download. e. Click the Continue button when prompted to confirm the firmware update. This step may also take some time to complete. f. Another window opens that shows you the progress.

Chapter 5. Hardware setup 81 4. After you download all the PKT files to the Management Module, restart the Management Module to make the firmware updates active. In the Management Module Web interface, select MM Control → Restart MM.

5.4.2 LAN Switch I/O Module firmware The method you should use to upgrade the firmware of a LAN switch module depends on the type of switch module. We demonstrate how you can upgrade firmware for the 4-port Gigabit Ethernet Switch Module, which is the switch module we used in our testing.

For information about upgrading firmware in the Nortel Networks Layer 2-7 Gigabit Ethernet Switch Module, refer to the Nortel Networks Layer 2-7 GbE Switch Module for IBM Eserver BladeCenter Installation Guide.

For information about upgrading firmware in the Cisco Systems Intelligent Gigabit Ethernet Switch Module, refer to the Cisco Systems Intelligent Gigabit Ethernet Switch Module for the IBM Eserver BladeCenter Installation Guide. 1. Download the firmware update package from the IBM support Web site. This is usually a ZIP file that contains the firmware image file. 2. Unzip the firmware update package into a directory on the workstation where you are running the Web browser that you use to connect the Management Module Web interface. 3. Before you can upgrade the firmware of the 4-Port Gigabit Ethernet Switch Module, configure an IP address for the switch module management interface. This enables the capability to directly connect to the switch module Web interface. To configure the IP address of the switch module management interface, see 5.2, “LAN Switch I/O Module configuration” on page 73. 4. Connect to the switch module Web interface via the Management Module Web interface using the following procedure: a. In the Management Module Web interface, select I/O Module Tasks → Management. b. Select the Advanced Management link for the I/O module bay where the switch module is installed. c. Scroll down to the bottom of the page to the Start Telnet/Web Session category. Click the Start Web Session button. d. A new window opens for the switch module Web interface. You are prompted to provide a user ID and a password. The default user ID is USERID, and the default password is PASSW0RD, where the number 0 is between the letters W and R in the password. Consider changing the defaults in a production environment.

82 The IBM Eserver BladeCenter JS20 You can also connect directly to the switch module user interface from any Web browser if you know the IP address of the switch module.

5. Upgrade the firmware using the following procedure: a. From the switch module Web interface, select Maintenance → Using

Browser → Upgrade Firmware/Configuration File. b. You now see the form shown in Figure 5-9. Select the Browse button and locate the switch module firmware file in the directory where you put it earlier. c. Select the Start button to initiate the switch module firmware update process. d. A window opens requesting confirmation. After you confirm the update, the main window displays the status of the update until it completes and reboots the switch module.

Figure 5-9 Firmware upgrade of 4-Port Gb Ethernet Switch Module

6. After you restart the switch module, verify that the IP address of the switch module is still set correctly through the Management Module Web interface. In some cases, you may need to reset the IP address.

Chapter 5. Hardware setup 83 5.4.3 BladeCenter JS20 firmware (BIOS)

There is currently no way to upgrade the firmware (BIOS) of a BladeCenter JS20 without first installing an operating system on the blade server. This is usually not an issue, because newly shipped BladeCenter JS20s already have current

firmware installed.

For completeness, we explain how to upgrade the firmware of a BladeCenter JS20 after you install an operating system.

Upgrading firmware under AIX AIX includes the update_flash utility as part of the AIX diagnostics package. To use the update_flash utility, you must first download the latest firmware from the IBM support Web site. Then transfer it to a disk that is accessible to the operating system running on the BladeCenter JS20. Next, you start the update_flash utility from the directory where the new firmware is located. The update_flash utility requires that you have root privileges.

The following example illustrates the usage of update_flash: /usr/lpp/diagnostics/bin/update_flash -f JS1FW419A.IMG

In this example, JS1FW419A.IMG is the firmware image that was previously downloaded from the IBM support Web site. The update_flash utility asks you to confirm that you want to update the firmware and then reboot the operating system to perform the actual update.

If you subsequently have a problem with the new firmware and want to revert to the previous firmware level, use the following command: /usr/lpp/diagnostics/bin/update_flash -r

Alternatively, if you are satisfied with the new firmware, commit the firmware update before you install any future firmware by using this command: /usr/lpp/diagnostics/bin/update_flash -c

Upgrading firmware under Linux The mechanism used to update the firmware (BIOS) of the BladeCenter JS20 under the Linux operating system is based on a Linux kernel module called rtas_flash. This kernel module is available in most recent 2.4 kernels and in all 2.6 kernels. All supported Linux distributions for the BladeCenter JS20 include this kernel module. The rtas_flash kernel module creates interfaces for manipulating the flash memory where the firmware is stored in the /proc/ppc64/rtas directory. In most Linux distributions, this kernel module is normally not loaded at system boot.

84 The IBM Eserver BladeCenter JS20 Therefore, the interfaces to manipulate the flash memory are not normally present in /proc/ppc64/rtas.

IBM provides a script for Linux called update_flash that uses the interfaces provided by the rtas_flash kernel module. It also supports the same options and

syntax as the update_flash utility under AIX.

You can obtain the latest version of the update_flash script by installing the diagnostic utilities for Linux on POWER that IBM distributes from the Web at: http://techsupport.services.ibm.com/server/lopdiags

Note: Some Linux distributions provide a version of the update_flash script that is not fully functional. Always obtain the latest version of the update_flash script from IBM before you attempt to update firmware.

To use the update_flash script, follow these steps: 1. Download the latest firmware from the IBM support Web site. 2. Transfer it to a disk that is accessible to the operating system running on the BladeCenter JS20. 3. Enter the update_flash command from the directory where the new firmware is located. The update_flash command requires that you have root privileges.

The following example illustrates the usage of update_flash: /usr/sbin/update_flash -f JS1FW419A.IMG

In this example, JS1FW419A.IMG is the firmware image that was previously downloaded from the IBM support Web site. The update_flash utility copies the new firmware image into the kernel and then reboots the operating system to perform the actual firmware update.

If you subsequently have a problem with the new firmware and want to revert to the previous firmware level, then use the following command: /usr/sbin/update_flash -r

If you are happy with the new firmware, commit the firmware upgrade before you install any future firmware by using the following command: /usr/sbin/update_flash -c

5.4.4 Integrated Systems Management Processor firmware The easiest way to upgrade the firmware of the BladeCenter JS20 Integrated Systems Management Processor (ISMP) is through the Management Module Web interface.

Chapter 5. Hardware setup 85 1. Download the firmware update package from the IBM support Web site. This package contains a file with a ZIP extension, which is the firmware for the ISMP.

2. Place the file into a directory on the workstation where you will run the Web browser that you will use to connect the Management Module Web interface. 3. Download the PKT file to the ISMP. Follow this upgrade procedure: a. As shown in Figure 5-10, select Blade Tasks → Firmware Update in the Management Module Web interface. b. The Update Blade Firmware pane is displayed (right side of Figure 5-10). In the Target drop-down list, select the blade server on which you want to upgrade the ISMPfirmware. c. Click the Browse button and locate the ISMP firmware ZIP file in the directory where you placed it earlier. d. Click the Update button to initiate download of the ZIP file to the Management Module. This step may take some time to complete.

Figure 5-10 Blade firmware update

86 The IBM Eserver BladeCenter JS20 e. A window opens that shows the progress of the download.

f. Click the Continue button when prompted to confirm the firmware update. This step may also take some time to complete. g. A window opens that shows the progress.

The new ISMP firmware is now active in the target BladeCenter JS20.

5.5 Providing a console for the JS20 blades

The BladeCenter JS20 differs from other blade servers that are available for the BladeCenter in that it does not provide an interface to a keyboard, video monitor, and mouse (KVM) console. Therefore, you must set up the SoL remote text console function to provide a console for a BladeCenter JS20. Use of SoL is optional for other types of blade server that support KVM consoles.

The SoL remote text console function involves several different components in the BladeCenter infrastructure as illustrated in Figure 5-11.

The SoL remote console function works as follows: 1. Using a Telnet or SSH client, connect to the BladeCenter Management Module CLI. This is usually via an external management network that is connected to the Management Module’s 10/100BaseT Ethernet interface. 2. From the Management Module’s CLI, initiate a SoL remote console session to the desired blade server. 3. The Management Module uses a private VLAN provided by the LAN switch module in I/O module bay 1, to transport the SoL data stream to the Ethernet interface of the target blade server. 4. The Ethernet controller of the target blade server passes the SoL data stream received from the private VLAN to the blade system management processor (BSMP), which manages the text console for the blade server.

Chapter 5. Hardware setup 87

Telnet or SSH Client

External Management Network

External Interface (eth0)

Management Module

Internal Interface LAN Switch (eth1) Module in I/O Module I/O Module Bay 2 Bay 1

Ethernet Ethernet Interface Interface (eth0) (eth1)

BSMP Ethernet Controller

JS20 Blade Server

Figure 5-11 Serial over LAN components

The sections that follow explain how to configure and use the SoL remote text console function.

88 The IBM Eserver BladeCenter JS20 5.5.1 Configuring Serial over LAN

Before you attempt to configure the SoL remote text console function, verify that you have all the prerequisites in place:

A supported LAN switch module installed in I/O module bay 1 This switch module is used to provide a VLAN that connects the Management Module to the first Ethernet interface on each blade server. The minimum firmware levels described in 5.4, “Firmware” on page 79 This is important if you install a BladeCenter JS20 in an existing BladeCenter chassis that may have back-level firmware in the Management Module or LAN switch modules. A reliable Telnet or SSH client A network connecting the Telnet or SSH client to the BladeCenter Management Module external 10/100BaseT Ethernet interface An identified range of IP addresses that will be used by the Management Module to communicate with the BSMP on each blade server via the private VLAN This was discussed in Chapter 4, “Planning considerations” on page 53.

The easiest way to configure the SoL remote text console function is through the Management Module Web interface as explained in the following steps.

To configure the SoL remote text console function, use the following procedure: 1. From the Management Module Web interface, select Blade Tasks → Serial Over LAN. 2. In the right pane, scroll down until you see the Serial Over LAN Configuration section (see Figure 5-12). Complete the following tasks: a. From the Serial over LAN list, select the Enabled option. b. Leave the value for SoL VLAN ID at the default (4095) if you have either a 4-port Gigabit Ethernet Switch Module or a Nortel Networks Layer 2-7 Gigabit Ethernet Switch Module installed in I/O module bay 1. If you have a Cisco Systems Intelligent Gigabit Ethernet Switch Module installed in I/O module bay 1, set the VLAN ID to the same value you used when you manually defined the VLAN that supports SoL. c. Enter the start of the IP address range that will be used by the Management Module to communicate with the BSMP on each blade server. d. Leave the values for Accumulate timeout, Send threshold, Retry count, and Retry interval at their defaults (5, 250, 3, and 250).

Chapter 5. Hardware setup 89 e. Leave the values for User Defined Keystroke Sequences at their defaults.

f. Click the Save button.

Figure 5-12 Serial over LAN configuration

3. Restart the Management Module. a. In the Management Module Web interface, select MM Control → Restart MM. b. Click the Restart button on the displayed form. c. You are prompted to confirm the restart before it occurs. 4. After the Management Module has restarted and you have reconnected to the Management Module Web interface from your Web browser, enable the SoL remote text console for each blade server. a. Select Blade Tasks → Serial Over LAN from the Management Module Web interface. b. In the right pane, scroll down until you see the Serial Over LAN Status section (see Figure 5-13). c. Select the blade servers you want to enable for SoL. You can choose all of them by clicking the check box at the top of the table.

90 The IBM Eserver BladeCenter JS20 d. Click the Enable Serial Over LAN link underneath the table. You may need to scroll down to see this link.

Figure 5-13 Serial over LAN status

e. After a few seconds, the window refreshes. In the SoL column of the table, verify that each blade server has a status of Enable.

The configuration of the SoL remote text console function is now complete.

5.5.2 Using Serial over LAN Access to the SoL remote text console function is via the Management Module CLI. The CLI is documented in the IBM ^ BladeCenter Management Module Command-Line Interface Reference Guide. You can connect to the Management Module CLI using either a Telnet or SSH client. Figure 5-14 shows an active Telnet session that displays the help command. The Management Module can support up to 20 concurrent CLI

Chapter 5. Hardware setup 91 connections. This is sufficient to support the concurrent use of a SoL remote text console to each blade server in a full BladeCenter chassis. At the same time, it supports six additional CLI connections for other administrative activities.

Figure 5-14 Management Module’s command line interface

Restriction: You can only have one active SoL remote text console connection to each blade server.

The Management Module CLI is context sensitive. When you issue CLI

commands, you can accept the current context or override the context using the -T option provided on most commands. In our examples, we use the -T option to make it clear on which entity the command is operating. You can change the context for the Management Module CLI by using the env -T command.

92 The IBM Eserver BladeCenter JS20 To use the SoL remote text console function, first connect to the Management Module from a Telnet or SSH client. You are prompted to enter a user ID and password. The default user ID is USERID and the default password is PASSW0RD, where 0 in the default password is a zero. Consider changing the defaults in a production environment. There are several different commands that you can use to access the SoL remote text console function. The most useful commands are listed in Table 5-2.

Table 5-2 Management Module commands for SoL Command What it does

console Open a SoL remote text console for the blade server. This command fails if another SoL remote text console is already open for the blade server.

console -o Terminate any existing SoL remote text console for the blade server and open a SoL remote text console for the blade server.

boot -c Reset the blade server, and then open a SoL remote text console for the blade server.

reset -c This is functionally equivalent to the boot -c command when used in a blade server context.

power -on -c Power on the blade server and then open a SoL remote text console for the blade server.

power -cycle -c Power on the blade server and then open a SoL remote text console for the blade server. If the blade server is already powered on, power it off first, and then power on.

Here are some examples of how to use these commands. To open a SoL remote text console to the blade server in bay 3, you use the following command: console -T system:blade[3]

To reset the blade server in bay 2 and then open a SoL remote text console to the blade server, you use the following command: boot -c -T system:blade[2]

To terminate an active SoL remote text console, press the Esc key followed by an open parenthesis (Shift+9 on U.S. keyboards). When the SoL remote text console ends, you return to the Management Module CLI prompt.

Inactivity timeout The Management Module CLI has an aggressive default inactivity timeout. If you do not use the CLI or a SoL remote text console accessed through the CLI for 2

Chapter 5. Hardware setup 93 minutes, you are disconnected. You can increase the timeout using the telnetcfg command. For example, the following command increases the inactivity timeout to 10 minutes (600 seconds):

telnetcfg -t 600 -T system:mm[1]

Setting CRLF Over the years, there have been many variations on the interpretation of the Enter key in a Telnet session. Some Telnet daemons expect carriage return + linefeed (CRLF) and some expect carriage return alone (CR). From our testing, CR seems to be the most compatible. During your SoL session, if when you press the Enter key, the key behaves as though it was pressed twice and not once, then correct the CRLF option. Under Windows 2000 Telnet, do this as shown in Figure 5-15 before you open a connection to the Management Module.

Microsoft (R) Windows 2000 (TM) Version 5.00 (Build 2195) Welcome to Microsoft Telnet Client Telnet Client Build 5.00.99206.1

Escape Character is 'CTRL+]'

Microsoft Telnet> unset crlf Microsoft Telnet> open bcmm

Figure 5-15 Setting CRLF in Windows 2000 Telnet

Refer to the developer of your client for other Telnet implementations.

5.5.3 Open Firmware interfaces In some situations, you may need to access the Open Firmware interfaces to the BladeCenter JS20 firmware. This section explains how to access Open Firmware interfaces and how to use some of the functions they provide.

Activating the Open Firmware interfaces You can access the Open Firmware interface whenever you power on a BladeCenter JS20. Before you can access the Open Firmware interface, set up the SoL remote text console function as described in 5.5, “Providing a console for the JS20 blades” on page 87.

Use the following procedure to access the Open Firmware interface: 1. Using a Telnet or SSH client, connect to the Management Module external Ethernet interface IP address.

94 The IBM Eserver BladeCenter JS20 2. When prompted, enter a valid user ID and password. The default Management Module user ID is USERID, and the default password is PASSW0RD, where the 0 in the default password is a zero. Consider changing the defaults in any production environment. 3. Power cycle the blade and start an SoL console by using the power -cycle -c command. For example, to power cycle and start a SoL remote text console with the blade server in the first bay (bay 1), use the command: power -cycle -c -T system:blade[1] 4. After approximately 30 seconds, you see a sequence of checkpoint codes displayed on the console. These codes are generated by the Power On Self Test (POST). The meaning of the checkpoint codes is documented in the IBM Eserver BladeCenter JS20 Type 8842 Installation and User’s Guide. 5. Wait until the checkpoint code D5BB is displayed and then type 8. The POST pauses for approximately five seconds at this checkpoint code to give you time to respond to the checkpoint code. A number is displayed after the D5BB checkpoint code, indicating how many seconds remain before the POST will proceed further. For example, Figure 5-16 shows that four seconds remain.

E153 U8842.P1Z.23A1073-P1-T7 EAA1 U8842.P1Z.23A1073-P1 E172 U8842.P1Z.23A1073-P1 EAA1 U8842.P1Z.23A1073-P1 E152 U8842.P1Z.23A1073-P1 E153 U8842.P1Z.23A1073-P1 E152 U8842.P1Z.23A1073-P1 E153 U8842.P1Z.23A1073-P1 EAA1 U8842.P1Z.23A1073-P1 E172 U8842.P1Z.23A1073-P1 EAA1 U8842.P1Z.23A1073-P1-C5 D001 D003 D004 E139 E14A D008 E1F0 E1F1

D099 D5BB 4

Figure 5-16 POST checkpoint codes

Chapter 5. Hardware setup 95 6. As soon as you type 8, the Open Firmware command prompt is displayed.

If you do not type any key when the D5BB checkpoint code is displayed, the blade server proceeds to boot using the default boot sequence. To learn how to configure the boot sequence, see 5.3, “Blade server configuration” on page 76.

5.5.4 Specifying IP parameters to Open Firmware You can use a directed bootp to install a JS20 blade from a NIM server. bootp does not require the NIM server to be on the same subnet as the JS20 blade. Nor does this option require that you have the MAC address of the network adapter on the JS20 blade.

To perform a directed bootp, you need an SoL connection to the blade so that you can specify the IP parameters to Open Firmware. Currently you must have two network adapters to perform a NIM installation if you are using SoL. You cannot install AIX 5L over the same adapter that is using SoL.

Step 1: Preparing your NIM server Follow these steps to prepare your NIM server. 1. Create a SPOT, lpp_source, and any other resources that you need at the level of AIX 5L that you want to install on your NIM server. Your NIM server is usually the NIM master, but you can also set up a NIM client as a NIM server. For instructions on how to create NIM resources, see 7.1, “Minimal NIM installation” on page 124. 2. Ensure that you have the information in the following worksheet (Table 5-3) for your JS20 blade before you proceed with the installation.

Table 5-3 Network configuration information worksheet Network attribute Value

Network interface

Host name

IP address ______.______.______.______

Network mask ______.______.______.______

Name server ______.______.______.______

Domain name ______.______.______.______Gateway ______.______.______.______

96 The IBM Eserver BladeCenter JS20 3. Define the JS20 blade as a NIM client on your NIM master by running the following command on the NIM master:

smitty nim_mkmac This command creates a client definition for your JS20 blade. You can also

define the JS20 blade using the define NIM operation on the command line. 4. If you want to set the JS20 blade's name server and domain name after the installation, use a resolv_conf resource. 5. Set up your NIM master to install the JS20 blade, by running this command: smitty nim_bosinst Select the JS20 blade that you defined earlier as your target. Then select the type of install that you want to perform and select the installation resources that you want to use to install the JS20 blade. You can also prepare the JS20 blade to install using the bos_inst NIM operation on the command line.

Note: If the JS20 blade is powered off or was never installed, set “Initiate reboot and installation now?” to no and press Enter in the SMIT interface.

If the JS20 blade is powered on and running AIX 5L, set “Initiate reboot and installation now?” to yes in the SMIT interface. If you choose this option, a directed bootp is initiated by default and you can skip step 2. Before you run this command, ensure that the JS20 blade is a registered NIM client. 1. Run smitty niminit on the JS20 blade. 2. Specify the host name of your NIM master and the interface you want to use for the installation.

You can also initialize the JS20 blade using the niminit command on the command line.

Step 2: Specifying a directed bootp from the JS20 blade Specify a directed bootp from the JS20 blade by using these steps: 1. Open a Web interface to the Management Module by navigating to the IP address or host name of the Management Module using a Web browser. 2. Enable SoL to the JS20 blade from the Management Module Web interface. Select Blade Tasks →Serial Over LAN. 3. Select the JS20 blade that you are installing and click Enable Serial Over LAN. 4. Power on the JS20 blade from the Management Module Web interface Blade Tasks →Power/Restart.

Chapter 5. Hardware setup 97 5. Select the JS20 blade that you are installing and click Power On Blade.

6. Open a Serial over LAN connection to the JS20 blade by using Telnet into the Management Module and running the console command. For example, if the JS20 blade is in slot 3, run the following command:

console -T blade[3] The Serial over LAN connection shows a series of LED numbers. 7. Type 8 when you see D5BB to go to the Open Firmware prompt. 8. Boot from the network by typing: boot net:bootp,server_ip,,client_ip,gateway_ip Because SoL uses ent0, you must use ent1. Run a command similar to this one (note the double comma): boot /pci@8000000f8000000/pci@0/ethernet@1,1:bootp,192.168.2.10,,192.168.1.11,192.168.1.1

Note: You must specify the full device path name with this command. To determine the full path to your device, list the device tree by running the ls command at the Open Firmware prompt. This command displays output similar to this: 0 > ls 000000c87f18: /ibm,serial 000000c88840: /chosen 000000c88a98: /packages ... 000000d31488: /vdevice 000000d327a8: /vty@0 000000d32f88: /IBM,sp@4000 000000d33f10: /rtc@4001 000000d34a18: /pci@8000000f8000000 000000d384d0: /pci@0 000000d4bbd0: /ethernet@1 000000d5af50: /ethernet@1,1 000000d3be00: /pci@3 000000d6a350: /usb@0 000000d845f8: /hub@1 000000d854b8: /usb@0,1 000000d9f760: /hub@1 000000d3f798: /pci@1f 000000d45ed8: /ide@4,1 000000d47b10: /disk@0

The highlighted items are the path to the second Ethernet adapter. You pass this information to the boot command to initiate a network boot from the second Ethernet adapter.

98 The IBM Eserver BladeCenter JS20 After you run the boot command, the network installation begins. Output similar to Example 5-1 is displayed on the SoL connection.

Example 5-1 Output of the boot command

BOOTP: chosen-network-type = ethernet,auto,none,auto BOOTP: server IP = 192.168.2.10 BOOTP: requested filename = BOOTP: client IP = 192.168.1.11 BOOTP: client HW addr = 0 d 60 1e c cb BOOTP: gateway IP = 192.168.1.1 BOOTP: device /pci@8000000f8000000/pci@0/ethernet@1,1 BOOTP: loc-code U8842.P1Z.23A0984-P1-T7 BOOTP R = 1 FILE: /tftpboot/js20blade1.austin.ibm.com Load Addr=0x0000000000004000, Max Size=0x0000000000bfc000 FINAL Packet Count = 21131 FINAL File Size = 10818623 bytes. load-base=0x4000 real-base=0xc00000 Elapsed time since release of system processors: 2 mins 28 secs

Chapter 5. Hardware setup 99

100 The IBM Eserver BladeCenter JS20

Chapter 6. Installing Linux

This chapter explains how to install Red Hat Enterprise Linux (RHEL) and SUSE LINUX Enterprise Server (SLES) on the IBM Eserver BladeCenter JS20.

Both Red Hat Enterprise Linux and SUSE LINUX Enterprise Server are supported operating systems for the BladeCenter JS20. This chapter explains how to set up and conduct a network-based installation of both distributions. Because the general process of doing this is similar for both operating systems, the focus of this chapter is on the installation of Red Hat Enterprise Linux 3.0 and SUSE LINUX Enterprise Server 9.0.

Tip: Prior to installing any operating system, we recommend that you upgrade all firmware to the latest level. See 5.4, “Firmware” on page 79, for instructions about how to complete this task.

6.1.1 Configuring the sources Since we are doing a network installation, the first step is to configure a file space that contains the source files for the operating system installation. For all cited examples, we created the sources on a pSeries 690 that was connected to the same local area network (LAN) as the BladeCenter. Since there are special tools available within each distribution for this purpose, we configured each network install server with a similar distribution. For example, the server that installed RHEL 3 also ran RHEL 3.

Tip: We recommend a 100 Mb or faster LAN that is local to both the machine that will host the files and the BladeCenter. We also recommend that the file spaces be shared via Network File System (NFS).

Red Hat sources The Red Hat installer, Anaconda, makes it simple to set up a remote file share. Simply copy the ISOs for the source CDs into your NFS mount. To create an ISO of the first CD and place it in /mnt/exports/, enter: # dd if=/dev/cdrom of=/mnt/exports/RHEL-3.0-U2-disc1.iso bs=32M

Repeat the previous step for all the CDs, changing the file name as appropriate. You may also increase the bs parameter as appropriate. This parameter controls block size. The larger the block size is, the more RAM is taken for the dd process, but the faster the process takes.

Important: Ensure that the CD is not mounted before you begin the dd. Also ensure that the destination of the ISO has enough space to store all the data. Copying a full complement of RHEL 3 ISOs can take over 4 GB.

102 The IBM Eserver BladeCenter JS20 Configure the NFS daemon to export the mount point where the ISOs are located. You can do this by editing /etc/exports. Example 6-1 shows the export file we used.

Example 6-1 /etc/exports file contents

/mnt/exports/ 192.168.1.0/24(ro)

After the /etc/exports file is created or updated, we enabled NFS as shown in Example 6-2.

Example 6-2 Starting NFS # service nfs start Starting NFS services: [ OK ] Starting NFS quotas: [ OK ] Starting NFS daemon: [ OK ] Starting NFS mountd: [ OK ] #

Attention: If NFS does not start properly, refer to the Red Hat Enterprise Linux Reference Guide at the following Web site for troubleshooting information: http://www.redhat.com/docs

Chapter 6. Installing Linux 103 SuSE sources SuSE installations are usually administered via a program called Yet Another Setup Tool (YaST). 1. Within YaST, there is an applet for configuring installation sources on the server. Access this from the Misc section in the left pane, as shown in Figure 6-1.

Figure 6-1 YaST installation server configuration

104 The IBM Eserver BladeCenter JS20 2. You now see the Source Configuration panel (Figure 6-2). Select SuSE SLES Version 9 (ppc).

Figure 6-2 Selecting the distribution

Chapter 6. Installing Linux 105 3. In the next panel as shown in Figure 6-3, you select which media to use as a source for the SuSE distribution. We found it easier to use ISOs rather than CDs. Proceeding past this window builds the source directory. You may be prompted for additional CDs (or locations of ISOs) while the tree is being built.

Tip: If you have any service packs or additional CDs that you want to add, select the Prompt for Additional CDs check box.

Figure 6-3 Source Configuration pane

106 The IBM Eserver BladeCenter JS20 4. After the source files are copied into the installation share, click Finish as shown in Figure 6-4. This completes any final tasks such as starting the NFS daemon.

Figure 6-4 Completing the setup

6.1.2 The zImage.initrd file PowerPC-based hosts support booting directly from the network provided that they are served a zImage.initrd via BOOTP. The zImage.initrd is a package that contains both the kernel and initial RAM disk in a single file.

Most distributions already have a zImage.initrd, which you can use for network booting. For RHEL, you can find this file on the first CD as images/netboot.img. For SuSE, this file is called install and is in the root directory of the first CD. Although these files are referred to by different names, they are the zImage.initrd file that we require.

Building your own zImage.initrd Although most users use the zImage.initrd provided by their distribution, you can also compile your own.

Tip: This section address the nuances of how to create a zImage.initrd. If you need more information about compiling a kernel, see the Kernel HOWTO at: http://www.tldp.org

Chapter 6. Installing Linux 107 To build your own zImage.initrd, follow these steps:

1. Obtain the necessary kernel sources and development utilities for your distribution as appropriate. If the necessary packages are not installed, you may experience some errors, which cause your compilation to fail. Consult your distribution’s documentation to determine the names of these packages. 2. With the packages installed, start by cleaning the source tree. This is a good procedure to ensure that there are no remnants of an old compile left hanging around. Here is an example of how to do this. # make mrproper 3. With the source tree clean, configure your kernel as required. The preferred method to do this is to use either make menuconfig (text based) or make xconfig (graphical; requires an X server). Both Red Hat and SuSE have their default kernel options selected when menuconfig / xconfig is launched after having done a make mrproper. From either interface, select the desired options. 4. After you configure the kernel, perform the compilation. Since RHEL 3 is based on the 2.4 kernel and SLES 9 is based on 2.6, the next steps are slightly different. Refer to Example 6-3 and Example 6-4 for each distribution.

Example 6-3 Compiling a kernel in Red Hat # make dep # make clean # make zImage # make modules # make modules_install # make install

Example 6-4 Compiling a kernel in SuSE # make # make modules_install # make install

108 The IBM Eserver BladeCenter JS20 5. After you compile the kernel, create the initial RAM disk. Both Red Hat and SuSE ship with a command called mkinitrd for this purpose. However, the syntax for each is slightly different, as shown in Example 6-5 and Example 6-6.

Example 6-5 mkinitrd for Red Hat # cd arch/ppc64/boot # mkinitrd -f ramdisk.image 2.4.21-15.ELcustom # gzip ramdisk.image # cd /usr/src/linux-2.4 # make zImage.initrd

Example 6-6 mkinitrd for SuSE # make install # mkinitrd # cd arch/ppc64/boot/ # cp /boot/initrd-2.6.5-7.39-pseries64 ramdisk.image # gzip ramdisk.image # cd /usr/src/linux # make zImage.initrd

Attention: The file names referred to are all relative to the kernel version strings within the Makefile for the kernel. The files on your system may be different.

After you complete these steps, you should have a bootable zImage.initrd file in /boot/ppc64/.

6.2 Configuring BOOTP and TFTP

There are three steps to network booting a POWER/PowerPC-based server. 1. Perform a Power On Self Test (POST) and then receive BOOTP information from BOOTP server, including the IP, Trivial File Transfer Protocol (TFTP) server IP, and zImage.initrd file name. 2. TFTP server sends the zImage.initrd file. 3. The server boots using the zImage.initrd file.

Both RHEL and SLES ship with Dynamic Host Configuration Protocol (DHCP) and TFTP packages. Ensure that you have installed the appropriate RPMs.

Chapter 6. Installing Linux 109 6.2.1 BOOTP

DHCP is an extension to the original BOOTP specification. As a result, you can use the DHCP daemon (dhcpd) to provide the BOOTP information for booting via the network.

The DHCP daemon is configured through the /etc/dhcpd.conf file. Example 6-7 shows the configuration that we used in our lab.

Attention: Keep in mind that Example 6-7 contains IP and MAC address information specific to our lab environment. See your product documentation or the dhcpd man page to customize this for your site.

With dhcpd configured, start the DHCP daemon as shown here: # /etc/init.d/dhcpd start

Example 6-7 /etc/dhcpd.conf always-reply-rfc1048 true; allow bootp;

deny unknown-clients; not authoritative;

default-lease-time 600; max-lease-time 7200;

subnet 192.168.1.0 netmask 255.255.255.0 { group { next-server 192.168.1.1;

host blade1 { fixed-address 192.168.1.117; hardware ethernet 00:0d:60:1e:0f:89; filename "zImage.initrd.suse"; }

host blade2 { fixed-address 192.168.1.118; hardware ethernet 00:0d:60:1e:0e:fb; filename "zImage.initrd.redhat"; }

} }

110 The IBM Eserver BladeCenter JS20 6.2.2 Trivial File Transfer Protocol

The TFTP daemon for both distributions considers the /tftpboot directory its home. Any file name statements made in the /etc/dhcpd.conf configuration file should be relative to this path.

Both RHEL and SLES spawn TFTP from the xinetd daemon. By default, tftpboot is disabled on both distributions. To enable both xinetd and TFTP, enter: [root@blade1 root]# chkconfig tftp on [root@blade1 root]# /etc/init.d/xinetd restart

Your installation server is now ready to network boot the BladeCenter JS20.

6.3 Preparing an unattended install

RHEL and SLES both have their own methods of conducting an unattended installation. RHEL uses Kickstart, and SLES uses a method known as AutoYaST.

Chapter 6. Installing Linux 111 6.3.1 Red Hat Kickstart

Red Hat provides a utility called redhat-config-kickstart to assist with the creation of Kickstart files. When you launch this utility, you see the Kickstart Configurator window (Figure 6-5).

Figure 6-5 redhat-config-kickstart main window

The separate sections of this application allow you to specify different attributes of the installation. Basic Configuration You can configure language support, keyboard, mouse, and root password here. You may also specify whether you prefer a text installation instead of a graphical installation. Lastly you may choose if the system should reboot after the automated installation is complete. Installation Method You can choose an upgrade or clean installation here. An upgrade usually allows the preservation of existing data by upgrading the installed applications

112 The IBM Eserver BladeCenter JS20 from a previous version of Red Hat. You can also specify the source of the installation (NFS, HTTP, CDROM, etc.) here.

Boot Loader Options Select the type of bootloader (GRUB or LILO) along with any relevant options

from here. You may also specify kernel parameters from this window. Partition Information You specify the partition size and type information here. You also specify the action to take with existing partitions and new disks. Network Configuration Define population network information, such as IP, DNS, and routers, from here. Authentication You may configure NIS, Keberos, and other authentication server options here. You may also specify local password encryption and security. Firewall Configuration Configure iptables rules from here. Xconfiguration Select the video card, resolution, monitor, and window manager here. Package Selection Select which programs are installed on the system here. Pre-Installation Script Identify the commands to run before the installation. Post-Installation Script Identify the commands to run after the installation.

When all your options are specified, save the file. This automatically checks for any missing options. If everything is correct, you may now use the ks.cfg file in an unattended installation.

Tip: We found that the redhat-config-kickstart program, by default, places a skipx parameter in the ks.cfg file. We had to comment out this prior to using ks.cfg. Otherwise, the installation would fail.

Chapter 6. Installing Linux 113 6.3.2 SuSE AutoYaST

SLES provides a YaST applet to assist with the creation and editing of AutoYAST files. You can launch it from the Misc section of YaST, as shown in Figure 6-6.

Figure 6-6 AutoYaST module within YaST

114 The IBM Eserver BladeCenter JS20 After the AutoYaST editor is created, you see a window similar to the one in Figure 6-7.

Figure 6-7 Main AutoYaST window

You can use the following sections to configure your AutoYaST file. Software Select the packages to install, and configure the automatic update client. Hardware

Configure partitioning, audio, printing, and X.

Chapter 6. Installing Linux 115 System

Set the general system information. You also configure the language, time zone, and other locale-related settings here. And specify boot loader, logging, and runlevel information here.

Network Devices Set network adapter information. Kernel module information as well as IP details can be set here. Network Services Network clients and daemons can be configured from here. There are over 15 different daemons to choose from. Refer to the Network Services section for more information. Security and Users From here, you may create users that you want to be present after install. You may also change the security policy to make things such as password changes more stringent. And you can configure VPN and firewall settings from here. Misc This section allows you to add complete configuration files, or to add special scripts to run before and after installation.

Tip: AutoYaST also lets you import Kickstart files from Red Hat or Alice (a precursor to AutoYaST) files. You do this by selecting File → Import.

After you are satisfied with the selections, save the file. This validates that the required minimum options are populated prior to saving.

6.4 Performing an unattended installation

The unattended installation process needs to pass the zImage.initrd the location of the unattended answer file during boot. Unfortunately, BOOTP does not have a facility to provide anything more than the location to the zImage.initrd and the tftpboot server to the client. To pass the required parameters, they need to be input into Open Firmware.

116 The IBM Eserver BladeCenter JS20 6.4.1 Open Firmware

For instructions about accessing the Open Firmware prompt, refer to “Activating the Open Firmware interfaces” on page 94.

When you reach the Open Firmware prompt, pass bootparm through the boot-file variable. Example 6-8 and Example 6-9 show the settings for SuSE and Red Hat respectively.

Example 6-8 Open Firmware settings for SuSE 0 > setenv boot-file autoyast=nfs://192.168.1.113:/mnt/SUSE-Source/autoyast.xml install=nfs://192.168.1.113:/mnt/SLES-Source/SLES9/ netdevice=eth1

Example 6-9 Open Firmware settings for Red Hat 0 > setenv boot-file ks=nfs:192.168.1.112:/mnt/install-images/ks.cfg ksdevice=eth1

After the installation is complete, remember to clear the boot-file value. If this value is not cleared, it is passed to the kernel as bootparm next time you boot. You can clear the value by using the nvsetenv command. We recommend that you run this command after the installation completes via either AutoYaST or Kickstart, as appropriate.

This command is provided by the ppc64-utils RPM in RHEL and the util-linux RPM in SLES. The syntax to clear the boot-file variable is: # /sbin/nvsetenv boot-file ""

6.4.2 mkzimage_cmdline: SuSE only mkzimage_cmdline is a utility contained in the SLES 9 lilo RPM. The SuSE kernel has special markers that allow mkzimage_cmdline to pass bootparm information directly through the zImage.initrd. Example 6-10 shows the syntax.

Example 6-10 mkzimage_cmdline # /lib/lilo/chrp/mkzimage_cmdline -a 1 -c -s "autoyast=nfs://192.168.1.113:/mnt/SUSE-Source/autoyast.xmlinstall=nfs://192.16 8.1.113:/mnt/SLES-Source/SLES9/ netdevice=eth1" netboot.suse.img # /lib/lilo/chrp/mkzimage_cmdline netboot.suse.img cmd_line size:512 cmd_line: autoyast=nfs://192.168.1.113:/mnt/SUSE-Source/autoyast.xml install=nfs://192.168.1.113:/mnt/SLES-Source/SLES9/ netdevice=eth1 active: 1

Chapter 6. Installing Linux 117 The syntax for mkzimage_cmdline is straightforward. When entered with only a file name parameter. The current bootparm is displayed. You can use the following options:

-h Display help.

-v Display version. -a 0|1 Disable/enable command line. Overrides bootparm passed from Open Firmware. -s STRING Stores STRING as bootparm in zImage.initrd. -c Clears previously stored bootparm from zImage.initrd before applying a new one.

6.5 Performing a network installation

When installing several machines, we recommend that you perform a network installation instead of a CD-based installation. The speed and ability to serve multiple machines makes it ideal for installing to a full complement of BladeCenter JS20s.

Attention: The cited examples are provided for illustration purposes only. For detailed instructions about how to install each distribution, refer to the product documentation.

6.5.1 SUSE LINUX Enterprise Server 9 For the SUSE LINUX Enterprise Server 9, follow this procedure: 1. After you load the zImage.initrd file, you see a language display similar to the one in Figure 6-8. In this example, we selected English.

118 The IBM Eserver BladeCenter JS20

Select the language.

1) Bosnia 2) Cestina 3) Deutsch 4) English 5) Español 6) Français 7) Hellenic 8) Italiano 9) Japanese 10) Magyar 11) Nederlands 12) Polski 13) Português 14) Português Brasileiro 15) Russian 16) Slovencina >

Figure 6-8 Language selection

2. After you select the language, the main installer display appears with a number of options. In this example, we chose to load additional kernel modules, as shown in Figure 6-9.

Main Menu

1) Settings 2) System Information 3) Kernel Modules (Hardware Drivers) 4) Start Installation or System 5) Eject CD 6) Exit or Reboot 7) Power off

Figure 6-9 Main menu

Chapter 6. Installing Linux 119 3. We select the install option and start the installation, as shown in Figure 6-10.

Start Installation or System

1) Start Installation or Update 2) Boot Installed System 3) Start Rescue System

Figure 6-10 Installation menu

4. We instruct the installer to use the network as a source for the installation, as shown in Figure 6-11.

Choose the source medium.

1) CD-ROM 2) Network 3) Hard Disk

Figure 6-11 Source menu

5. We chose NFS as the protocol to use from the menu in Figure 6-12.

Choose the network protocol.

1) FTP 2) HTTP 3) NFS 4) TFTP

Figure 6-12 Protocol menu

120 The IBM Eserver BladeCenter JS20 6. We chose the following site-specific information. You may have different information for your environment, but the general process is the same as shown in Figure 6-13.

Automatic configuration via DHCP?

1) Yes 2) No

> 1 Sending DHCP request...

Enter the IP address of the NFS server [192.168.1.113]>

Enter the directory on the server [/mnt/SLES/SLES9-RC1/]>

Figure 6-13 Installation location

With these steps complete the normal YaST process begins.

Tip: With this, the network specific portion of the installation is complete. Refer to the product documentation for further information.

6.5.2 Red Hat Enterprise Linux 3 After the zImage.initrd is served by the TFTP boot server, the installation process is the same as a CD-based installation. Refer to the product documentation for more information regarding this.

Chapter 6. Installing Linux 121

122 The IBM Eserver BladeCenter JS20

Chapter 7. Installing AIX on the JS20

This chapter discusses NIM. It explains how to configure NIM to install AIX and how to install AIX on an IBM Eserver BladeCenter JS20 using NIM.

BladeCenter JS20 supports the installation of AIX 5.2f and later. We recommend that you install AIX using a network installation via NIM. NIM is a unified console that allows the management of several AIX logical partitions (LPARs) or machines. You may install AIX on new machines or manage existing ones from a unified interface.

Tip: Prior to installing any operating system, we recommend that you upgrade all firmware to the latest level. See 5.4, “Firmware” on page 79, for instructions about how to complete this task.

To use NIM, there should be an available AIX instance with NIM configured on the same network segment as the BladeCenter. In the interest of speed and security, a private dedicated subnet is recommended.

Our NIM master was configured on a pSeries 690 connected to the same local area network (LAN) as the BladeCenter. This p690 was running the same level of AIX that we installed on the BladeCenter JS20.

Note: An installation of AIX may be several gigabytes in size. We recommend that you use a 100 Mb or faster network for NIM-based installations.

Follow these steps to install NIM: 1. Launch the Web-based Systems Manager (WebSM) and point to the server

that is running NIM. Select Configure the Network Installation Management environment from the NIM object shown in Figure 7-1.

Figure 7-1 NIM configuration window

2. In the window that opens (Figure 7-2), select Configure this host as NIM Master.

124 The IBM Eserver BladeCenter JS20

Figure 7-2 Configuring as a NIM Master

3. Select the network interface that is bound to the same network as the BladeCenter, and name it appropriately, as shown in Figure 7-3.

Figure 7-3 Selecting the NIM interface and network name

Chapter 7. Installing AIX on the JS20 125 4. In the next window (Figure 7-4), select Initialize Minimal NIM Environment.

Figure 7-4 Choosing to initialize minimal NIM environment

5. Click View Settings to confirm the gathered settings and complete the minimal NIM environment as shown in Figure 7-5.

Figure 7-5 Finalizing the NIM environment settings

126 The IBM Eserver BladeCenter JS20 7.2 Adding resources

Follow these steps to add resources:

1. With the minimal NIM environment complete, configure lpp_sources and a SPOT on the NIM master. Return to the NIM configuration window within WebSM and select Add a new resource to the Network Installation Management environment. 2. In the Add New Resource window (Figure 7-6), select the lpp_source & SPOT option.

Figure 7-6 lpp_source and SPOT configuration

Chapter 7. Installing AIX on the JS20 127 3. In the Configure NIM Environment window (Figure 7-7), select the medium that you will use for your AIX source. We use the DVD RAM drive.

Figure 7-7 AIX source

4. As shown in Figure 7-8, select the default values and begin populating the NIM resources.

Figure 7-8 Default resource settings

128 The IBM Eserver BladeCenter JS20 5. Click View Settings, as shown in Figure 7-9, to confirm the settings. Assuming a CD-based installation, insert the first CD and proceed to complete the addition of the new resources.

Figure 7-9 Completing the resource installation

Tip: This step make take some time to complete, since many files must be incorporated into the resource.

Chapter 7. Installing AIX on the JS20 129 7.3 Adding machine information

With the resources added, add information regarding the machine on which you

want to perform a base operating system (BOS) installation. 1. From the main NIM menu (Figure 7-10), select Add a new Machine to the Network Installation Management environment.

Figure 7-10 Adding new machine information to NIM

130 The IBM Eserver BladeCenter JS20 2. In the Add New Machine window (Figure 7-11), enter the host name of the server that you want to add to the NIM environment from this window.

Figure 7-11 Host name of new machine

Tip: The NIM server must be able to resolve the host name of the new server. Ensure that this host name can be resolved by DNS or a hosts file.

Chapter 7. Installing AIX on the JS20 131 3. In the next window (Figure 7-12), select Multiprocessor and confirm the other metrics of the machine that you are adding to the NIM environment.

Figure 7-12 Entering machine-specific information

132 The IBM Eserver BladeCenter JS20 4. As shown in Figure 7-13, enter the MAC address for the machine that you are adding to the Network adapter hardware address field and proceed to complete the addition of the new machine to the NIM environment.

Figure 7-13 Entering MAC address information

With the NIM resources created and the machine object created, simply proceed with the installation.

NIM allows you to install new operating systems on pSeries servers by booting the machine via BOOTP. This diskless or network installation makes it simple to install new operating systems to servers in a matter of minutes.

Chapter 7. Installing AIX on the JS20 133 7.4 Preparing the NIM master

With all the NIM resources in place, it is simple to begin the installation of a new

operating system to the target machine, in this case a BladeCenter JS20. 1. Launch WebSM and view the configured machine objects from the NIM section. 2. Right-click the object for the desired machine and select Install Operating System, as shown in Figure 7-14.

Figure 7-14 Installing an operating system from NIM

3. Select the location of the SPOT and lpp_source resources created earlier.

134 The IBM Eserver BladeCenter JS20 4. Proceed with the installation by setting the BOS preferences as shown in Figure 7-15.

Figure 7-15 Setting BOS preferences

Tip: To fully automate the AIX installation, create a bosinst_data resource and associate it with the operating system installation in NIM settings. Although this is beyond the scope of this book, you can find instructions in the AIX product manuals.

7.5 Preparing the client

With the NIM server configuration complete, simply boot the BladeCenter JS20 from the network as explained in 5.5.3, “Open Firmware interfaces” on page 94.

When the NIM master supplies the boot image via TFTP, the AIX installation begins just as though you booted from the CD.

Chapter 7. Installing AIX on the JS20 135 1. On the first panel, select the desired language. We selected English, as shown in Figure 7-16.

******* Please define the System Console. *******

Type a 1 and press Enter to use this terminal as the system console. Pour definir ce terminal comme console systeme, appuyez sur 1 puis sur Entree. Taste 1 und anschliessend die Eingabetaste druecken, um diese Datenstation als Systemkonsole zu verwenden. Premere il tasto 1 ed Invio per usare questo terminal come console. Escriba 1 y pulse Intro para utilizar esta terminal como consola del sistema. Escriviu 1 1 i premeu Intro per utilitzar aquest terminal com a consola del sistema. Digite um 1 e pressione Enter para utilizar este terminal como console do sistema.

Figure 7-16 Installation language panel

2. In the next panel (Figure 7-17), you may proceed with the default installation options or customize them. We chose to customize them.

Welcome to Base Operating System Installation and Maintenance

Type the number of your choice and press Enter. Choice is indicated by >>>.

>>> 1 Start Install Now with Default Settings

2 Change/Show Installation Settings and Install

3 Start Maintenance Mode for System Recovery

88 Help ? 99 Previous Menu >>> Choice [1]:

Figure 7-17 Operating system selection

136 The IBM Eserver BladeCenter JS20 3. In the Installation and Settings window (Figure 7-18), we proceeded with the default installation type and language environment values.

Installation and Settings

Either type 0 and press Enter to install with current settings, or type the number of the setting you want to change and press Enter.

1 System Settings: Method of Installation...... Preservation Disk Where You Want to Install.....hdisk0

2 Primary Language Environment Settings (AFTER Install): Cultural Convention...... C (POSIX) Language...... C (POSIX) Keyboard...... C (POSIX)

3 More Options (Desktop, Security, Kernel, Software, ...)

>>> 0 Install with the settings listed above.

+------88 Help ? | WARNING: Base Operating System Installation will 99 Previous Menu |destroy or impair recovery of SOME data on the |destination disk hdisk0. >>> Choice [0]:

Figure 7-18 Installation and language options

Chapter 7. Installing AIX on the JS20 137 4. As illustrated in Figure 7-19, we elected to include additional options such as the 64-bit kernel and JFS2. We then selected 0 and began the installation process.

Install Options

1. Enable Trusted Computing Base...... no 2. Enable CAPP and EAL4+ Technology...... no (English only, 64-bit kernel enablement, JFS2 file systems) 3. Enable 64-bit Kernel...... yes 4. Create JFS2 File Systems...... yes

5. Graphics Software...... yes 6. Documentation Services Software...... no 7. Enable System Backups to install any system...... no (Installs all devices and kernels)

>>> 8. Install More Software

0 Install with the settings listed above.

88 Help ? 99 Previous Menu

>>> Choice [8]:

Figure 7-19 Kernel options

Tip: As with any operating system, AIX enables you to customize many different options during the installation process. We have provided a sample of what you can select.

138 The IBM Eserver BladeCenter JS20

Chapter 8. System management scenarios

Your installation may be comprised of a single BladeCenter chassis with only a few BladeCenter JS20s. If so, you may find that the capabilities provided by the BladeCenter Management Module Web interface and command line interface (CLI) are sufficient to meet your system management requirements. For larger installations comprised of multiple BladeCenter chassis and many BladeCenter JS20s, consider using IBM Director, IBM Cluster Systems Management (CSM), or both.

We demonstrate how to use many of the functions provided by the BladeCenter Management Module Web interface and CLI in Chapter 5, “Hardware setup” on page 69.

This chapter provides information about setting up and configuring the IBM Director and the IBM Cluster Systems Management facilities. It also describes several systems management scenarios for the BladeCenter JS20.

IBM Director is a comprehensive systems-management solution. A powerful suite of tools and utilities, IBM Director automates many of the processes required to manage systems proactively, including preventive maintenance, diagnostic monitoring, troubleshooting, and more. It offers a graphical user interface (GUI) that provides system administrators easy access to both local and remote systems. IBM Director can be used in environments with multiple operating systems (heterogeneous environments).

At the time of writing, the current version of IBM Director was Version 4.2.

IBM Director is the industry-leading client/server workgroup manager. IBM Director’s tools provide customers with flexible capabilities to realize maximum system availability and lower IT costs. With IBM Director, IT administrators can view and track the hardware configuration of remote systems in detail and monitor the usage and performance of critical components, such as processors, disks, and memory.

Extensions to IBM Director are available for clients who want advanced capabilities. These extensions include: Server Plus Pack Software Distribution Premium Edition Remote Deployment Manager Application Workload Manager

Some of the features that you find in IBM Director include: Self-managing, smart tools that are automated, and proactive capabilities that reduce IT costs and maximize uptime Support for non-IBM hardware Innovative use of industry standards from Common Information Model (CIM) to Simple Network Mail Protocol (SNMP) enables heterogeneous hardware management, protecting your existing IT investment. Seamless Integration IBM Director protects your investments in other management packages, complementing them with more extensive hardware manageability. Single-click management GUI A convenient user interface delivers the ability to drag and drop tasks to specific systems or groups of systems.

140 The IBM Eserver BladeCenter JS20 Integrated, centralized SQL database

An internal database makes system-related data available, even when the specific system is not directly available. Multiple operating system support

IBM Director smoothly handles a variety of popular operating systems. Comprehensive BladeCenter support IBM Director provides an easy, single point of deployment and management of new blade server architectures. Consistent framework that can be extended with plug-ins for advanced management CLI support for many IBM Director management tasks

IBM Director 4.20 has many new features, support for new operating systems, and enhancements to existing functions. Together, these changes provide significantly richer and broader capabilities for superior hardware management.

Key enhancements include: Expanded Linux support including aggregated, color-coded systems health status for systems running Linux Convenient subscription service included with the IBM Director license to proactively deliver the latest product updates Enhanced optional Server Plus Pack, a collection of advanced tools to help optimize server performance and availability Some of the new features include XML report format for the Capacity Manager and System Availability tools, the ability to schedule System Availability reports, and Active PCI Manager support for Linux. The latest hardware support including support for additional hardware industry standards for manageability, ASF 2.0 and IPMI, to further facilitate the management of heterogeneous hardware environments A multitude of new features and refinements to make IBM Director even easier to install and use

Chapter 8. System management scenarios 141 8.1.1 Setting up an IBM Director Server

Before Director clients can be installed, you must first configure the Director server. The following instructions explain how we set up our server.

1. Mount the Director CD. sysmgr:/ # mount -t iso9660 -o map=off /dev/cdrom /media/cdrom

Important: It is critical that the CD is mounted in this manner. If you do not specify -o map=off, the capitalization of RPM names is not preserved and the installation script fails.

2. With the CD mounted, copy the contents of the appropriate directory to the hard drive of the server. sysmgr:/root/Director-Source/ # cp /media/cdrom/director/server/linux/i386/* . 3. With the sources copied to the hard drive, launch the installer. sysmgr:/root/Director-Source/ # ./dirinstall 4. After the Director code is installed, configure a database for it to use. Because PostgreSQL ships as a standard part of SLES, we chose to use it. Example 8-1 shows how we configured it.

Example 8-1 Configuring PostgreSQL sysmgr:/root/Director-Source/ # cd /usr/share/pgsql sysmgr:/usr/share/pgsql/ # ln -s jdbc7.1-1.2.jar postgresql.jar sysmgr:/usr/share/pgsql/ # cd /etc/TWGserver/ sysmgr:/etc/TWGServer/ # echo export CLASSPATH=/usr/share/pgsql/psotgresql.jar > setup_env sysmgr:/etc/TWGServer/ # chmod u+x setup_env

Important: We assume that PostgreSQL is configured and running properly on the system. Refer to the documentation regarding how to do this.

5. By default, PostgreSQL on SLES does not bind to a TCP/IP socket. To enable this, modify /etc/sysconfig/postgresql and add the following line: POSTGRES_OPTIONS="-i"

6. Begin the database configuration, as shown in Example 8-2. Example 8-2 Launching the database configuration sysmgr:/etc/TWGServer/ # passwd postgres Changing password for postgres

142 The IBM Eserver BladeCenter JS20 New password: Re-enter new password: Password changed sysmgr:/etc/TWGServer/ # cd /opt/IBM/director/bin/ sysmgr:/opt/IBM/director/bin # ./cfgdb

7. The installer launches. In the first window (Figure 8-1), select PostgreSQL.

Figure 8-1 Selecting the type of database

Chapter 8. System management scenarios 143 8. Enter the database connection information, as shown in Figure 8-2. Click Next to complete the configuration of the database.

Figure 8-2 Database connection information

144 The IBM Eserver BladeCenter JS20 9. We decided to use encrypted communication for our IBM Director environment. To do this, execute cfgsecurity as shown in Figure 8-3.

sysmgr:/opt/IBM/director/bin # cfgsecurity

Do you want IBM Director to encrypt data communications on this system? 0 - No 1 - Yes

Enter number corresponding to desired security setting (0 is default): 1

Which encryption algorithm do you want to use? 1 - DES encryption (56-bit key) 2 - Triple DES encryption (128-bit key)

0 - No encryption 2

Encryption set or unset successfully.

All IBM Director servers and encrypted systems are set to "secured" automatically. Setting secured or unsecured system successful.

Figure 8-3 Setting Director server encryption

10.After you configure the database and security, start the Director daemon. sysmgr:/opt/IBM/director/bin # ./twgstart

With the daemon started, the Director server configuration is now complete.

8.1.2 Installing an IBM Director Agent Follow these steps to install an IBM Director Agent. 1. Mount the Director CD and copy the files as explained in steps 1 and 2 on page 142. This time, substitute /media/cdrom/director/server/linux/i386/ with /media/cdrom/director/agent/linux/ppc/ in step 2.

Chapter 8. System management scenarios 145 2. With the sources copied to the hard drive, launch the installer, as shown in Figure 8-4.

blade:/root/Director-Source/ # ./dirinstall

Attempting to install ITDAgent-4.12-1.ppc.rpm Preparing... ################################################# ITDAgent ################################################# To start the IBM Director Agent manually, run /opt/ibm/director/bin/twgstart

Installation of selected components is successful. To start the IBM Director Agent manually, run /opt/ibm/director/bin/twgstart

Figure 8-4 Installing the Director Agent

3. After the installation is complete, configure security as desired. We chose to use encryption, as shown in Figure 8-5.

blade:/etc/TWGServer/ # cd /opt/IBM/director/bin/ blade:/opt/IBM/director/bin/ # cfgsecurity

Do you want IBM Director to encrypt data communications on this system? 0 - No 1 - Yes

Enter number corresponding to desired security setting (0 is default): 1 Encryption set or unset successfully.

All IBM Director servers and encrypted systems are set to "secured" automatically. Setting secured or unsecured system successful.

Figure 8-5 Configuring Director Agent security

4. With the security set, start the Director Agent. blade:/opt/IBM/director/bin # ./twgstart

146 The IBM Eserver BladeCenter JS20 8.1.3 Using Director to manage JS20s

With the agent installed, you can add the BladeCenter JS20 to the Director console as another managed server. For details about how to do this, see the IBM Director 4.12 Installation and Configuration Guide, SC09-P290-50.

The following subset of Director management features are available for the BladeCenter JS20: Software Distribution Standard Edition Software Distribution Premium Edition Update Assistant Inventory Hardware Inventory Software Event Action PLans Hardware Health Status Resource Monitors Eventlog Agent Discovery Process Management SNMP Alerts SNMP Browser Rack Manager Remote Session File Transfer

8.2 IBM Cluster Systems Management

BladeCenter JS20 installations may involve clusters of many blade servers. Many of these installations use IBM Cluster Systems Management to both install and manage BladeCenter JS20s.

CSM for Linux has been under development since early 2000 and was first released as a product in June 2001. The original requirement for this product was to provide common cluster management for Linux and AIX, leveraging IBM’s existing cluster software portfolio to work with Open Source packages and best practices from the Linux cluster market.

CSM provides basic cluster management functions, such as hardware control and operating system and software installation. It also provides significant value-added functions such as configuration file management, a robust monitoring infrastructure, automated event management, diagnostic probes, and common management of AIX and Linux clusters.

Chapter 8. System management scenarios 147 As CSM was being developed, the Extreme Cluster Administration Toolkit (xCAT), a script-based package, was developed by IBM’s advanced technical sales support team. It was provided to clients who purchase Linux clusters based on IBM servers. It was also used by IBM Global Services to address their need for tools to deploy and manage Linux clusters. In the interim, xCAT became a proving ground for new concepts and practices centered on the management of Linux clusters. It can be a model for collaborative development between IBM and our clients.

At the time of the development of this book, CSM was available in its fourth release (V1.4.0.4) with expanded scalability and hardware support. Many xCAT functions have been integrated into CSM, along with utilities to automate the process of migrating from xCAT to CSM. Through the use of CSM, clients who have implemented xCAT can now have many of these functions with full product support from IBM.

The key benefits of CSM can help to reduce costs, provide higher availability of the cluster for productive use, and improve system utilization. Customers who have existing AIX 5L operating system-based cluster systems can leverage those skills to manage their Linux clusters.

System administrators can automate repetitive installation and configuration tasks and automate problem determination and recovery. They can monitor and report health information and resource utilization, and automatically recover from node, storage, or network failures. This can lead to overall simplification of cluster administration.

This section describes a fast-path CSM implementation to install and manage BladeCenter JS20s. For a more comprehensive view, refer to the CSM for Linux Planning and Installation Guide, SA22-7853.

Our implementation of CSM assumes the type of network configuration described in 4.1.1, “Minimal network requirements” on page 54. Review this along with other CSM planning considerations described in 4.3.2, “IBM Cluster Systems Management” on page 64.

8.2.1 Setting up a CSM management node The first step in installing CSM is to set up a management node. We focus on how to set up a management node based on an xSeries server. We believe this is a common scenario for users of BladeCenter JS20s, particularly if they also have other blade servers that use Intel central processing units (CPUs).

Using an xSeries server to install and manage BladeCenter JS20s cluster nodes introduces some complications that are not required when the management

148 The IBM Eserver BladeCenter JS20 node is the same architecture as the compute nodes. We focus our discussion around these issues.

We use CSM to both install and manage BladeCenter JS20s. Therefore, we must use a compatible operating system on both the management node and the

BladeCenter JS20 cluster nodes. The steps we require to set up the CSM management node are: 1. Install a supported operating system. 2. Configure network interfaces and set up host name resolution. 3. Obtain the latest CSM software and prerequisites. 4. Install the CSM base RPM. 5. Run the CSM installation program. 6. Copy the operating system, service packs, CSM code, and prerequisites for BladeCenter JS20 compute nodes. 7. Accept the CSM license. 8. Set up hardware control point passwords. 9. Verify the installation.

We review each of these steps in the sections that follow. For our tests, we performed a standard installation of SUSE LINUX Enterprise Server (SLES) 9 for x86 on our management node directly from the distribution CDs.

The management node network configuration Our implementation of CSM assumes the type of network configuration described in 4.1.1, “Minimal network requirements” on page 54.

The CSM management node is connected to both the hardware management subnet and the operating system management subnet using separate network interfaces. You can use additional network interfaces to connect the management node to other networks if required.

In our scenario, we used the IP address ranges 9.3.5/24 for the hardware management subnet and 192.168.1/24 for the operating system management subnet. There should be no other active DHCP servers on the operating system management subnet, because the CSM management node performs the DHCP server function on that subnet.

Define host names and allocate IP addresses for the following interfaces before you attempt to use CSM: The management node network interface that connects to the operating system management subnet

Chapter 8. System management scenarios 149 This is the interface that will be used to install the operating system on the BladeCenter JS20 cluster nodes.

The network interface on each BladeCenter JS20 that connects to the operating system management subnet

The external network interface of each BladeCenter Management Module

You can either set up a hosts file on the management node or use a Domain Name System (DNS) server to define the host name to IP address mapping. In our scenario, we used a hosts file as shown in Example 8-3.

Example 8-3 /etc/hosts on management node 127.0.0.1 localhost 192.168.1.20 mgr 192.168.1.117 blade1 192.168.1.118 blade2 192.168.1.119 blade3 9.3.5.190 bcmm

Obtaining the latest CSM software and prerequisites You can obtain the latest CSM software from: http://techsupport.services.ibm.com/server/cluster/fixes/csmfixhome.html

Check this Web site for the latest CSM software even if you have CSM available on CD, because new CSM software releases occur every few months.

The CSM software available at the previous Web site is under a 60-day Try and Buy license. If you are performing a permanent CSM installation, you need to have purchased an appropriate quantity of CSM licenses. You also need the full license file that is contained on the CSM product CD.

We downloaded two different versions of CSM: one for the management node (xSeries based) and one for the BladeCenter JS20 cluster nodes. The two versions of CSM that we downloaded were: CSM for Linux on xSeries V1.4.0.4 CSM for Linux on pSeries V1.4.0.4

Follow these steps: 1. Type the following command: mkdir /tmp/csm_i386 2. Now type this command: mkdir /tmp/csm_js20

150 The IBM Eserver BladeCenter JS20 3. Download the two CSM tarballs into their respective directories created in steps 1 and 2.

4. Change to /tmp/csm_i386 and run the following command: tar -xvzf csm-linux-1.4.0.0.i386.tar.gz

5. Change to /tmp/csm_js20 and run the following command: tar -xvzf csm-linux-1.4.0.ppc64.tar.gz 6. We created a directory for such prerequisite packages as Autoupdate called /tmp/csmpreq. Autoupdate is a package that is used by CSM. We downloaded Version 5.3.11-1 from the following URL and placed it into the /tmp/csmpreq directory. http://ftp.mat.univie.ac.at/pub/teschl/autoupdate/index.html

Installing core CSM RPM You must be logged into the management node as root to perform this step.

We installed the core CSM RPM on the management node using this command: # rpm -ivh /tmp/csm_i386/csm.core*.i386.rpm

When you run the rpm command and install the CSM core, CSM sets the $PATH and $MANPATH environment variables. Therefore, you must log out and then log in again to the management node after performing the installation to accept the changes to the environment variables that are required when performing subsequent installation steps.

Running CSM installation program After logging back into the management node as root, we ran the CSM installation program using the following command: # installms -p /tmp/csm_i386:/tmp/csmpreq

Note: The installms command is found in the /opt/csm/bin directory after you install the CMS core.

The -p option of the installms command allows you to specify the location where the CSM installation program should search for the files it needs. We specified the two directories where we placed the CSM software and prerequisites for the management node hardware architecture that we were using.

The installms command prompts you to insert operating system CDs if it cannot find the operating system packages that it needs in one of the directories listed after the -p option. You are also prompted to replace the Trivial File Transfer Protocol (TFTP) daemon provided with the Linux distribution with a replacement

Chapter 8. System management scenarios 151 version packaged with CSM. And you are prompted to accept the licenses associated with some software packages distributed with CSM.

Make sure you resolve any errors reported by the installms command before you continue with the next step.

Copying the code required for BladeCenter JS20s The BladeCenter JS20 has a different hardware architecture than the xSeries server that we used for our management node. Therefore, you must copy the code required for the BladeCenter JS20s to the management node. This includes: SUSE LINUX Enterprise Server 9 for IBM POWER CSM for Linux on pSeries CSM prerequisites

We used the CSM copycds and copycsmpkgs commands to copy the operating system, service pack, CSM software, and prerequisites for the BladeCenter JS20 cluster nodes to the management node. Both of these commands require you to specify any attributes that are different to the operating system environment on the management node. In our scenario, we specified the InstallPkgArchitecture attribute, because the blade server cluster nodes have a different architecture than the management node.

We copied the operating system CDs using the following command: # copycds InstallPkgArchitecture=ppc64

We copied the CSM software and prerequisites for the BladeCenter JS20 cluster nodes using this command: # copycsmpkgs -p /tmp/csm_js20:/tmp/csmpreq InstallPkgArchitecture=ppc64

The copycsmpkgs command prompts you to insert operating system CDs if it cannot find the operating system packages that it needs in one of the directories listed after the -p option.

Accepting the CSM licence Accept the CSM licence. If you want to use a 60-day trial licence, use the following command: # csmconfig -L

If you want to accept the permanent license, mount the CSM product CD and run the following command: # csmconfig -L /media/cdrom/csmlum.full

152 The IBM Eserver BladeCenter JS20 Setting up hardware control point passwords CSM requires user IDs and passwords to communicate with BladeCenter Management Modules so that it can: Control power to BladeCenter JS20s Provide remote console access to BladeCenter JS20s Retrieve the MAC addresses for the Gigabit Ethernet interfaces on BladeCenter JS20s

CSM lets you define one user ID and password for power control and a different user ID and password for remote console access for each BladeCenter Management Module. Both of these profiles must match login profiles that are defined in the corresponding BladeCenter Management Module. You can configure these profiles through the BladeCenter Management Module Web interface.

In a production environment, you should define two different login profiles, one for power control and the other for remote console access. You should also limit the access for each of these profiles to the minimum needed. In our test environment, we used the same profile for both power control and remote console access.

We recommend that you use the same user ID and password for power control for all BladeCenter Management Modules in a cluster and define these as the default user ID and password for power control in CSM.

You define a default BladeCenter Management Module user ID and password for power control using the following command: # systemid -p blade USERID

The systemid command prompts you to enter the password associated with the default user ID you specified.

Similarly, we recommend that you use the same user ID and password for remote console access for all BladeCenter Management Modules in a cluster and define these as the default user ID and password for remote console access in CSM.

You define a default BladeCenter Management Module user ID and password for remote console access using the following command: # systemid -c -p blade USERID

The systemid command prompts you to enter the password associated with the default user ID you specified. This step is necessary for the Serial over LAN (SoL) access.

Chapter 8. System management scenarios 153 You can verify what user IDs you configured as the default for BladeCenter power control and remote console access by issuing the systemid command with no parameters. This displays output similar to what is shown in Example 8-4.

Example 8-4 Output from systemid command

blade_#DEFAULT# USERID blade_#DEFAULT#_console USERID

Verifying the installation CSM is provided with an installation verification probe that can pick up many installation errors. You can run this probe by using the following command: # probemgr -p ibm.csm.ms -l 0

Review the output of this command and investigate any error messages.

8.2.2 Installing and managing BladeCenter JS20s using CSM After you establish a CSM management node, you can install and manage BladeCenter JS20s.

The required steps are: 1. Define your BladeCenter JS20 cluster nodes to CSM. 2. Set up the installation environment. 3. Start installation and wait for completion.

We examine each of these steps in more detail in the sections that follow.

Defining the BladeCenter JS20 cluster nodes We find that the easiest way to define BladeCenter JS20 cluster nodes to CSM is to use a node definition file. There are many other ways to define nodes in CSM. Review the CSM for Linux Planning and Installation Guide, SA22-7853, to determine the most appropriate method for your environment.

The node definition file that we used for our small cluster of BladeCenter JS20s is shown in Example 8-5.

Example 8-5 CSM node definition file default: ManagementServer=mgr PowerMethod=blade HWControlPoint=bcmm ConsoleMethod=blade ConsoleServerName=bcmm

154 The IBM Eserver BladeCenter JS20 InstallDiskType=ide InstallPkgArchitecture=ppc64 InstallAdapterName=eth1 blade1: HWControlNodeId=blade1 ConsolePortNum=1 blade2: HWControlNodeId=blade2 ConsolePortNum=2 blade3: HWControlNodeId=blade3 ConsolePortNum=3

The default stanza of the node definition file contains the node attributes that are common to all the BladeCenter JS20 cluster nodes in our cluster. You can have multiple default stanzas in a node definition file, with the new defaults applying to those stanzas listed in the node definition file after the new default stanza.

The remaining stanzas define each BladeCenter JS20 cluster node, and contain only those node attributes that are unique to that cluster node. The name of the stanza for each BladeCenter JS20 cluster node should match the host name that you assigned to that cluster node in your hosts file or DNS.

Use the following guide when you specify attribute values: ManagementServer: Host name of management node network interface connected to the operating system management subnet HWControlPoint: Host name of the BladeCenter Management Module ConsoleServerName: Host name of the BladeCenter Management Module InstallAdapterName: Set to eth1 if you are using the network configuration described in 4.1.1, “Minimal network requirements” on page 54 This network configuration performs operating system network installation via eth1 to prevent issues with the SoL remote text console function, which uses eth0. HWControlNodeId: Same name you assigned the BladeCenter JS20 cluster node when you set up the hardware

We recommend that you make it the same as the host name. We demonstrate how to configure this in 5.3, “Blade server configuration” on page 76. ConsolePortNum: Bay in the BladeCenter chassis where the BladeCenter JS20 cluster node is installed

Chapter 8. System management scenarios 155 PowerMethod: Should always be blade for BladeCenter JS20 cluster nodes

ConsoleMethod: Should always be blade for BladeCenter JS20 cluster nodes

After you create a node definition file for your cluster, you can define the nodes to CSM using the following command: # definenode -f /tmp/nodedefs

After the definenode command runs, the management server is set up with all the node information for CSM, and you are ready to verify the node definitions. Since the actual node installation has not happened yet, you can make changes to any of the node definitions at this time. To determine whether the nodes have been defined, enter the lsnode command from the management server.

After you run the lsnode command, the system responds with a line for each node that was successfully defined. If a node is not defined, it does not appear in the output for the lsnode command.

To display all the information about each node, use the lsnode command from the management server with the –l (lowercase L) option, such as: # lsnode -l

The system responds with a list (output) that contains the extended information for each node that was successfully defined. If a node is not defined, it does not appear in the output for the lsnode command.

Some of the attributes for a node may have null values at this point. If a node is not defined, it is not installed. If you must correct the information, you can change the attributes of the nodes either by running the chnode command, or by rerunning the definenode command with the -m (modify) flag. The definenode -m command accepts a new nodedef file or a new command line and only modifies nodes that have changed attributes. To change attribute values of a node, enter the chnode command from the management server: chnode hostname attr=value

Setting up the installation environment After you successfully define the node, configure how the node is to be installed in the cluster. For Red Hat AS or Red Hat EL 3 installations, use Kickstart, and run the csmsetupks command. For SUSE LINUX Enterprise Server installations, use AutoYaST, and run the csmsetupyast command. The csmsetupks and csmsetupyast commands configure the install process, collect MAC addresses, and configure the network boot of the nodes.

156 The IBM Eserver BladeCenter JS20 In our lab environment using SuSE, the csmsetupyast did several things, including:

Retrieved the MAC addresses for the Gigabit Ethernet interfaces of each BladeCenter JS20 cluster node

Built a Autoyast configuration file for each BladeCenter JS20 cluster node from a template Built the DHCP server configuration file /etc/dhcpd.conf

Before you attempt to use csmsetupyast for the first time, we recommend that you verify that power control and remote console functionality is working correctly. You can verify that power control is working using the following command: # rpower -a query

This command displays the current power status of all nodes in the cluster. If this is not successful, determine why before you proceed any further.

You can verify that the remote consoles are working by using the following command: # rconsole -a

This command opens a separate console window for every cluster node that is defined to CSM. If a BladeCenter JS20 cluster node is not powered on, you see messages indicating that the SoL is not ready. Close each remote console window using the key sequence Ctrl+E c . before you continue further.

When you are satisfied that power control and remote console functionality is working correctly, verify that the DHCP server on the management node is configured to listen on the correct network interface.

Under SUSE LINUX Enterprise Server, the interfaces that the DHCP server uses to listen are configured in the file /etc/sysconfig/dhcpd. Set the DHCP_INTERFACE variable in this file to include the Ethernet interface on the management node that is connected to the operating system management subnet described in “Operating system management subnet” on page 57.

Now set up the installation environment using the CSM command: # csmsetupyast -x -P

We use the -x option because we already copied the necessary files to support the network installation of the operating system to the management node. If you do not specify this option, you are prompted to insert operating system CDs. We use the -P option to specify all nodes that are in the premanaged state and are waiting to be installed.

Chapter 8. System management scenarios 157 The csmsetupyast command uses an XML template file found in /opt/csm/install as a default template for the Autoyast configuration files that it creates for each BladeCenter JS20 cluster node. Depending upon the release of SuSE that you are using at the time, the file has the format: yastcfg.InstallDistributionNameInstallDistributionVersion-Arch.xml

For example, in our lab environment, the file was called yastcfg.SLES9-ppc64.xml.

If you want to use a different template, first create the template and then specify it on the csmsetupyast command using the -k option. We recommend that you copy the default template when creating new templates to ensure that you include all the elements that CSM requires in the template.

Review the output of the csmsetupyast command carefully and resolve any errors before you proceed further.

Starting the cluster node installation Start the cluster node installation using the installnode command. We recommend that you try to install a few nodes at first and make sure it works before you try to install many the nodes concurrently. To start installation of a single node, use the command: installnode -n blade1

The installnode command takes a few minutes to complete when used with BladeCenter JS20 cluster nodes. This is because the installnode command must wait until the BladeCenter JS20 boot process has reached the point where it can pass network installation parameters to the firmware.

When the installation is underway, the installnode command exits and displays the status for each node it was asked to install. You can monitor the status of the install as it progresses using the monitorinstall command. You can also open a remote console to watch the installation progress using the rconsole command.

For more information about installing CSM, see CSM for Linux: Planning and Installation Guide, SA22-7853.

158 The IBM Eserver BladeCenter JS20 Abbreviations and acronyms

ac alternating current GPR General Purpose Register AIX Advanced Interactive GUI Graphical User Interface eXecutive HTTP Hypertext Transfer Protocol BIU Bus Interface Unit HTTPS Hypertext Transfer BOOTP Bootstrap Protocol Protocol-Secure BOS base operating system I/O input/output BPU branch processing unit IBM International Business BSMP Blade System Management Machines Corporation Processor IGESM CISCO Intelligent Gigabit CISC complex instruction set Ethernet Switch Module computer IP Internet Protocol CLI command line interface IPL Initial Program Load (Boot) CRLF carriage return/line feed ITSO International Technical CSM Cluster Systems Support Organization Management KVM keyboard, video and mouse CTR count register LAN local area network DC direct current LPAR logical partition DDR double data rate LR Link Register DHCP Dynamic Host Configuration MAC Media Access Protocol Mb megabit DNS Domain Name Server MB megabyte ECC Error Correction Code MFLOPS Million Floating Point ERAT effective to real address Operations Per Second translation MP multiprocessor ESM Ethernet Switch Module NFS Network File System FPR Floating Point Register NIC Network Interface Controller FPU Floating Point Unit NIM Network Install Manager FSB Front Side Bus PKT packet FTP File Transfer Protocol POSIX Portable Operating System for FXU Fixed Point Unit UNIX

Gb gigabit POST Power On Self Test GB gigabyte POWER Performance Optimization gcc GNU C Compiler With Enhanced RISC RAM random access memory

RISC Reduced Instruction Set Computer ROM Read Only Memory

RTC Real Time Clock SAN storage area network SCSI small computer systems interface SMP Simultaneous Multi-Processor SOL Serial over LAN SPOT Shared Product Object Tree SSH Secure Shell TCP/IP Transmission Control Protocol/Internet Protocol TFTP Trivial File Transfer Protocol TLB Translation Lookaside Buffer USB Universal Serial Bus UXBC UpdateXpress firmware update scripts for BladeCenter VLAN virtual local area network VMX Vector Multimedia eXtensions VPN Virtual Private Network VXU Vector Processing Unit WAN Wide Area Network XER Fixed Point Exception Register XML eXtended Markup Language YAST Yet Another Setup Tool (SuSE)

160 The IBM Eserver BladeCenter JS20 Related publications

The publications listed in this section are considered particularly suitable for a more detailed discussion of the topics covered in this redbook.

IBM Redbooks

For information about ordering these publications, see “How to get IBM Redbooks” on page 164. Note that some of the documents referenced here may be available in softcopy only. Implementing System Management Solutions using IBM Director, SG24-6188 The Cutting Edge: IBM Eserver BladeCenter, REDP-3581 IBM Eserver BladeCenter Systems Management, REDP-3582 Deploying Lotus Domino on IBM Eserver BladeCenter, REDP-3584 Deploying Apache on IBM Eserver BladeCenter, REDP-3588 Deploying Samba on IBM Eserver BladeCenter, REDP-3595 IBM Eserver BladeCenter Networking Options, REDP-3660 IBM Eserver BladeCenter Layer 2-7 Network Switching, REDP-3755 IBM Eserver BladeCenter Systems Management with IBM Director V4.1 and Remote Deployment Manager 4.1, REDP-3776 IBM Eserver BladeCenter Configuration Tips, TIPS-0454

Other publications

These publications are also relevant as further information sources: Peterson and Davie, Computer Networks: A Systems Approach, 3rd Edition, Morgan Kaufmann, 2003, ISBN 1-55-860832-3 Stevens, TCP/IP Illustrated, Volume 1: The Protocols, Addison-Wesley, 1993, ISBN 0-20-163346-9 Tanenbaum, Computer Networks, 4th Edition, Prentice Hall, 2002, ISBN 0-13-066102-3 Tanenbaum, Modern Operating Systems, 2nd Edition, Prentice Hall, 2001, ISBN 0-13-031358-0

http://publib.boulder.ibm.com/epubs/pdf/am7il103.pdf IBM Cluster Systems Management for Linux Planning and Installation Guide, SA22-7873

IBM Cluster Systems Management for AIX5L Planning and Installation Guide, SA22-7919 IBM Director 4.12 Installation and Configuration Guide, SC09- P290-50 IBM Director 4.0 for BladeCenter Products: Systems Management Guide, SC01-R051-40 CSM for Linux Planning and Installation Guide, SA22-7853

Online resources

These Web sites and URLs are also relevant as further information sources: Installing AIX 5L on the IBM Sserver BladeCenter JS20 whitepaper http://www.ibm.com/servers/aix/whitepapers/aix_js20.pdf IBM Eserver Support Web site http://www.ibm.com/pc/support Red Hat Linux http://www.redhat.com SUSE LINUX http://www.suse.com Instructions for installing SuSE from CD http://www.ibm.com/pc/support/site.wss/document.do?sitestyle=ibm&lndocid= MIGR-54819 PowerPC 970 Training http://www.technonics.com/powerpc.htm Linux at IBM http://www-1.ibm.com/linux Linux Kernel Archives

http://www.kernel.org AIX Version 5L Operating System http://www-1.ibm.com/servers/aix

162 The IBM Eserver BladeCenter JS20 IBM Microelectronics Division - PowerPC Technical Documentation

http://www.chips.ibm.com IBM ^ BladeCenter Power Module Upgrade Guidelines Technical Update

http://www.ibm.com/pc/support/site.wss/document.do?lndocid=MIGR-53353 Myrinet Web site http://www.myri.com IBM ^ Cluster 1350 product http://www.ibm.com/eserver/cluster PowerPC Architecture books from the IBM DeveloperWorks Web site http://www.ibm.com/developerworks/eserver/articles/archguide.html IBM white paper POWER3: Next Generation 64-bit PowerPC Processor Design http://www.ibm.com/servers/eserver/pseries/hardware/whitepapers/power3wp.ht ml IBM white paper entitled POWER4 System Microarchitecture http://www.ibm.com/servers/eserver/pseries/hardware/whitepapers/power4.html PowerPC Microprocessor Family: AltiVec Technology Programming Environments Manual http://www-306.ibm.com/chips/techlib/techlib.nsf/techdocs/FBFA164F824370F98 7256D6A006F424D RETAIN tip H181655 http://www-307.ibm.com/pc/support/site.wss/document.do?lndocid=MIGR-55282 CSM publication site http://publib.boulder.ibm.com/clresctr/windows/public/clusterbooks.html Service and productivity tools for Linux on POWER http://techsupport.services.ibm.com/server/lopdiags The Linux Documentation Project - Kernel HOWTO http://www.tldp.org The latest CSM software http://techsupport.services.ibm.com/server/cluster/fixes/csmfixhome.html

CSM for Linux: Planning and Installation Guide, SA22-7853 http://publib.boulder.ibm.com/epubs/pdf/am7il103.pdf

Related publications 163 How to get IBM Redbooks

You can search for, view, or download Redbooks, Redpapers, Hints and Tips,

draft publications and Additional materials, as well as order hardcopy Redbooks or CD-ROMs, at this Web site: ibm.com/redbooks

Help from IBM

IBM Support and downloads ibm.com/support

IBM Global Services ibm.com/services

164 The IBM Eserver BladeCenter JS20 Index

I/O modules 16–21 Symbols Management Module 13–16 /boot/ppc64 109 power module options 10 /etc/dhcpd.conf 110 power options 10–163 /etc/exports 103 power requirements 5 /tftpboot 111 standard features 9 supported bays 8 Numerics versus IBM rack server 5 64-bit computing 32, 34, 36–37, 42, 46 BladeCenter T 10 boot sequence 78 A Bootstrap protocol (BOOTP) 60 Brocade Enterprise SAN Switch Module 17, 19 activating the open firmware interface 94 Brocade Entry SAN Switch Module 17, 20 Aggressive branch prediction 44–45 Bus Interface Unit (BIU) 45 AIX operating system 48, 54, 57, 60, 62, 64, 79, 84–85 installing 123 C installing with NIM 124–138 C2 security 48 Altivec 40 cache structure 44 AMD 3DNow! 40 Cisco Systems Intelligent Gigabit Ethernet Switch Apple Computer 36 Module 17–18, 76, 82 ARP function 56 Command line interface (CLI) 56, 91 array 39 env command 92 array processing 39 Complex instruction set computer (CISC) 30 AS/400 37 console for JS20 blades 87 AutoYaST 114 copycds command 152 copycsmpkgs command 152 B count register (CTR) 35 CSM blade servers 21 see IBM Cluster Systems Management assigning names 77 csmconfig command 152 configuration 76–78 setting the boot sequence 78 Blade Systems Management Processor (BSMP) D 15, 22, 26 definenode command 156 BladeCenter DHCP (dynamic host configuration protocol) 60–61 advantages dynamic host configuration protocol (DHCP) 54, high availability 4 60–61, 70, 110, 149 lower cost 4 SAN optimization 4 server consolidation 4 E effective to real address translation (ERAT) 44 switch technology 4 en0 62 chassis 8–10 en1 62 management module 13 env command 92 HS20 blade server 21

© Copyright IBM Corp. 2005. All rights reserved. 165 eth0 62 PowerPC 970FX 22, 29, 38, 42, 46 eth1 62 IBM microprocessors PowerPC 601 36 PowerPC 603 36 F PowerPC 604 36 Fibre Channel Expansion Card 28 IBM rack server 5 firmware 79–87 IBM RS/6000 36 JS20 blade server 84 installms command 151 LAN switch I/O module 82 installnode command 158 Management Module 80 integrated switch technology 4 upgrading under Linux 84 Integrated Systems Management Processor (ISMP) vital product data (VPD) 79 85 fixed-point exception register (XER) 34 Intel Multimedia Extensions (MMX) 40 floating-point status and control register (FPSCR) Intel Streaming SIMD Extensions (SSE) 40 34 J G JS20 Blade Server general purpose registers (GPRs) 34 base features 24 Gigabit Ethernet Expansion Card 28 ethernet controller 25 GNU C Compiler (gcc) 49 I/O expansion option connectors 25 I/O subsystem 24 H IDE controller 25 high availability 4 memory subsystem 24 HTTP 16 miscellaneous I/O 26 HTTPS 16 optional features 27 expansion cards 28 I hard disk drives 27 I/O modules 16–21, 73–75 memory modules 27 configuration 73 processor subsystem 24 IBM 4-Port Gigabit Ethernet Switch Module 17, 76, USB controllers 26 79, 82 IBM AS/400 37 K IBM Cluster Systems Management (CSM) 16, keyboard, video monitor and mouse (KVM) 14 50–51, 56, 60, 64, 66, 147–158 KVM (keyboard, video monitor and mouse) 14 obtaining latest software and prerequisites 150 setting up 148 IBM Director 16, 56, 63–64, 140 L link register (LR) 35 agent 64 Linux installing agent 145 compiling a Red Hat kernel 108 IBM General Parallel File System (GPFS) 59 compiling a SuSe kernel 108 IBM Microelectronics Division 46 configuring the sources 102 chip documentation 46 installing 102 IBM Microprocessors Red Hat sources 102 POWER2 31 SuSE sources 104 POWER3 37 local area network (LAN) 16 POWER4 37 lsnode command 156 POWER5 38 PowerPC 970 22, 29, 38, 42, 46

166 The IBM Eserver BladeCenter JS20 M PowerPC architecture Management Module 13–16, 70–73 background 32 configuration 70–73 definition 30 console commands 93 Operating Environment Architecture (OEA) 33 default IP address 70 PowerPC 601 36 default password 72 PowerPC 603 36 default userid 71 PowerPC 604 36 firmware 80 PowerPC 970 22, 29, 38, 42, 46 mkinitrd command 109 PowerPC 970FX 22, 29, 38, 42, 46 mkzimage_cmdline command 117 User Instruction Set Architecture (UISA) 33 Motorola 36 Virtual Environment Architecture (VEA) 33 Myrinet 16, 20, 22, 28, 58–59 preparing an unattended installation 111 probemgr command 154 N Network File System (NFS) 60 R network installation 118 rack server 5 network installation planning 60 rconsole command 157 network interface selection 62 Red Hat Enterprise Linux 3 48, 121 network planning 54 Red Hat’s Kickstart 112 minimal requirements 54 Redbooks Web site 164 network services 60 Contact us xv Nortel Networks Layer 2-7 Gigabit Ethernet Switch reduced instruction set computer (RISC) 30 Module 17–18, 76, 82 Register renaming 44 required network services 60 rpm command 151 O rpower command 157 obtaining latest CSM software and prerequisites RS64 processors 36 150 rtas_flash kernel module 84 Open Firmware 117 Operating Environment Architecture (OEA) 33 operating system support 48 S Optical Pass-thru Module 20 segment lookaside buffer (SLB) 44 Serial over LAN (SoL) 14, 50, 57, 62, 76, 87 components 88 P configuring 89 PCI using command line interface 91 see Peripheral Component Interconnect server consolidation 4 performing a network installation 118 setting up a CSM management node 148 performing an unattended installation 116 setting up network installation 61 Peripheral Component Interconnect (PCI) 5, 25, small computer system interface (SCSI) 5 141 SoL (Serial over LAN) 14 POWER architecture SSH 16, 89, 91 definition 30 storage area network (SAN) 16 POWER2 31 optimization 4 POWER3 37 superscalar features 44 POWER4 37 SuSE AutoYaST 114 POWER5 38 SUSE LINUX Enterprise Server 9 49, 118 Power On Self Test (POST) 109 symmetrical multiprocessor (SMP) 36 power requirements for BladeCenter 5 systemid command 153

Index 167 T translation lookaside buffer (TLB) 44 Trivial File Transfer Protocol (TFTP) 60, 111

U unattended installation 111, 116 Universal Manageability Initiative 49 update_flash script 85 User Instruction Set Architecture (UISA) 33

V vector 39 vector length 38 vector register file (VRF) 41 Vector/SIMD Multimedia eXtension (VMX) 38 Virtual Environment Architecture (VEA) 33 Virtual LAN (VLAN) 15, 17–18, 54, 57, 61, 76, 87, 89 vital product data (VPD) 79 VMX extensions to PowerPC architecture 40 VMX status and control register (VSCR) 41 VPD (vital product data) 79 VRP (vector register file) 41 VSCR (VMX status and control register) 41

X xinetd daemon 111

Y Yet Another Setup Tool (YaST) 104

Z zImage.initrd 116 zImage.initrd file 107, 109 building your own 107–108

168 The IBM Eserver BladeCenter JS20

The IBM Eserver BladeCenter JS20

Back cover ®

The IBM Eserver

BladeCenter JS20

Learn about IBM Blade servers are a relatively new technology. They have INTERNATIONAL Eserver captured industry focus because of their modular design, BladeCenter JS20 which can reduce cost with a more efficient use of valuable TECHNICAL and blade server floor space. They offer simplified management, which can SUPPORT technology help to speed such tasks as installing, reprovisioning, ORGANIZATION updating, and troubleshooting hundreds of blade servers. You can do all of this remotely using one graphical console with Understand the IBM Director systems management tools. PowerPC 970 BUILDING TECHNICAL microprocessor In addition, blade servers provide improved performance by INFORMATION BASED ON PRACTICAL EXPERIENCE architecture doubling current rack density. By integrating resources and sharing key components, costs decrease and availability Plan for, install, and increases. IBM Redbooks are developed by configure the IBM International Technical BladeCenter JS20 The IBM Eserver BladeCenter boasts innovative modular Support Organization. Experts technology, leadership density, and availability. It was from IBM, Customers and Partners from around the world designed to help solve a multitude of real-world problems. create timely technical information based on realistic This IBM Redbook takes an in-depth look at the IBM scenarios. Specific Eserver BladeCenter JS20. This is a two-way blade server recommendations are provided for applications requiring 64-bit computing. It is ideal for to help you implement IT solutions more effectively in computer-intensive applications and transactional Internet your environment. servers. This IBM Redbook helps you to install, tailor, and configure the IBM Eserver® BladeCenter™ JS20.

For more information: ibm.com/redbooks

SG24-6342-01 ISBN 073849061X