How to configure Network Interface Backup on ~ BladeCenter JS20 with AIX 5L

White Paper

Trina Bunting pSeries Advanced Technical Support

September 2005 Table of Contents

Introduction...... 3 Prerequisites...... 3 Network Interface Backup Overview...... 3 BladeCenter JS20/NIB Configuration Limitations...... 4 Sample NIB Environment ...... 4 My BladeCenter JS20/NIB Environment...... 5 Setting up the CIGESM...... 8 A. Installing Firmware on the CIGECSM...... 8 B. Change the Port Setting on CIGESM ...... 9 C. Configure Link State Tracking (Trunk Failover) ...... 9 Troubleshooting NIB in AIX 5L...... 10 How to Test Adapter Failover ...... 10 A. Testing Adapter Failover from AIX 5L...... 10 B. Testing Adapter Failover from the CIGESM...... 11 C. Testing Adapter Failover from the Management Module...... 12

bctrjs20HA090805.doc Page 2 of 13 Introduction

This document provides an overview of Network Interface Backup (NIB) feature of the AIX 5L™ operating system as well as how to configure and implement NIB style EtherChannel® on an IBM ~® BladeCenter® JS20 with ® Intelligent Gigabit Ethernet Switch Module (CIGESM). I also cover how to view and troubleshoot your NIB setup in the AIX 5L O/S, as well as, how to successfully test Adapter Failover from AIX 5L, CIGESM and the Management Module.

Prerequisites

The following hardware/software was used for my configuration and testing: • Two CIGESMs • CIGESM firmware level 12.1(22)AY1 or higher • AIX 5L V5.3 ML2

Although the CIGESM was used for my configuration, other Ethernet Switch Modules can be used as well (i.e. Nortel and DLink Switch Modules).

Note: In the recommended version of firmware for the CIGESM, there is a potential problem with using a Management VLAN higher than 255. This problem is planned to be resolved in the next firmware level. Until a fix is available it is recommended to use a Management VLAN between 1 and 255. Also, the CIGESM must not be configured for EtherChannel on any of the downstream ports going to the BladeCenter JS20 blades.

Network Interface Backup Overview

NIB is designed to protect against a single point of network failure by providing failure detection and failover. In a NIB environment, only one adapter is active at any given time. Two key tests, link status and ping, protect against local link failure and network path failure, respectively. When a failure is detected, all traffic is sent and received on the backup adapter (backup adapter has the same MAC address as the primary adapter when NIB is configured, regardless of whether there’s a failover or not). It provides rapid detection/failover (in 2-4 seconds) with no disruption to currently active user connections.

The AIX 5L NIB function is transparent to the network equipment therefore no special hardware or network equipment is needed. The overall effectiveness of this solution depends upon the implementer's knowledge of network design and AIX 5 L system attachments. The goal is to make sure that a single failure (at an adapter, node, or network component) doesn't impact all the AIX 5L links configured for the NIB group.

bctrjs20HA090805.doc Page 3 of 13 BladeCenter JS20/NIB Configuration Limitations

The BladeCenter JS20 Network Interface Cards (NICs) are hard-wired to two different switches (Bay 1 and Bay 2) in the BladeCenter chassis so only a NIB-style EtherChannel configuration can be implemented. The ent0 NIC is hard wired to Bay 1 and ent1 NIC is hard wired to Bay 2. In a NIB configuration ent0 should be configured as the primary adapter and ent1 should be configured as the backup adapter. Currently, if you configure ent1 as the primary adapter and ent0 as the secondary adapter, Serial Over LAN (SOL) will not function.

Note: The SOL function will only work on the ent0 adapter so this function will not be available when the ent1 backup adapter is activated. During this time, the administrator would use telnet to access the BladeCenter JS20.

Sample NIB Environment

The following diagram is a sample NIB configuration where the Ethernet adapters (ent0 and ent1) are configured as EtherChannel pseudo-device ent2. The en2 interface should be configured to run TCP/IP therefore en0 or en1 will not be used. The ent0 (primary) adapter is active and ent1 is the backup adapter.

9.19.126.108 JS20 00:0d:60:1e:16:c0 Ethernet Switch ent2 Module Bay 1 ent0 en2

ent1 Ethernet Switch Module Bay 2 (backup)

The link status test detects failures in the adapter link to the network. This includes adapter failure, cable break, switch or hub unplugged or failed. Link status tests for an electrical signal, meaning link-up or link-down state detection. When the electrical signal disappears, the switch to the backup is immediately triggered.

The optional ping test checks the path through the network to the user-specified IP node. By default, the ping test sends a ping every second. If at least one response in three pings (default; can be modified by the user) is received, the network path is considered to be good. If three pings go without a response, failover is triggered.

bctrjs20HA090805.doc Page 4 of 13 If the link or ping test on ent0 fails, the NIB software begins receiving traffic for the ent2 pseudo-device on the backup adapter ent1. Once the problem on ent0 has been resolved, all traffic is once again sent and received on the primary adapter.

Note: Only if the link has failed will the traffic move back to the primary adapter. If the ping failed it will remain on the backup adapter until it itself fails.

My BladeCenter JS20/NIB Environment

It is important to consider how the CIGESM is managed before implementing NIB into your environment. In my test environment, the CIGESMs are managed using the Management Module’s (MM) uplink port. The Management Module and Data Traffic are in a common VLAN. The CIGESMs are on an internally different VLAN, but shares Management Module uplink VLAN for management. SOL is provided on a separate VLAN. The feature to manage the CIGESM over its uplink ports MUST be disabled (by setting “External management over all ports” to “Disabled” in the MM Advanced Management section for each CIGESM). For more detailed information on BladeCenter CIGESM Management, refer to the following Redpaper: Cisco Systems Intelligent Gigabit Ethernet Switch Module for IBM eServer™ BladeCenter (http://www.redbooks.ibm.com/abstracts/redp3869.html?Open).

Note: Using a single VLAN to carry both user and management traffic is NOT recommended in a Production Environment.

The following is a diagram of my test environment. In my environment, the VLANs

bctrjs20HA090805.doc Page 5 of 13

are carried as follows: • On ports going to the JS20 Blades (g0/1 – g0/14): Data VLAN 1 SOL VLAN 4094 • On ports going to the MM (g0/15 – g0/16): MM/Switch VLAN 30 SOL VLAN 4094 • On ports going to the JS20 Production Network (g0/17 – g0/20): Data VLAN 1

Setting up NIB in AIX 5L

Configuring NIB

1. This step should only be performed if the adapter interfaces are configured. To remove any pre-existing IP interface configurations type the following commands: Type “ifconfig en0 detach” and press “Enter”

bctrjs20HA090805.doc Page 6 of 13 Type “rmdev –dl en0” and press “Enter”

Type “ifconfig en1 detach” and press “Enter” Type “rmdev –dl en1” and press “Enter”

Type “ifconfig et0 detach” and press “Enter” Type “rmdev –dl et0” and press “Enter”

Type “ifconfig et1 detach” and press “Enter” Type “rmdev –dl et1” and press “Enter”

Note: The above commands can also be executed in smit from the fast path “smitty tcpip”.

2. Verify the network interfaces have been removed: Type “netstat –in” and press “Enter” (You should only see the loopback interface) 3. Create the ent2 pseudo-device: a. Type the fast path “smitty etherchannel” and press “Enter” b. Select “Add an EtherChannel / ” and press “Enter” c. Select “ent0” as the primary adapter and “ent1” as the backup adapter and press “Enter”

For more information on setting up Network Interface Backup style EtherChannel refer to the documentation at http://publib.boulder.ibm.com/infocenter/pseries/index.jsp?topic=/com.ibm.aix.doc/inf ocenter/base/aix53.htm.

Note: If Jumbo Frames are enabled, the CIGESM is already enabled to support Jumbo Frames with firmware level 12.1(22)AY1. The max frame size on the CIGESM is 9216 bytes.

bctrjs20HA090805.doc Page 7 of 13

4. Configure the IP Address for the pseudo-device ent2: a. Type the fast path “smitty chinet” and press “Enter” b. Select “en2” interface and press “Enter”

5. Enter the “IP/Address, Network Mask”, change the “Current STATE” to “up” and press “Enter”

Setting up the CIGESM

A. Installing Firmware on the CIGECSM

The latest release of CIGESM IOS code (currently 39r6667.tar, 12.1(22)AY1) must be installed on both CIGESMs. To install the latest version of firmware do the following: 1. Download the latest release of firmware from: http://www.ibm.com/pc/support/site.wss/document.do?lndocid=MIGR-58132 2. Install and configure a TFTP Server. A free version of a TFTP Server can be downloaded from: http://support.solarwinds.net/updates/MoreSoftware.cfm?Program=. To configure the TFTP server follow the procedure in Chapter 6.13 of the Cisco Systems Intelligent Gigabit Ethernet Switch Module for IBM eServer BladeCenter Redpaper (http://www.redbooks.ibm.com/redpapers/pdfs/redp3869.pdf ) 3. Test the connectivity between the Cisco Switch Module and the TFTP Server a. ping the CIGESM from the TFTP Server b. ping the TFTP Server from the CIGESM 4. To install the updated firmware on both CIGESMs do the following:

bctrjs20HA090805.doc Page 8 of 13 a. Type “archive download tftp://XXXX/YYYY” and press “Enter” (where X is the TFTP Server’s IP address and Y the name of the tar file “39r6667.tar, 12.1(22)AY1, at the time of this writing” you downloaded) b. Type “write memory” and press “Enter” c. Type “reload” and press “Enter”

B. Change the Port Setting on CIGESM

To ensure failover and failback occurs as expected it is recommended to hard code the speed of the ports going to the JS20 on both CIGESMs. Telnet to the Cisco Switch Module and type the following commands on both switches: 1. Type “config t” and press “Enter” 2. Type “int range g0/1 -14” and press “Enter” 3. Type “speed noneg” and press “Enter” 4. Type “end” and press “Enter” 5. Type “write” and press “Enter”

C. Configure Link State Tracking (Trunk Failover)

With the Link State Tracking feature configured on each CIGESM, if the upstream connection goes down it cuts the downstream port (toward the JS20) off. Refer to Chapter 7.7 of the latest version of the Cisco Systems Intelligent Gigabit Ethernet Switch Module for IBM eServer BladeCenter Redpaper for more details (http://www.redbooks.ibm.com/redpapers/pdfs/redp3869.pdf).

Link state tracking configuration is only necessary when using the link status test feature in AIX 5L to determine upstream connectivity. If you are using the ping test feature it is not necessary to configure link state tracking.

For our example environment (only port 17 is being used for upstream connection), Link State Tracking configuration would be done as follows: 1. Type “configure terminal” and press “Enter” 2. Type “link state track 1” and press “Enter” 3. Type “int g0/17” and press “Enter” 4. Type “link state group 1 upstream” and press “Enter” 5. Type “int range g0/1 -14” and press “Enter” 6. Type “link state group 1 downstream” and press “Enter” 7. Type “end” and press “Enter” 8. Type “write” and press “Enter”

A couple of notes about the above config: A) The above assumes you want to take all BladeServer facing ports down when the uplink goes down. If your goal is to only shut down a specific set of ports on uplink

bctrjs20HA090805.doc Page 9 of 13 failure, this can be configured as well by appropriate placement of the “downstream” commands. B) The above config assumes you are only tracking a single upstream port (g0/17). If you are using more then one uplink or have EtherChannel-ed the upstream ports, you will need to apply the “upstream” command differently. C) As noted above, please see section 7.7 in the CIGESM deployment Redpaper for more details on configuring link state tracking (AKA Trunk Failover and Layer 2 Failover) options.

Troubleshooting NIB in AIX 5L

• The lsdev –Cc adapter and lsdev –Cc if commands will identify ent2 as EtherChannel and en2 as the IP Interface. The lsattr –El ent2 command will show the current attribute values for the pseudo-device ent2. • If an adapter failure is triggered by the link status or ping test, an error is posted in the error log. The netstat –v ent2 command will show information about the primary and backup adapter.

How to Test Adapter Failover

A. Testing Adapter Failover from AIX 5L

To Test Adapter Failover from AIX 5L do the following: 1. Type the fast path “smitty etherchannel” and press “Enter” 2. Select “Force a Failover In An EtherChannel / Link Aggregation” and press “Enter”

bctrjs20HA090805.doc Page 10 of 13 3. Select “ent2” adapter and press “Enter” 4. Press “Enter” on the pop-up menu confirming your action

To verify adapter failover check the error report for an error message (ECH_CHAN_FAIL) indicating the primary EtherChannel failed and the backup adapter has taken over. If you repeat the above steps and check the error report again, an error message (ECH_CHAN_RCVRY) indicates the primary adapter has been recovered.

Note: Regardless of which adapter is active, ent0 will always be the primary and ent1 will always be the backup adapter.

B. Testing Adapter Failover from the CIGESM

There are a number of ways to test the failover feature, including physically pulling the upstream cables or disabling ports at various locations in the path between the BladeCenter chassis and the upstream switches. The following is one example way to perform this test.

Note: This might not be an appropriate test if your management path to the CIGESMs is via its own uplinks (you will be shutting down the path you are managing the switch by). If you are managing the CIGESM’s via their own uplinks it is possible to do the test below by plugging directly into the console port of the referenced CIGESMs and running the commands from there.

Also note this procedure assumes link state tracking is configured on the CIGESMs. If link state tracking is not configured, the below test may not produce the desired results . For example, if link state tracking is not configured (on the CIGESM), and ping test is not configured (in AIX 5L EtherChannel config), then shutting down port g0/17 will not cause a fail over and the connection will fail.

1. Setup looping ping from a network attached device (i.e. your laptop) to the sever with the NIB setup (ping –t ipaddress). 2. The following commands should be typed on both CIGESMs, starting with the top ESM: a. Type “config t” and press “Enter” b. Type “int g0/17” and press “Enter” c. Type “shut” and press “Enter” d. Type “end” and press “Enter” e. Type “write” and press “Enter”

Note: (Step 2 should be executed ONLY if you are managing the ESM via the MM)

Notice the ping as you shutdown connection to the top switch. The ping will stop for a few seconds then it will resume until you type the above commands on the second switch. Once you type the above commands on the second switch the ping will stop altogether

bctrjs20HA090805.doc Page 11 of 13 (no remaining path to the JS20). It will not resume until you turn one of the switches upstream connection back on with the following commands: a. Type “config t” and press “Enter” b. Type “int g0/17” and press “Enter” c. Type “no shut” and press “Enter” d. Type “end” and press “Enter” e. Type “write” and press “Enter”

Notice the ping will resume. This proves the adapter can fail over since ent0 is the top switch and ent1 is the bottom switch. This configuration may affect SOL, in particular if the top switch goes down or the port to the top NIC goes down.

Note: Make sure you re-enable any ports shut-down in the above steps once testing is complete.

C. Testing Adapter Failover from the Management Module

To Test Adapter Failover from the Management Module do the following: 1. Setup looping ping from a network attached device (i.e. your laptop) to the sever with the NIB setup (ping –t ipaddress). 2. From the Management Module under I/O Module Task select “Power/Restart”. Now select the switch in Bay 1 then select “Restart Module(s) and Standard Diagnostics”. Select “Ok” on the pop-window confirming your reboot request. 3. Now watch the looping ping window. Your ping should only be interrupted for a second but the ping will continue which indicates successful fail over.

Note: The same test can be repeated for the Ethernet Switch Module in Bay 2.

bctrjs20HA090805.doc Page 12 of 13

© IBM Corporation 2005

IBM Corporation Marketing Communications

Systems and Technology Group Route 100 Somers, New York 10589

Produced in the United States of America August 2005

All Rights Reserved

This document was developed for products and/or services offered in the United States. IBM may not offer the products, features, or services discussed in this document in other

countries.

The information may be subject to change without notice. Consult your local IBM business contact for information on the products, features and services available in

your area.

All statements regarding IBM’s future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.

IBM, the IBM logo, ~, AIX 5L, BladeCenter are trademarks or registered trademarks of International Business Machines Corporation in the United States or

other countries or both. See http://www.ibm.com/legal/copytrade.shtml.

Cisco Systems and EtherChannel are trademarks or registered trademarks of Cisco Systems, Inc.

UNIX is a registered trademark of The Open Group in the United States, other countries or both.

Linux is a trademark of Linus Torvalds in the

United States, other countries or both.

Information concerning non-IBM products was obtained from the suppliers of these products. Questions on the capabilities of the non-IBM products should be addressed with the

suppliers.

All performance information was determined in a controlled environment. Actual results may vary. Performance information is provided “AS IS” and no warranties or guarantees are expressed or implied by IBM.

The IBM home page on the Internet can be found at http://www.ibm.com.

The IBM BladeCenter JS20 home page on the Internet can be found at http://www.ibm.com/servers/eserver/bladecent er/index.html.

bctrjs20HA090805.doc Page 13 of 13