How to configure Ethernet Network Interface Backup on ~ BladeCenter JS20 with AIX 5L White Paper Trina Bunting pSeries Advanced Technical Support September 2005 Table of Contents Introduction.....................................................................................................................3 Prerequisites....................................................................................................................3 Network Interface Backup Overview...............................................................................3 BladeCenter JS20/NIB Configuration Limitations...........................................................4 Sample NIB Environment ...............................................................................................4 My BladeCenter JS20/NIB Environment.........................................................................5 Setting up the CIGESM...................................................................................................8 A. Installing Firmware on the CIGECSM...................................................................8 B. Change the Port Setting on CIGESM .....................................................................9 C. Configure Link State Tracking (Trunk Failover) ....................................................9 Troubleshooting NIB in AIX 5L....................................................................................10 How to Test Adapter Failover .......................................................................................10 A. Testing Adapter Failover from AIX 5L................................................................10 B. Testing Adapter Failover from the CIGESM........................................................11 C. Testing Adapter Failover from the Management Module......................................12 bctrjs20HA090805.doc Page 2 of 13 Introduction This document provides an overview of Network Interface Backup (NIB) feature of the AIX 5L™ operating system as well as how to configure and implement NIB style EtherChannel® on an IBM ~® BladeCenter® JS20 with Cisco Systems® Intelligent Gigabit Ethernet Switch Module (CIGESM). I also cover how to view and troubleshoot your NIB setup in the AIX 5L O/S, as well as, how to successfully test Adapter Failover from AIX 5L, CIGESM and the Management Module. Prerequisites The following hardware/software was used for my configuration and testing: • Two CIGESMs • CIGESM firmware level 12.1(22)AY1 or higher • AIX 5L V5.3 ML2 Although the CIGESM was used for my configuration, other Ethernet Switch Modules can be used as well (i.e. Nortel and DLink Switch Modules). Note: In the recommended version of firmware for the CIGESM, there is a potential problem with using a Management VLAN higher than 255. This problem is planned to be resolved in the next firmware level. Until a fix is available it is recommended to use a Management VLAN between 1 and 255. Also, the CIGESM must not be configured for EtherChannel on any of the downstream ports going to the BladeCenter JS20 blades. Network Interface Backup Overview NIB is designed to protect against a single point of network failure by providing failure detection and failover. In a NIB environment, only one adapter is active at any given time. Two key tests, link status and ping, protect against local link failure and network path failure, respectively. When a failure is detected, all traffic is sent and received on the backup adapter (backup adapter has the same MAC address as the primary adapter when NIB is configured, regardless of whether there’s a failover or not). It provides rapid detection/failover (in 2-4 seconds) with no disruption to currently active user connections. The AIX 5L NIB function is transparent to the network equipment therefore no special hardware or network equipment is needed. The overall effectiveness of this solution depends upon the implementer's knowledge of network design and AIX 5 L system attachments. The goal is to make sure that a single failure (at an adapter, node, or network component) doesn't impact all the AIX 5L links configured for the NIB group. bctrjs20HA090805.doc Page 3 of 13 BladeCenter JS20/NIB Configuration Limitations The BladeCenter JS20 Network Interface Cards (NICs) are hard-wired to two different switches (Bay 1 and Bay 2) in the BladeCenter chassis so only a NIB-style EtherChannel configuration can be implemented. The ent0 NIC is hard wired to Bay 1 and ent1 NIC is hard wired to Bay 2. In a NIB configuration ent0 should be configured as the primary adapter and ent1 should be configured as the backup adapter. Currently, if you configure ent1 as the primary adapter and ent0 as the secondary adapter, Serial Over LAN (SOL) will not function. Note: The SOL function will only work on the ent0 adapter so this function will not be available when the ent1 backup adapter is activated. During this time, the administrator would use telnet to access the BladeCenter JS20. Sample NIB Environment The following diagram is a sample NIB configuration where the Ethernet adapters (ent0 and ent1) are configured as EtherChannel pseudo-device ent2. The en2 interface should be configured to run TCP/IP therefore en0 or en1 will not be used. The ent0 (primary) adapter is active and ent1 is the backup adapter. 9.19.126.108 JS20 00:0d:60:1e:16:c0 Ethernet Switch ent2 Module Bay 1 ent0 en2 ent1 Ethernet Switch Module Bay 2 (backup) The link status test detects failures in the adapter link to the network. This includes adapter failure, cable break, switch or hub unplugged or failed. Link status tests for an electrical signal, meaning link-up or link-down state detection. When the electrical signal disappears, the switch to the backup is immediately triggered. The optional ping test checks the path through the network to the user-specified IP node. By default, the ping test sends a ping every second. If at least one response in three pings (default; can be modified by the user) is received, the network path is considered to be good. If three pings go without a response, failover is triggered. bctrjs20HA090805.doc Page 4 of 13 If the link or ping test on ent0 fails, the NIB software begins receiving traffic for the ent2 pseudo-device on the backup adapter ent1. Once the problem on ent0 has been resolved, all traffic is once again sent and received on the primary adapter. Note: Only if the link has failed will the traffic move back to the primary adapter. If the ping failed it will remain on the backup adapter until it itself fails. My BladeCenter JS20/NIB Environment It is important to consider how the CIGESM is managed before implementing NIB into your environment. In my test environment, the CIGESMs are managed using the Management Module’s (MM) uplink port. The Management Module and Data Traffic are in a common VLAN. The CIGESMs are on an internally different VLAN, but shares Management Module uplink VLAN for management. SOL is provided on a separate VLAN. The feature to manage the CIGESM over its uplink ports MUST be disabled (by setting “External management over all ports” to “Disabled” in the MM Advanced Management section for each CIGESM). For more detailed information on BladeCenter CIGESM Management, refer to the following Redpaper: Cisco Systems Intelligent Gigabit Ethernet Switch Module for IBM eServer™ BladeCenter (http://www.redbooks.ibm.com/abstracts/redp3869.html?Open). Note: Using a single VLAN to carry both user and management traffic is NOT recommended in a Production Environment. The following is a diagram of my test environment. In my environment, the VLANs bctrjs20HA090805.doc Page 5 of 13 are carried as follows: • On ports going to the JS20 Blades (g0/1 – g0/14): Data VLAN 1 SOL VLAN 4094 • On ports going to the MM (g0/15 – g0/16): MM/Switch VLAN 30 SOL VLAN 4094 • On ports going to the JS20 Production Network (g0/17 – g0/20): Data VLAN 1 Setting up NIB in AIX 5L Configuring NIB 1. This step should only be performed if the adapter interfaces are configured. To remove any pre-existing IP interface configurations type the following commands: Type “ifconfig en0 detach” and press “Enter” bctrjs20HA090805.doc Page 6 of 13 Type “rmdev –dl en0” and press “Enter” Type “ifconfig en1 detach” and press “Enter” Type “rmdev –dl en1” and press “Enter” Type “ifconfig et0 detach” and press “Enter” Type “rmdev –dl et0” and press “Enter” Type “ifconfig et1 detach” and press “Enter” Type “rmdev –dl et1” and press “Enter” Note: The above commands can also be executed in smit from the fast path “smitty tcpip”. 2. Verify the network interfaces have been removed: Type “netstat –in” and press “Enter” (You should only see the loopback interface) 3. Create the ent2 pseudo-device: a. Type the fast path “smitty etherchannel” and press “Enter” b. Select “Add an EtherChannel / Link Aggregation” and press “Enter” c. Select “ent0” as the primary adapter and “ent1” as the backup adapter and press “Enter” For more information on setting up Network Interface Backup style EtherChannel refer to the documentation at http://publib.boulder.ibm.com/infocenter/pseries/index.jsp?topic=/com.ibm.aix.doc/inf ocenter/base/aix53.htm. Note: If Jumbo Frames are enabled, the CIGESM is already enabled to support Jumbo Frames with firmware level 12.1(22)AY1. The max frame size on the CIGESM is 9216 bytes. bctrjs20HA090805.doc Page 7 of 13 4. Configure the IP Address for the pseudo-device ent2: a. Type the fast path “smitty chinet” and press “Enter” b. Select “en2” interface and press “Enter” 5. Enter the “IP/Address,