SIOS Protection Suite for Linux v8.3.2 Technical Documentation Dec 2014 This document and the information herein is the property of SIOS Technology Corp. (previously known as SteelEye® Technology, Inc.) and all unauthorized use and reproduction is prohibited. SIOS Technology Corp. makes no warranties with respect to the contents of this document and reserves the right to revise this publication and make changes to the products described herein without prior notification. It is the policy of SIOS Technology Corp. to improve products as new technology, components and software become available. SIOS Technology Corp., therefore, reserves the right to change specifications without prior notice. LifeKeeper, SteelEye and SteelEye DataKeeper are registered trademarks of SIOS Technology Corp. Other brand and product names used herein are for identification purposes only and may be trademarks of their respective companies. To maintain the quality of our publications, we welcome your comments on the accuracy, clarity, organization, and value of this document. Address correspondence to: [email protected] Copyright © 2014 By SIOS Technology Corp. San Mateo, CA U.S.A. All rights reserved Table of Contents Chapter 1: Introduction 1 About SIOS Protection Suite for Linux 1 SPS for Linux Integrated Components 1 Documentation and Training 1 Documentation 1 Training 2 Technical Support 2 Chapter 3: SIOS LifeKeeper for Linux 5 Introduction 5 Protected Resources 5 LifeKeeper Core 6 LifeKeeper Core Software 6 File System, Generic Application, IP and RAW I/O Recovery Kit Software 7 LifeKeeper GUI Software 8 LifeKeeper Man Pages 8 Configuration Concepts 8 Common Hardware Components 8 Components Common to All LifeKeeper Configurations 9 System Grouping Arrangements 9 Active - Active Grouping 10 Active - Standby Grouping 11 Intelligent Versus Automatic Switchback 12 Logging With syslog 13 Resource Hierarchies 13 Resource Types 13 Table of Contents i Resource States 14 Hierarchy Relationships 15 Shared Equivalencies 15 Resource Hierarchy Information 16 Resource Hierarchy Example 17 Detailed Status Display 17 Resource Hierarchy Information 19 Communication Status Information 20 LifeKeeper Flags 21 Shutdown Strategy 22 Short Status Display 22 Resource Hierarchy Information 22 Communication Status Information 23 Fault Detection and Recovery Scenarios 23 IP Local Recovery 23 Local Recovery Scenario 23 Command Line Operations 24 Resource Error Recovery Scenario 25 Server Failure Recovery Scenario 27 Installation and Configuration 29 SPS for Linux Installation 29 SPS for Linux Configuration 29 SPS Configuration Steps 29 Set Up TTY Connections 30 LifeKeeper Event Forwarding via SNMP 31 Overview of LifeKeeper Event Forwarding via SNMP 31 LifeKeeper Events Table 31 Configuring LifeKeeper Event Forwarding 33 Prerequisites 33 Configuration Tasks 33 Table of Contents ii Verifying the Configuration 34 Disabling SNMP Event Forwarding 34 SNMP Troubleshooting 34 LifeKeeper Event Email Notification 35 Overview of LifeKeeper Event Email Notification 35 LifeKeeper Events Generating Email 35 Configuring LifeKeeper Event Email Notification 37 Prerequisites 37 Configuration Tasks 37 Verifying the Configuration 37 Disabling Event Email Notification 37 Email Notification Troubleshooting 38 Optional Configuration Tasks 39 Adding the LifeKeeper GUI Icon to the Desktop Toolbar 39 Changing the Icon Position 39 Confirm Failover and Block Resource Failover Settings 39 Confirm Failover On 39 When to Select This Setting 40 Block Resource Failover On 40 Conditions/Considerations 41 Setting the Flags on Each Server 41 Examples 43 Block All Automatic Failovers Completely 43 Block Failover in One Direction 43 Setting Server Shutdown Strategy 44 Tuning the LifeKeeper Heartbeat 45 Overview of the Tunable Heartbeat 45 Example 45 Configuring the Heartbeat 45 Configuration Considerations 46 Table of Contents iii Using Custom Certificates with the SPS API 46 How Certificates Are Used 47 Using Your Own Certificates 47 Linux Configuration 47 Data Replication Configuration 50 Network Configuration 50 Application Configuration 51 Storage and Adapter Configuration 52 HP Multipath I/O Configurations 77 EMC PowerPath Multipath I/O Configurations 80 IBM SDD Multipath I/O Configurations 82 Hitachi HDLM Multipath I/O Configurations 83 Device Mapper Multipath I/O Configurations 116 LifeKeeper I-O Fencing Introduction 118 SCSI Reservations 119 Storage Fence Using SCSI Reservations 119 Alternative Methods for I/O Fencing 120 Disabling Reservations 120 Non-Shared Storage 121 Configuring I/O Fencing Without Reservations 121 I/O Fencing Chart 121 Quorum/Witness 123 Quorum/Witness Server Support Package for LifeKeeper 123 Feature Summary 123 Package Requirements 123 Package Installation and Configuration 124 Configurable Components 124 Available Quorum Modes 125 Available Witness Modes 126 Available Actions When Quorum is Lost 127 Table of Contents iv Additional Configuration for Shared-Witness Topologies 127 Adding a Witness Node to a Two-Node Cluster 128 Expected Behaviors (Assuming Default Modes) 129 Scenario 1 129 Scenario 2 129 Scenario 3 130 Scenario 4 130 STONITH 131 Using IPMI with STONITH 131 Package Requirements 131 STONITH in VMware vSphere Environments 131 Package Requirements 131 STONITH Server 131 Monitored Virtual Machine 131 Installation and Configuration 131 <vm_id> 133 Expected Behaviors 134 Watchdog 134 Components 134 Configuration 135 Uninstall 136 Resource Policy Management 136 Overview 136 SIOS Protection Suite 137 Custom and Maintenance-Mode Behavior via Policies 137 Standard Policies 137 Meta Policies 138 Important Considerations for Resource-Level Policies 138 The lkpolicy Tool 139 Example lkpolicy Usage 139 Table of Contents v Authenticating With Local and Remote Servers 139 Listing Policies 140 Showing Current Policies 140 Setting Policies 140 Removing Policies 140 Configuring Credentials 140 Adding or Changing Credentials 141 Listing Stored Credentials 141 Removing Credentials for a Server 141 Additional Information 141 LifeKeeper API 141 Network Configuration 142 Authentication 142 LifeKeeper Administration 143 Overview 143 Error Detection and Notification 143 N-Way Recovery 143 Administrator Tasks 144 Editing Server Properties 144 Creating a Communication Path 144 Deleting a Communication Path 145 Server Properties - Failover 146 Creating Resource Hierarchies 147 LifeKeeper Application Resource Hierarchies 148 Recovery Kit Options 148 Creating a File System Resource Hierarchy 148 Creating a Generic Application Resource Hierarchy 149 Creating a Raw Device Resource Hierarchy 151 Editing Resource Properties 151 Editing Resource Priorities 152 Table of Contents vi Using the Up and Down Buttons 153 Editing the Priority Values 153 Applying Your Changes 153 Extending Resource Hierarchies 153 Extending a File System Resource Hierarchy 154 Extending a Generic Application Resource Hierarchy 155 Extending a Raw Device Resource Hierarchy 155 Unextending a Hierarchy 155 Creating a Resource Dependency 156 Deleting a Resource Dependency 157 Deleting a Hierarchy from All Servers 158 LifeKeeper User Guide 159 Using LifeKeeper for Linux 160 GUI 160 GUI Overview - General 160 GUI Server 160 GUI Client 160 Exiting GUI Clients 161 The LifeKeeper GUI Software Package 161 Menus 161 SIOS LifeKeeper for Linux Menus 161 Resource Context Menu 161 Server Context Menu 162 File Menu 163 Edit Menu - Resource 164 Edit Menu - Server 164 View Menu 165 Help Menu 166 Toolbars 166 SIOS LifeKeeper for Linux Toolbars 166 Table of Contents vii GUI Toolbar 166 Resource Context Toolbar 168 Server Context Toolbar 169 Preparing to Run the GUI 169 LifeKeeper GUI - Overview 169 GUI Server 170 GUI Client 170 Starting GUI clients 170 Starting the LifeKeeper GUI Applet 170 Starting the application client 171 Exiting GUI Clients 171 Configuring the LifeKeeper GUI 171 Configuring the LifeKeeper Server for GUI Administration 171 Running the GUI 171 GUI Configuration 172 GUI Limitations 172 Starting and Stopping the GUI Server 173 To Start the LifeKeeper GUI Server 173 Troubleshooting 173 To Stop the LifeKeeper GUI Server 173 LifeKeeper GUI Server Processes 174 Java Security Policy 174 Location of Policy Files 174 Policy File Creation and Management 175 Granting Permissions in Policy Files 175 Sample Policy File 176 Java Plug-In 176 Downloading the Java Plug-in 177 Running the GUI on a Remote System 177 Configuring the GUI on a Remote System 177 Table of Contents viii Running the GUI on a Remote System 178 Applet Troubleshooting 178 Running the GUI on a LifeKeeper Server 179 Browser Security Parameters for GUI Applet 179 Firefox 179 Internet Explorer 180 Status Table 180 Properties Panel 181 Output Panel 181 Message Bar 181 Exiting the GUI 182 Common Tasks 182 Starting LifeKeeper 182 Starting LifeKeeper Server Processes 182 Enabling Automatic LifeKeeper Restart 182 Stopping LifeKeeper 183 Disabling Automatic LifeKeeper Restart 183 Viewing LifeKeeper Processes 184 Viewing LifeKeeper GUI Server Processes 185 Viewing LifeKeeper Controlling Processes 185 Connecting Servers to a Cluster 186 Disconnecting From a Cluster 187 Viewing Connected Servers 187 Viewing the Status of a Server 188 Viewing Server Properties 188 Viewing Server Log Files 189 Viewing Resource Tags and IDs 189 Viewing the Status of Resources 189 Server Resource Status 190 Global Resource Status 191 Table of Contents ix Viewing Resource Properties 191 Setting View Options for the Status Window 192 Resource Labels 192 Resource Tree 193 Comm Path Status 193 Row Height