High Availability Cluster User Guide 4.0.3

Date: October, 2014 Subject: High Availability Cluster User Guide Software: High Availability Cluster Software Version: 4.0.3 Part Number: 3000-hac4.x-000012-B

ems (hereinafter referred time without notice and time without notice ored in a database orored in a database retrieval ly can be ordered under the terms the the in features described this ed States and other countries. other and ed States rmission of Nexenta Syst rmission of Nexenta ed or transmitted in any form or by any means, or by any any form transmitted in ed or is documentation at at any is documentation ation on feature and product availability. This Version Number 4.0.3 ion available at the time of publication. e currently. Refer the latest to announcement product or contact photocopying and recording,photocopying and or st applicable agreements. All of agreements. All applicable and company names in this documentation are properties of their Nexenta products and serviceson k of Nexenta Systems in the Unit ems, ALL RIGHTS RESERVED ems, to this Documentation: to TM

NexentaStor HA Cluster 4.0.3 Product Name Product Copyright © 2014 Nexenta Systems, ALL RIGHTSALL RESERVED Systems, © 2014Copyright Nexenta www.nexenta.com Product Versions Applicable Versions Product to as “Nexenta”). to th changes to make reserves the right Nexenta Copyright © Copyright Nexenta Syst 2014 may be reproducthis publication No part of Notice: electronic or mechanical, including system for any purpose, without the express written pe assumes no responsibility for its use. conditions of NexentaandSystems’ documentation may not be availabl your local Nexenta Systems sales office for inform documentationincludes the latest informat Nexenta is a registered trademar marks, service All other trademarks, respective owners. ii High Availability Cluster User Guide User High Cluster Availability High Availability Cluster User Guide

Contents

Preface ...... v

1 Introduction to HA Cluster ...... 1 About the Nexenta HA Cluster ...... 1 Storage Failover ...... 2 Exclusive Access to Storage ...... 2 About SCSI Reservation ...... 2 Service Failover ...... 3 Sample Network Architecture ...... 3 Additional Resources ...... 4

2 Configuring the HA Cluster ...... 5 About Configuring the HA Cluster ...... 5 Prerequisites ...... 5 Binding the Nodes using SSH ...... 6 Registering the HA Cluster ...... 7 Adding a Virtual IP Address ...... 8 Modifying the Default Netmasks ...... 8 Configuring the HA Cluster ...... 9 Adding a Shared Volume to the HA Cluster ...... 9 Importing a Shared Volume ...... 11 Removing a Shared Volume ...... 12

3 Heartbeat and Network Interfaces ...... 13 About Heartbeat and Network Interfaces ...... 13 Heartbeat Mechanism ...... 13 Modifying Heartbeat Properties ...... 14 About Configuring Storage Failover ...... 15 Cluster Configuration Data ...... 15

4 Configuring Storage Failover ...... 15 Mapping Information ...... 16

NFS/CIFS Failover ...... 16 Configuring iSCSI Targets for Failover ...... 17 Configuring Fibre Channel Targets for Failover ...... 18 Setting the HA Cluster to ALUA Mode ...... 18 Changing the HBA Port Mode ...... 19 Creating a Target Group ...... 19 Adding WWNs to an Initiator Group ...... 20 Creating a Zvol ...... 21 Mapping a Zvol ...... 21

5 Advanced Setup ...... 23 About Advanced Setup ...... 23 Setting Failover Mode ...... 23 Setting Manual Failover Mode ...... 23 Setting Automatic Failover Mode ...... 24 Adding a Virtual IP Address and Hostname ...... 25 Managing Additional Volumes ...... 26 Managing Global Cluster Properties ...... 26 About Auto-Sync Replication in the HA Cluster Environment ...... 28 Creating an Auto-Sync Service in the HA Cluster Environment ...... 28 Establishing the SSH-binding for an Auto-Sync Service ...... 29 Binding a NexentaStor Appliance to a Virtual IP Address ...... 30

6 System Operations ...... 31 About System Operations ...... 31 Checking the Status of a Cluster ...... 31 Viewing the HA Cluster Status ...... 32 Manually Triggering a Failover ...... 33 Verifying Shared Volume Status ...... 33 Viewing Support Logs ...... 34 Operations with a Shared Volume ...... 34

7 Upgrading HA Cluster ...... 37 About Upgrading HA Cluster ...... 37 About SAS SSD Firmware Upgrade ...... 37

Overview of the Upgrade Process ...... 37 Verifying the Status of the Shared Volume(s) ...... 38 Failing Over the Shared Volume(s) ...... 38 Upgrading High Availability Cluster from Version 3.1.x to 4.0.3 ...... 39 Overview of the Rolling Upgrade Procedure ...... 39 Verifying the Status of the High Availability Cluster ...... 39 Upgrading the Secondary Node ...... 40 Upgrading Primary Node ...... 41 Completing the Post-Upgrade Procedures ...... 42

8 Testing and Troubleshooting ...... 45 Repairing a Broken Cluster Service ...... 45 Replacing a Faulted Node ...... 46 Restoring a faulted node ...... 46

Index ...... 49

This page intentionally left blank

v Preface http:// High Availability Cluster User Guide User High Availability Cluster or partner portal partner or Description directory on the HA Cluster appliance. the HA Cluster on directory ministrators. It assumes that you have experience experience have you that It assumes ministrators. /var/tmp /var/tmp . Date ts, such as NAS, SAN, NFS, and ZFS. and NFS, SAN, NAS, as such ts, http://nexenta.force.com/customerportal . Log in and browse a knowledge base. a knowledge and. Log in browse support . to this Documentation: to . Make Request Make Support Send the support request to the Nexenta support email.support the to Nexenta Send the support request the to the support information Saves • Send by email • disk to Save b. support wizard. the Complete nexenta.force.com/partnerportal c. form. request the Complete d. Click a. line, type the command At a. Click b. an action: Select Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com 3000-hac-4.x-000012-A3000-hac4.x-000012-B 2014 September, 2014 GA October, procedure the upgrade Updated Revision Product Versions Applicable Versions Product •Console): Management (Nexenta NMC command line, the NexentaStor Using Choose a method for contacting support: contacting for method a Choose • portal customer Visit the Nexenta Contacting Support Contacting Documentation History revisions of thisdocumentation. the released table lists The following Intended Audience Ad Storage Network for is intended This documentation NexentaStor and with data storage concep storage with data and NexentaStor This documentation presents information specific to Nexenta products. The information is for reference reference is for The information products. specific to Nexenta information This documentation presents change. to subject is and purposes • View): Management (Nexenta NMV user interface, the NexentaStor Using

title, number, and revision. and revision. title, number, cumentation are greatly appreciated. Send any Send any appreciated. greatly are cumentation and include the documentation and include the documentation , and paragraphs whenever possible., and paragraphs [email protected]

feedback to Your comments and suggestions to improve this do this improve to and suggestions comments Your Comments Refer to specific pages, sections specific to pages, Refer Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com vi High Availability Cluster User Guide User High Cluster Availability

1 1 r server, then, when HAC HAC then, when server, r , through which they exchange heartbeats that that heartbeats exchange which they , through as the primary or secondary system. You manage manage You system. or secondary primary the as manually, or automatically, if one appliance fails. or automatically, manually, Introduction to HA Cluster Introduction Protection of services, such as iSCSI, involves involves iSCSI, such as of services, Protection and share it with the othe with the it and share orage, although only one system provides the access to a shared volume volume a shared to the access provides only although system one orage,

Sample Network Architecture Network Sample Additional Resources Storage FailoverStorage Storage to Access Exclusive About SCSI Reservation Service Failover About the Nexenta HA Cluster Nexenta About the • • • • • • • provide information about their states and the services that reside on them. on reside that services the and states about their information provide Service Cluster of: consists that unit A transferable • and shutdown code Application start-up • Network identity and appliance data NexentaStor Appliances NexentaStor NexentaStor these connects HAC failures. for other each monitors and services of set defined a Runs channels communication various through appliances Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com The Nexenta HA (High-Availability) Cluster consists of two NexentaStor appliances and provides a storage storage a and provides appliances NexentaStor of two consists (High-Availability) HA Cluster The Nexenta is designated service. Neither system volume-sharing st shared for actively both systems server one on volume a create you time. After a at About the Nexenta HA Cluster About the Nexenta HA This section includes the following topics: topics: the following section includes This • appliances cluster between services migrate can You detects a system failure, it transfers ownership of the shared volumes to the other server in the Cluster in the Cluster server other the to volumes shared the of ownership transfers it failure, system a detects pair. serverand monitoringfailover. provides HA Cluster plugin. the SCSI Target as such other modules with cooperation includes: An HA Cluster •

ve node to the passive node. This the passive ve node functionality to igured volume services failover to the other server. server. other the to failover services volume igured If the state of one changes to offline, it prevents prevents it offline, to one changes of the state If over storage services if one of the cluster nodes is nodes cluster of the one if services storage over of time it takes to failover the data volume on the volume the data failover to takes it time of h network device to monitor based on the services based deviceto monitor h network . can place a disk before accessing the file systems, and file the systems, accessing before disk a place can g logic; the disk heartbeats, however, are essential. are however, g logic; the disk heartbeats, nal events, including that appliances power outages, nal events, en one HA Cluster node has access to a shared volume, volume, a shared to access node has en one HA Cluster the disk heartbeat. Generally, additional heartbeat heartbeat additional Generally, the disk heartbeat. appliance that currently owns the corresponding the corresponding owns currently that appliance over time include using fewer zvols and file systems for for file and systems zvols using fewer time include over that interface. When the interface recovers, HA Cluster HA Cluster recovers, When the interface interface. that shared volume. The reserved disk drives do not process process do not disk drives The reserved volume. shared e for fencing in computer clusters. in computer fencing e for that an appliance has failed is approximately 10 seconds. The failover The failover seconds. 10 is approximately failed has an appliance that [email protected] is analogous to Stonith, the techniqu Stonith, to is analogous

• acti the from volume the shared export to Failure

commands from the alternate HA Cluster node until the lock is released. released. is lock node untilthe HA Cluster the alternate from commands drives. heartbeat or devices spare to applied not are SCSI reservations contact information, more For In a cluster environment multiple hosts have access to the same shared storage. To control access to the the to access control To storage. the same shared to access have multiple hosts environment a cluster In used. are reservations SCSI storage shared wh Therefore, reservation. SCSI-2 supports Cluster HA About SCSI Reservation in the drives the disk to reservation SCSI applies it You access a shared volume exclusively through the through exclusively volume a shared access You the utilization through fencing reliable provides Cluster HA this exclusivity, ensure volume-sharing service.To protecting and/or HA Cluster, an in a node of isolating process the is Fencing heartbeats. of types of multiple communication pinging, constant or a nodemalfunctions. whenallow for Heartbeats, resources shared is of these important The most the servers. between Exclusive Access to Storage HA Cluster ensures service during continuity exceptio ensures Cluster HA fencin cluster's of the reliability mechanisms increase appliancein certain cases: the failed reboot can Cluster HA The primary benefit of HA Cluster is to detect storage system failures and transfer ownership of the shared shared of the ownership and transfer failures system storage detect is to Cluster of HA benefit primary The appliance. All conf NexentaStor the alternate to volumes Storage FailoverStorage other failures. and memory or crash, run out of detect the minimum time to Currently, on the amount dependent largely is time and recovery alternate appliance. Best practices to reduce the fail reduce to practices Best appliance. alternate each datavolume. fail implements HA Cluster configuration, default the In whic determines automatically HA Cluster unavailable. so even if a isnode services, not running any in the all cluster, It checks nodes boundan interface. to are that HACluster continues to monitor the unused interfaces. which you support a SCSI reservation, disk On systems on a disk system. the data protect to serves also feature This reservation. the if it loses panic to the system set failover to this node for services that are bound to bound to are servicesthat this node for to failover enables failover for that interface again. interface that for failover enables Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com 2 High Availability Cluster User Guide User High Cluster Availability

3 High Availability Cluster User Guide User High Availability Cluster node B node NEXENTA APPLIANCE: APPLIANCE: SHARED VOLUME ALTERNATE PATH ALTERNATE J4400 Su n S to rage of the HA Cluster. of the HA Cluster. 1000GB 7.2Krpm 1000GB 7.2Krpm 1000GB 7.2Krpm 1000GB 7.2Krpm 1000GB 7.2Krpm 1000GB 7.2Krpm SATA SATA SATA SATA SATA SATA OK OK OK OK OK OK 1000GB 1000GB 1000GB 1000GB 1000GB 1000GB 7.2Krpm 7.2Krpm 7.2Krpm 7.2Krpm 7.2Krpm 7.2Krpm SATA SATA SATA SATA SATA SATA K K K K K K O O O O O O 1000GB 7.2Krpm 1000GB 7.2Krpm 1000GB 7.2Krpm 1000GB 7.2Krpm 1000GB 7.2Krpm 1000GB 7.2Krpm SATA SATA SATA SATA SATA SATA OK OK OK OK OK OK e failover of ownership of the shared volume to the to volume of the shared ownership of e failover 1000GB 7.2Krpm 1000GB 7.2Krpm 1000GB 7.2Krpm 1000GB 7.2Krpm 1000GB 7.2Krpm 1000GB 7.2Krpm SATA SATA SATA SATA SATA SATA ETHERNET NETWORK ETHERNET ich are connected to the Nexenta appliances and nodeA appliances the Nexenta to connected ich are CLIENT SIDENETWORK OK OK OK OK OK OK Cluster deployment of a Nexenta iSCSI environment. The iSCSI environment. a Nexenta of deployment Cluster HA Cluster migrates the storage services that are are that services the storage migrates Cluster HA tive/passive function function tive/passive arts the services on the alternate node. on the alternate the services arts JBODs PATH PRIMARY PRIMARY node A node NEXENTA APPLIANCE: APPLIANCE: Nexenta recommends that you separate management and network traffic, so the heartbeat the heartbeat so traffic, and network management separate you that recommends Nexenta network. the management through goes traffic • storage SAS with a shared systems x86/64-bit metal bare Two • practice) but good mandatory, (not cards interface network Two associated with the shared volume and rest volume with the shared associated Architecture Sample Network Asample cluster hardwaresetup includes: As discussed previously, system failures result in th in result failures system previously, As discussed Service Failover Service process, failover of the part node. As alternate The following illustration is an example of an HA is an example illustration following The host server attaches to in to iSCSIhost server wh LUNs the JBOD, attaches nodeB. The NexentaStor appliances use the ac appliances NexentaStor The nodeB. configuration. Cluster of HA shows an example diagram following The Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com

training courses on high courses training plugin. Nexenta also offers also offers plugin. Nexenta assist with managing HA Cluster. Nexenta strongly strongly Nexenta with managingCluster. HA assist

encourages a services engagement to plan and install the plan install and to engagement a services encourages features. other and availability Nexenta has various professional services offerings to to offerings services professional various has Nexenta Additional Resources Additional Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com 4 High Availability Cluster User Guide User High Cluster Availability

5 2 ious cluster, so the following sections so the following ious cluster, . Configuring the HA Cluster the HA Configuring ugh the appliance’s web interface, the Nexenta Nexenta the interface, web ugh the appliance’s “Creating a volume” “Creating . . , section This section applies to new installations of HA Cluster. When upgrading, you save save When you upgrading, of HA Cluster. new installations This section to applies prev your from the configuration and restore apply. do not HA Cluster the NexentaStor Installation Guide Installation NexentaStor Guide User NexentaStor Modifying the Default Netmasks Modifying the Default Note: Configuring the HA Cluster Configuring the HA Cluster to Volume Adding a Shared Volume a Shared Importing Volume Shared a Removing Prerequisites SSH using Binding the Nodes Registering Adding a Virtual IP Address About Configuring the HA Cluster the HA About Configuring • • • • • • • • See See See • Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com •appliances. NexentaStor both on plugin HA Cluster the Register Management View (NMV), or the Nexenta Management Console (NMC). Console Management ViewNexenta the Management (NMV), or tasks: following the complete the HA Cluster, configuring Before • appliances. metal NexentaStor bare two Configure •appliances. NexentaStor the to storage a shared Connect • on the primary HA volume node. a shared Create Prerequisites About Configuring the HA Cluster About Configuring the thro Cluster HA the and manage configure can You • SSH. using appliances the NexentaStor Bind This section includes the following topics: topics: the following section includes This

. You must bind bind must . You on the other node. Step 7 Step and file. HA Cluster does not support file. does HACluster Step 5 Step — er default failover hostnames or the or the hostnames failover er default Modifying the Default Netmasks the Default Modifying /etc/hosts Step 1 Step . . , repeat , repeat volume and modify the NexentaStor host table. host and modifythe NexentaStor volume HA-pair SSH port number and user name. user and number SSH port SSH-Bind . on the other node. on the other . Bidirectionally field, select Bidirectionally. ndings, complete the steps in the steps ndings, complete Step 4 Step panel, click panel, — HA Cluster . Step 1 Step binding using IP addresses. bindingusing IP addresses. You must bind the appliances using eith bind the appliances must You in the specified you that hostnames Network type Binding the Bind Settings > Network Settings Host name of the remote HA Cluster node HA Cluster remote the name of Host node HA Cluster of the remote IP address node. HA Cluster on the remote user root for Password nmc:/$ setup network ssh-bind ssh-bind setup network nmc:/$ ssh-bindings show network nmc:/$ Type the host name of the NexentaStor appliance that you want to bind. to want you appliance that NexentaStor of the name host the Type password. the root Type Repeat Log in to one of the NexentaStor appliances NexentaStor the one of in to Log the following: Type Click To verify that you set up the bindings correctly, type: bindings correctly, the up set you that verify To In the In the Specify the following: • • • modify the default Optionally, privileges. administrative have The user must settings. default use doubt, in When Optionally, select Click If you did not select If you 3. 4. 5. To bind the two nodes, using NMC: using two nodes, the bind To 1. 2. To bind the two nodes, using NMV: using two nodes, the bind To 1. 6. 2. 3. 4. 5. 6. 7. 8. Adding a VirtualAddress a IP Adding Registering Note:

See   See

• shared the for address virtual IP Configure Before you configure the SSH bi configure you Before Binding the Nodes using SSH Binding the the two HA nodes together with the SSH protocol so that they can communicate. communicate. can they so that SSH protocol the with together HA nodes two the Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com 6 High Availability Cluster User Guide User High Cluster Availability

7 High Availability Cluster User Guide User High Availability Cluster . Modifying the Default Modifying the Default Start RSF Start r license to use this functionality. This use this r license functionality. to nce. You can use it right after installation using installation after right it use can nce. You . key or type the existing one. the existing or type key OK for the second node. second the for of the following options: the following of . Step 4 Step - . . . to install the HA license to the the HA license othernode. to to install Step 1 Step Yes Manual Confirm You may get a warning that you do not have sufficient number of appliances to to appliances of number sufficient have not do you that warning a get may You this warning, see get If you group. Cluster HA an create Netmasks If the other HA Cluster node is unlicensed, NMV notifies you about that. node you NMV is unlicensed, notifies HA Cluster other the If Click Click OK. Repeat Click an HA node. Select license. the HA Cluster Type When prompted, accept the HA Cluster license agreement. license agreement. Cluster HA the accept When prompted, e-mail address. your Type click When prompted, Click functionality servers is that for provided do not access have to the Internet. You must have a hard copy copy of the HA Cluste a hard have must You The temporary HA Cluster license automatically registers on your NexentaStor NexentaStor on your registers license automatically HA Cluster The temporary e-mail address. the provided to sent is also package license The appliance. Settings> HACluster 4) 5) 1) 2) 3) 2) 3) 4) 1) If you do not have an access to the Internet: an to access doIf you not have If you have Internet connection: Internet have If you Click Accept the HA Cluster license agreement. agreement. license HA Cluster the Accept isnotwarning HA Cluster running, a that get If you click You can install a temporary 45-days trial license. 45-days temporary a install can You license. HA Cluster type a permanent can you Alternatively, • NexentaStor starts the HA Cluster daemon. Cluster HA the starts NexentaStor license an HA Cluster Request select one When prompted • To register HA Cluster, using NMV: Cluster, HA register To 1. 2. 3. 5. 4. Note: Note:  HA Cluster is pre-installed on the NexentaStor applia on the NexentaStor is pre-installed HA Cluster Registering the HA Cluster the Registering Nexenta. from received you license that permanent the or enter trial license temporary Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com

IP_address hostname [aliases...] hostname IP_address hosts appliance setup nmc:/$ nodeB.example.com> Example: Type the following to open to the the following Type Type the following to open to the the following Type the Using Log in to the NMC Log in one of on to 4. To modify the default netmask, using NMC: using netmask, default the modify To 1. 2. 3. To verify using NMC: hostnames, To 1. Note: Adding a Shared Volume to the HA Cluster the to Volume Shared Adding a

•  

See Also: When you add a volume service to the HA Cluster, Nexe HA Cluster, the service volume to a add When you Modifying the Default Netmasks change to need may you However, is on. it network changes. There is a name associated with a shared volume service that is referred to as a as to referred is servicethat volume a shared with associated name is a There Adding a Virtual IP Address a Virtual Adding hostname, or virtual IP address (VIP). The network clie The network (VIP). virtual IP address or hostname, volume. must specifyvirtual a VIPandsh a corresponding You NexentaStor SSH-binding, an create you When communicate. and network the on other each detect on tables IP host the to records adds automatically a VIP. add when you correctly added information: withfollowing the single a should linebe host present each For The appliances in the HA cluster group must be resolvab must group HA cluster the in appliances The Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com 8 High Availability Cluster User Guide User High Cluster Availability

9 High Availability Cluster User Guide User High Availability Cluster Modifying the Default Modifying the Default the hostnames of NexentaStor nodes are are nodes of NexentaStor the hostnames . liance to more than one HA Cluster. more liance to Initialize. . . You need to configure multiple options for the HA cluster HA cluster the multiple options for configure to need . You rify that you completed the steps in the steps completed you rify that Settings > HA Cluster > HA Settings for nodeB. Cluster name Cluster Step 2 Step . — . Step 1 Step If you boundusingyou the appliances If SSH, the HA Cluster. of nodes as selected automatically You cannot assign a NexentaStor app assign a NexentaStor cannot You Enable Network Monitoring Network Enable Configure Yes Modifying the Default Netmasks Modifying the Default nmc:/$ show group rsf-cluster show group nmc:/$ nmc:/$ create group rsf-cluster create group nmc:/$ 192.168.1.0192.168.13.0192.168.0.0 255.255.255.0 255.255.255.0 Repeat 255.255.0.0 Type: instructions. the on-screen Follow type: group, the HA Cluster created you that Verify On the first node, select the first On Add the netmask for each network address: network each for Add the netmask Example: In the Cluster clickSettings Inpanel, the Cluster the or change Type Optionally, type a description. a type Optionally, Select nodes. for network the monitors The cluster Click Click and To configure an HA cluster, using NMC: using an HA cluster, configure To 1. 2. 3. To configure an HA cluster, using NMV: using an HA cluster, configure To 1. 2. 3. 2. 4. 5. 6. 7. Note: Note:   After you configure an HA Cluster, you are prompted to add a shared volume to the volume service. volume the to volume a shared add to prompted are you Cluster, HA an configure you After time. later any at Cluster the to volume a shared add can you Alternatively, Adding a Shared Volume to the HA Cluster Adding a Shared Volume to the HA Cluster Configuring the HA Cluster Configuring ve the HA Cluster, configure you Before Netmasks before you can use it successfully. can you before Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com

file. /etc/hosts shared-volume add shared-volume different JBODs as heartbeat devices. General General devices. heartbeat JBODs as different . . eat signature from each node which can reduce the life life the reduce can which node each from signature eat Volumes . and netmask. NONE . devices. . Failover Hostname You can also type also can You Do not assign SSDs as heartbeat devices. Heartbeat devices perform a constant constant a perform devices Heartbeat devices. heartbeat as SSDs not assign Do heartb the of periodic write of SSDs. To ensure better performance and reliability, use non-primary network interfaces non-primary use interfaces network reliability, and performance better ensure To for heartbeats. Primary appliance Heartbeat Heartbeat Add this volume to the cluster to this volume Add Confirm System response: System volume Shared : for 'shared-volume' HB disks c1t6d0 : c1t5d0, 192.168.60.22 192.168.60.22 Select heartbeat devices. devices. Select heartbeat response: System volume. selected the for Specifyvirtual IP address a Type: In the Cluster clickSettings Inpanel, the Cluster the for address the IP type to you prompots NMV hostname, up the failover did not set If you volume. shared Select Example: Select a volume from the drop-down menu. the drop-down from Select a volume the Type in the specified previously have you that hostname failover the Use services. Cluster HA the for timeouts standard modify initial and Optionally, Select from multiple use JBODs, select drives If you Select failoverinterfaces for both nodes. Click Click practice is to assign 2 — 6(max) heartbeat drives. heartbeat 6(max) — 2 assign to is practice setup group rsf-cluster rsf-cluster setup group 2. 3. To create a shared service, a shared create using NMC: To 1. 4. 5. To add a shared volume to HA Cluster, using volume addshared a HACluster, NMV: to To 1. 2. 9. 3. 7. 6. 8. 10. Note: Note:

Warning:  

11 High Availability Cluster User Guide User High Availability Cluster loghost loghost e1000g1 20 8 e1000g1 20 8 20 e1000g1 . . Step 5 Step - Step 3 Step Standard timeout Standard tor appliance where the shared volume is volume the shared where appliance tor and 192.168.60.79 Example: Main node Main stopped unblocked auto running running unblocked auto Failover Netmask If you did not define a hostname/IP address for the shared volume, NMC the shared prompts for If you did a not define address hostname/IP table. host modifythe NexentaStor to you VIP1 Failover Netmask : 255.255.255.0 : Netmask VIP1 Failover response: System (y/n) VIPs? Stop adding VIP1 Shared logical hostname : : hostname logical VIP1 Shared imported after you complete th complete you after imported Mainor Primary node node is the NexentaS modify the Optionally, Confirm the VIP configuration by Confirm thetyping VIP configuration Example: Select network interfaces for the selected VIP(s) on both HA Cluster nodes. Cluster HA both on VIP(s) selected the for interfaces Select network Specify System response: System If you want toadd additional VIPs, type Select the Example: 9. 6. 4. 5. 7. 8. Note: Importing a Shared Volume ma you control, volume under cluster a shared put To NodeB: or appliance NexentaStor a upgrade you when Enable SCSI PGR reservation by typing y. y. by typing reservation SCSI PGR Enable ' volume shared Adding Jan 8 23:20:49 nodeA RSF-1[19746]: [ID 702911 local0.alert] RSF-1 cold restart: RSF-1 cold restart: [ID 702911 local0.alert] Jan 8 nodeA RSF-1[19746]: 23:20:49 stopped. All services done...... to complete operation for add Waiting HA-Cluster STATUS: HA CLUSTER NodeA: Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com

shared-volume shared-volume , please wait ... please wait , rted on the active HA Cluster node. However, the However, node. Cluster HA on the active rted . . Volumes . y Import tab. panel, click for the relevant volume. the relevant for Remove a volume All Volumes Data Management > Data Sets. > Data Management Data Import Removing shared volume shared Removing nmc:/$ setup group rsf-cluster group rsf-cluster nmc:/$ setup response: System ? HA Cluster and restart Remove shared volume (y/n) remove nmc:/$ setup volume import setup volume nmc:/$ System response: System Type: by typing the operation Confirm In the Cluster clickSettings Inpanel, the Cluster Click Select the In the volume. Select a Click . Click Type: 2. To remove a shared volume, using NMC: using volume, a shared remove To 1. To remove a shared volume, using NMV: using volume, a shared remove To 1. To import a shared volume, using NMC: using volume, a shared import To  To import a shared volume, using NMV: using volume, a shared import To 1. 2. 3. 4. 2. 3.

   

volume becomes unavailable through the failover hostname. hostname. the failover through unavailable becomes volume You can remove a shared volume from the HA Cluster control any time. The remove operation does not not does operation time. The remove any control Cluster HA the from volume a shared remove can You impo remains The volume the volume. on data any delete Removing a Shared Volume Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com 12 High Availability Cluster User Guide User High Cluster Availability

13 3

split brain egate) for additional in the blank 8K space of the VDEV label on VDEV label of the space 8K in the blank e, the heartbeat drives are still used to store store to used still are drives e, the heartbeat ted "cross-over" connection between the nodes. the nodes. between connection "cross-over" ted pe (simple, IPMP, aggr pe (simple, IPMP, monitors the state and status of the other appliance and status the state monitors r servers must determine that an appliance (member an that determine must r servers s services, you configure the cluster to use several several use to the cluster configure you s services, nning on another server, leading to so-called so-called to leading server, another on nning s of heartbeat communication channels: communication heartbeat s of

Heartbeat and Network Interfaces Network and Heartbeat each of the selected heartbeat devices. Therefor devices. heartbeat the selected of each data. is a dedica connection heartbeat The preferred ty interface other any or add use also can You Accessible and writable from all appliances in the cluster or VDEV labels of the devices in the VDEVdevices of the or labels in the cluster all appliances Accessiblewritable and from volume. shared the perform volume in the shared devices VDEV labels for devices, select the heartbeat you If heartbeat function. If VDEV volume uses a shared disks, labels consists NexentaStor of a few for which disks. specify can mechanism.You heartbeat the for disks more one or and 518 512 sectors mechanism uses The heartbeat resiliency. Heartbeat Mechanism Heartbeat Properties Modifying Heartbeat About Heartbeat and Network Interfaces and Network About Heartbeat • • • • Network Interfaces • Network • Heartbeat/Reserved devices • Heartbeat/Reserved Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com in the Cluster through heartbeats. Because HA Cluste heartbeats. in through the Cluster Heartbeat Mechanism it may failure, a detects appliance wrongly If an failure. a represents channels heartbeat all The loss of ru is already a service that start to attempt A NexentaStor appliance in the HA Cluster constantly constantly appliance in HA the Cluster A NexentaStor About Heartbeat and Network Interfaces About Heartbeat and it over taking before failed has the cluster) of This section includes the following topics: topics: the following section includes This communication channels through which to exchange heartbeats. communication channels exchange whichthrough to this prevent heartbeats Multiple, corruption. redundant and data confusion in result This can syndrome. occurring. from type supports the following HA Cluster

. Yes | No Yes hb_properties button that will automatically create the the create will that automatically button Add a network heartbeat network a Add cluster_name> . and select and dialog, type the IP address or a hostname available on a available hostname or a address IP dialog,type the tab, tab. . Heartbeats Recommend Disk Layout Recommend . . . , click Test Network heartbeats Network Appliance Heartbeats Volume Heartbeats . You can click can the You configuration. heartbeat You can include a NexentaStor node only to one HA Cluster. node only to includecana NexentaStor You Create network heartbeat network Create Settings> HACluster OK Save Settings Save Save Settings Save Cluster Settings Cluster Enable inter-appliance heartbeat through primary interfaces?: primary through interfaces?: heartbeat Enable inter-appliance nmc:/$ setup group rsf-cluster < rsf-cluster setup group nmc:/$ response: System • Follow the on-screen instructions. the on-screen Follow Click the Click the Type: Click the Click the Click In Click remote NexentaStor appliance. NexentaStor remote click Optionally, Right click on a disk and select a vdev. select and disk a on click Right Click on click Right In the Click 2. To change heartbeat properties, using NMC: using properties, heartbeat change To 1. 1. 1. To change heartbeat properties, using NMV: using properties, heartbeat change To 1. 2. 5. 4. 2. 2. 3. 3. 6. Note: Note:

 

Adding network heartbeats Adding device heartbeats When you define a cluster, you define the heartbeat properties. However, you can modify them later, if modify later, them can you However, properties. heartbeat the define you cluster, a define When you needed. Modifying Heartbeat Properties Heartbeat Modifying Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com 14 High Availability Cluster User Guide User High Cluster Availability

15 4 ion results in a warning a warning in ion results r. By default, the operat default, By r. inuity in the presence of service level exceptional inuity of service exceptional level in the presence is case, the action fails completely if the remote remote if the completely fails action the is case, that you always verify that shared volume is in volume shared verify that always you that I in a cluster environment, make sure that you are are you that sure make environment, I in a cluster er. Moreover, HA Cluster does not disk does detect HA and Cluster Moreover, er. ers ownership of shared volumes to the alternate the to alternate volumes of shared ownership ers Therefore, if one or more disks in the shared volume volume in the shared disks one or more if Therefore, ter you place the volume under cluster control. control. place the under cluster volume you ter ross the cluster members. HA Cluster automatically automatically Cluster HA members. cluster the ross Configuring Storage Failover Storage Configuring be reflected on the other node. on the other reflected be nmc:/$ setup appliance nms property rsf_config_update_synchronous nms property appliance setup nmc:/$ Type: To set the synchronous mode, using NMC: NMC: using mode, synchronous the set To 1. Configuring Fibre Channel Targets for Failover for FibreConfiguring Channel Targets Cluster Configuration Data Configuration Cluster Mapping Information Failover NFS/CIFS Failover for Configuring iSCSI Targets About Configuring Storage Failover Storage About Configuring  • • • • • • Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com events, including power outage, appliance running out of memory, or crashing, etc. crashing, or memory, appliance running out of outage, including power events, failures. disk not repair does failover The Cluster HA or iSCS either FC for SCSI targets configure When you Cluster Configuration Data Cluster Configuration NexentaStor appliance. HA Cluster ensures service ensures cont appliance. HA Cluster NexentaStor them using failov repair cannot you unreachable, are long time take may operation the failover volume, a degraded for failover execute If you failures. volume recommends Nexenta Therefore, or hang the system. failover. executing before state, healthy and mappings ac with configurations consistent at configured not or available node is not if the alternate However, operations. SCSI Target all propagates occu can problems change, of the configuration the time About Configuring Storage Failover About Configuring Storage transf and failures system storage detects HA Cluster This section includes the following topics: topics: the following section includes This to the User that the remote update failed. failed. update remote the that User the to af changes and iSCSI all FC configuration Execute willnot Otherwise, the changes In th mode. synchronous to HA set also Cluster can You update fails. update

n is specific to the volume and is stored with the and the is stored n is specific to volume ion in sync betweensituations, In certainnodes. thesync ion in two cluster nodes offline, the configuration may become become may the configuration offline, nodes cluster two nger to fail over and in some cases may require require may and in some cases over fail to nger ) de) and then use NMC commands to restore it in the event of a failover. a of it in the event restore to NMC commands use then de) and properties that migrate between nodes automatically upon failover. failover. upon automatically nodes between migrate that properties iscsi.conf stmf.config) — Strict Synchronous update across the cluster across update — Strict Synchronous fails. update if synchronous update — Asynchronous 1 0 command restores previously saved configuration data that includes: that data configuration saved previously restores command System response: System RSF-1 ‘rsf_config_update_synchronous'. property modify NMS View or update Synchronous Strict 1 - update mode. configuration Appliance update synchronous if update, Asynchronous 0 - the cluster, across exit. to or Ctrl-C hjkl), (or keys with arrow Navigate fails. Select an appropriate value: Select an appropriate • • 1.

• Target groups • Target • ( groups Host • Targets • Initiators • portal ( groups Target restore

volume itself. volume the HA configurat keep to attempts software cluster The of the with one changes SCSI making Target such as NFS/CIFS Failover does Cluster HA that note However, users. to shares NFS of the availabilityensure to Cluster use HA can You server software. of NFS the the failure detect not volume-level are settings NFS/CIFS on both nodes be defined to need idmapsuchmay as settings However, failures. server CIFS detect not does Cluster HA Mapping Information nodes cluster the that is critical It systems. client to nodes cluster the from map zvols to SCSI Target Use theMapping same mappingcontain informatio information. lo takes configuration An out-of-sync out-of-sync. administrator’s actions to resolve. To protect local configuration information that did not migrate, periodically save this configuration to a to this configuration periodically save didnot migrate, that information configuration local protect To no the (perhaps alternate site remote The Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com 16 High Availability Cluster User Guide User High Cluster Availability

17 High Availability Cluster User Guide User High Availability Cluster and portal define group. a target zvol to the other appliance on failover. on failover. appliance other the to zvol rs to the target portal group. The newly created The newly created group. portal the target to rs the network, however, you can verify it with it with the verify can you however, the network, y also see command level failures or disconnects or disconnects failures level also y see command ified when the volume service was created, rather rather created, service volume the was when ified one cluster node to another. The target IQN moves IQN moves The target another. node to one cluster ed logical hostname configuration. Therefore, do not do Therefore, configuration. hostname logical ed rthermore, unless the shared volume serviceis volume the shared unless rthermore, lume as a shared cluster volume, then when you share share when you then volume, cluster a shared lume as y experience some delays. Depending on the network, network, the on Depending delays. some y experience . e active appliance, using NMV: using appliance, active e iSCSI > Target Portal Groups Portal iSCSI > Target create . Note that you perform the process of creating a zvol and sharing it through iSCSI iSCSI through it sharing and a zvol of creating the process perform you that Note configuration. Cluster HA the from separately It is critical that the IPv4 portal address is the shared logical hostname specified specified hostname logical is the shared address IPv4 portal the that critical is It with associated hostname a real of instead created, service was volume the when the appliances. one of Data Management > SCSI Target. Management Data iSCSI > Targets Click Click HAC automatically replicates the newly created target portal group to the other appliance. to group portal target created the newly replicates automatically HAC In the zvols panel, click panel, In the zvols virtual theblock device > 200MB. Make created the newly migrates automatically HAC it manually. duplicate to do not have you Therefore, click pane, the iSCSI From visibility Thiszvol limits initiato client from the other appliance. to replicated is automatically iSCSI target an alias. a name and Type page. displays iSCSI inthetarget The Targets newly created To create an iSCSI target and add it to the target portal group, using NMV: using group, portal target the it to add and target iSCSI an create To 1. 1. To configure iSCSI targets on th on iSCSI targets configure To 3. 4. 2. 2. Note: Note:   application environment and active workload, you ma you workload, and active environment application during this period. iSCSI targets, for When as backvolume add a you shared which to a cluster up storage has zvols created those to access system, of the operating regardless iSCSI initiators, all client configure you itthat isvital is spec that hostname logical using the shared targets the appliances. with one of associated hostname a real than of the shar all aspects manages the cluster that Note Fu manually. hostname logical the shared configure the cluster volume as an active iSCSI session, it ma iSCSI session, an active as volume the cluster on running, logicalis hostnamenot present the shared You can use HA Cluster to failover iSCSI volumes from iSCSI volumes failover use to HA can Cluster You vo marking the zvol’s before zvols iSCSI create If you Configuring iSCSI Targets for Failover Targets for iSCSI Configuring the failover. of part as volume. shared the in up a zvol setting involves failover up iSCSI Setting ICMP ping command. Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com

port port ACTIVE state. This would be would This state. mode on it and the second mode on it and the second ting failover for a pool containing a pool containing ting for failover STANDBY ACTIVE backup storage for the iSCSI for The newlytarget. backup storage th the volume service and the target portal portal target service and the th the volume e both the IQN of the iSCSI target created and created target iSCSI the e both the IQN of mode. In case of a failover, the a failover, of mode. In case ent IP addresses for iSCSI. Normal running mode is iSCSI. for IP addresses ent . metric Logical Unit Access (ALUA) mode, NexentaStor mode, NexentaStor (ALUA) Access Unit Logical metric r failover, you must complete the following tasks: tasks: the following complete must you r failover, Nodes may stall while the failover occurs, but but otherwise occurs, while the failover stall may Nodes STANDBY ACTIVE Mappings. . NexentaStor User Guide User NexentaStor becomes becomes e environment. As an example, initia example, As an e environment. STANDBY and the and Settings> HA Cluster Log in to an HA Cluster node. an HA Cluster in to Log Click This creates a LUN mapping to the zvol for use as for the mapping zvol to a LUN This creates failover. on the other appliance to migrated is automatically mapping LUN created us to the iSCSI initiator configure the client, On From the SCSI Target pane, click pane, the SCSI Target From with bo associated hostname logical the shared iSCSI. through the zvol access to group STANDBY To set the HA Cluster to ALUA mode:set the HA Cluster ALUA to To 1. 2. 2. To create a LUN mapping to the zvol, using NMV: using zvol, the to mapping LUN a create To 1.

  Creating a Zvol a Creating Mapping a Zvol Setting the HA Cluster to ALUA Mode ALUA to the HA Cluster Setting Mode Port the HBA Changing Group Creating a Target Group an Initiator to Adding WWNs • “Managing SCSI Targets” in the • “Managing SCSITargets” • SCSI Target FC User Guide User FC • SCSI Target

similar to having the same LUN presented via two differ via two presented the same LUN having similar to to have the first head doing the FC work with the FC port with the LUNS in port with the LUNS with FC work the doing the FC head the first have to The FC target is tied to the WWN. WWN is hardware based and since each FC HBA has a different HW WWN, different a has HBA FC and since each based WWN.WWN the is hardware to is tied target The FC Asym enable the you When over. fallen be cannot it • • Mode Setting the HA Cluster to ALUA To configure the fiber channel targets for HA Cluste for the fiberchannel targets configure To • • • • first. with the node second the on ports the FC communicate can nodes node in the second on LUNs the exported presents COMSTAR Therefore, Configuring Fibre Channel Targets for Failover Configuring Fibre Channel becomes becomes NexentaStor node would have the same LUNs in the same LUNs nodewould have NexentaStor recover quickly. See Also: Failover time varies on depending th time varies Failover time is 32 seconds. failover the observed six zvols, Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com 18 High Availability Cluster User Guide User High Cluster Availability

19 High Availability Cluster User Guide User High Availability Cluster client to maintain access to the to access maintain client to . . at least one local and one remote port defined within port defined and one local one remote least at checkbox. may result in the inability a SCSI of result may node that you are configuring. are node that you in the same cluster. node second GLobal Cluster Properties GLobal Cluster from the Mode dropdown menu. the Mode dropdown from Enable ALUA mode Enable ALUA Target Data Management > SCSI Target Plus Plus Target SCSI > Management Data Data Management > SCSI Target Plus Target SCSI > Management Data Click Click Select Advanced > Advanced Select Select the Select Fibre> Ports. Channel Select Target mode to Initiator bothof from appliances modes port the HBA change you Once mode, reboot mode both effect. changes appliances can take so the Target

To create a target group, using NMV: using group, target a create To 1. To change the HBA port mode, using NMV: using port mode, HBA the change To 1. 3. 4. 2. 4. 3. Creating a Zvol a Creating Setting the HA Cluster to ALUA Mode Creating a Zvol a Creating Group Creating aTarget Adding WWNs to an Initiator Group Initiator an WWNs to Adding Creating a Target Group Creating aTarget  • •  • • An FC port that resides on the resides that port An FC Remote An FC port that resides on the resides that port An FC Local • • • have must Group a Target failover, proper ensure To storage upon a failover event. a failover upon storage FC ports identified are as: Target • it. Creating a Target Group Creating a target create to Failure nodes. both from ports FC using the desired target an FC create useALUA, To nodes both from ports FC with group Changing the HBA Port Mode Changing the both of appliances port modes configuring the HBA change FibreChannel for targets, As a prerequisite mode. Target mode to Initiator from See Also: See Also: Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com

. here . window, click window, . de and assign initiators that are visible to that node. node. visible that to are that initiators and assign de tors not visible to the node that you are working on. working are you that the node visibleto not tors . window: Initiator Groups Initiator Target groups Target , specify a custom group name group , specifya custom . field, type the name of the target group. the target of the name type field, panel, click panel, Initiator Group here or Group Name Group SCSI Target Manage Groups Initiators of Remote Create New Initiator Group New Initiator Create Group Name Group Data Management > SCSI Target Management Data Create Log in to an HA Cluster node. an HA Cluster in to Log Click In the SCSI Target panel, click click panel, SCSI Target the In In the In the Click In the In the field In the ports. FC remote and one local one least Select at To create an initiator group, using NMV: an group, initiator create using NMV: To 1. 1. 2. 3. 2. 4. 3. 5. 4. 5. Mapping a Zvol Creating a Zvol a Creating Adding WWNs to an Initiator Group Initiator an to WWNs Adding

 • • •

Adding WWNs to an each SCSI client, Depending for anor of clients. group initiator group configure must on the configuration You to need may you Therefore, visiblenodes. be on some not may initiators SCSI client some SAN, of your initia for information initiator the client manually add Alternatively, youcan create an initiator group on one no second that visible from initiators adding by node other the from group initiator the modify to need Then you node. See Also: Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com 20 High Availability Cluster User Guide User High Cluster Availability

21 High Availability Cluster User Guide User High Availability Cluster . Create . s of additional initiators, not visible not this of additionalto initiators, s . here ppropriate initiator and target groups to ensure LUN LUN ensure to groups and initiator target ppropriate . . . . dialog, fill the required fields. required fill the dialog, NexentaStor User Guide User NexentaStor . window, fill in the required fieldsclick and fillin the required window, window, click window, Mapping Create panel, click panel, panel, click panel, . . Mapping a Zvol Mapping Management > SCSI Target SCSI > Management Management > SCSI Target SCSI > Management

If you do not have target or initiator groups, you can Share the zvol. If you share a share you If the zvol. Share can you groups, or initiator target have do not you If it is visible zvol, clients. all network to ZVOLS ZVOLS Create a New ZVOL Create Mappings Manage New Mapping Create Data Data Create Create For more information, see information, more For Click Click In the Additional Initiators field, type the WWN In the Additional Initiators comma. by separated node, Click In the In the In the In the In the Click Proceed to to Proceed To a zvol: map To 1. To create a zvol, usingNMV: create To 1. 6. 7. 2. 2. 3. 4. 3. 5. 4. Note: Mapping a Zvol Creating a Target Group Creating aTarget Adding WWNs to an Initiator Group Initiator an to WWNs Adding Creating a Target Group Creating aTarget Mapping a Zvol Creating a Zvol a Creating •  • •  • • • See Also: visibility and failover capability. capability. visibilityfailover and Mapping a Zvol a to LUNs using the FC created you that Map the zvol Creating a Zvol Creating LUNs. FC using the required a zvol Create See Also: Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com

Creating a Target Group Creating aTarget Group Initiator an WWNs to Adding Creating a Zvol a Creating

• • •

23 5 when it is not when it running. Advanced Setup Advanced not running else, anywhere g the failover mode, adding virtual virtual mode, adding g the failover pliance is not available. If the appliance cannot cannot If the appliance available. not pliance is itiate the failover when it detects a failure. a failure. detects it when the failover itiate rvice, or the service is attempts to start a service a start to attempts for each appliance that can run a service. a run can that appliance each for ons of HA Cluster, such as settin such as Cluster, ons of HA the HA Cluster Environment the HA Cluster other miscellaneous options. other miscellaneous Set failover mode to automatic every time you perform any maintenance to avoid avoid to maintenance any perform time you every automatic to mode failover Set events. failover unwanted Note: About Auto-Sync Replication in Replication About Auto-Sync Environment Service in HA Cluster the an Auto-Sync Creating Service Auto-Sync an for SSH-binding the Establishing Virtual a Applianceto IP Address BindingNexentaStor a Setting Failover Mode Failover Setting Addinga Virtual Hostname and Address IP Volumes Additional Managing Properties Cluster Global Managing About Advanced Setup About Advanced obtain a definitive answer about the state of the se of the state about answer a definitive obtain However, it generates warnings when the parallel ap when the parallel warnings it generates However, In manual mode, the HA Cluster service does not in not service does mode, the HA Cluster In manual • • • • • • • • Setting Automatic Failover Mode Failover Automatic Setting Setting Manual Failover Mode • Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com • Setting Manual Failover Mode Setting Failover Mode appliance an or not whether defines mode failover The About Advanced Setup functi advanced This section describes and volumes, and hostnames mode settings failover separate are There modes: the following to the failover set can You • This section includes the following topics: topics: the following section includes This

. shared-volume shared-volume shared-volume service is already running does not stop running not stop service does already is Set all Manual Set

service when it detects that there is no available available is no there that detects serviceit when pliance starts its primary its service(s)pliancestarts boot on up. tomatic failover mode is the default setting. default is the mode failover tomatic . manual automatic toconfirm. toconfirm. toconfirm. Setting a service to manual mode when the manual mode a service to Setting appliance. that on starting service from the service,prevents it only that in the cluster, services of the state the saves it an operation, performs HAC Before HA Cluster Once button. restore using the cluster the to re-apply later can you which state. saved the it clears the servicestate, restores Advanced Setup > Cluster Operations all Automatic > Set Operations > Setup Cluster Advanced Stop All Services Advanced Setup > Cluster Operations > Operations Setup > Cluster Advanced Yes Yes Yes nmc:/$ setup group rsf-cluster rsf-cluster group setup nmc:/$ nmc:/$ setup group rsf-cluster rsf-cluster group setup nmc:/$ Click Click Click Click Click Click Type: Type: To stop all services in the Cluster, using NMV: in the Cluster, all services stop To 1. To set the failover mode to automatic, using NMC: using automatic, to mode failover the set To  To set the failover mode to automatic, using NMV: using automatic, to mode failover the set To 1.  To set the failover mode to manual, using NMC: using manual, to mode failover the set To To set the failover mode to manual, using NMV: using manual, to mode failover the set To 1. 2. 2. 2. Note: Note:

    the appropriate timeout must expire before you can take any action. The primary service failover modes modes failover primary The action. service any take can you before expire must timeout appropriate the are typicallyset automatic to to ensure that an ap 

parallel appliance running in the cluster. Au cluster. the appliance running in parallel In automatic mode, the appliance attempts to start the start to attempts the appliance mode, automatic In Setting Automatic Failover Mode Setting Automatic Failover Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com 24 High Availability Cluster User Guide User High Cluster Availability

25 High Availability Cluster User Guide User High Availability Cluster . Add vips add vips . . Step 10 Step one or more HA Cluster nodes cannot resolve, resolve, cannot nodes HA Cluster or more one .

. Step 9 Step — network interfaces rather than specifying than rather interfaces network Step 1 Step tab. . from the Advanced tab in NMV add an advanced level level an advanced NMV tab in add the Advanced from Advanced . y . . Global Cluster Properties Global Cluster panel, click > to add more VIPsand repeat r properties NMV: using r properties n . Advanced to finish to VIPs. adding Cluster Settings y Settings> HA Cluster Additional Volumes Additional Save Settings Save If you want to add a volume to the cluster service, drag the volume from the left panel and and panel left the from the volume drag service, the cluster to a volume add to want If you the right from volume the service, drag the cluster from a volume remove to want If you Nexenta recommends that you configure additional configure you that recommends Nexenta the primary network interface. node. remote the for interface Select network netmask. the failover Type by typing the settings Confirm response: System (y/n) VIPs? Stop adding Click Select the Click required as Modify the properties In the Type type Alternatively, Select a shared volume from the drop-down list. the drop-down from volume Select a shared service. cluster the from a volume or remove Add • panel. in right the drop • in panel. panel and the left drop Click 6. 7. 8. To manage global cluste manage To 1. 2. 3. 2. To add aservice: to cluster volumes To 1. 9. 10. 3. 4. 5.

 

Managing Global Cluster Properties using in properties cluster the global manage can You fine-tuning the HA cluster. for of control When you configure an HA Cluster, you usually add a shared volume to the cluster service. Later you can add add can you service.the cluster to Later volume shared a usually you add Cluster, HAan configure When you service. cluster the to/from volumes existing or remove additional Managing Additional Volumes Managing Additional Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com 26 High Availability Cluster User Guide User High Cluster Availability

27 High Availability Cluster User Guide User High Availability Cluster so that the path to the to the path that so out scanning all of the all of out scanning to which the volume is which the volume to pport theiSCSI for on a targets Description per pool basis. the pool, and about of information cache a saves rapid enable to on, found is it the devices a pool with importing of system. on your devices on, the to option is set the default, By setting. recommended is that a group to refers iSCSI view an if function node the on found not failing over. only one on present is physically it though even node is considered the with node. The target passive. The other node is considered active. multi- side client with in conjunction used When is not rescanning target ensures ALUA pathing, al and failover, on required standbyineven when use, not node is valid checks continuously client the multi-path (because the to of the node standby and the path the state standby target). and getting for is used This property Cluster. Exploreranalyzing Internet is support information. supported. not the node. failover If disabled alternate the mode to mode is not transferred. failover the following has Cluster High Availability modes: •Automatic •Manual Comstar supportComstar Use ZFS cache file on pool import This cluster. the in pools all for cache ZFS Configure notificationEvent iSCSI creationForce group su failover Enable modeALUA Enable iSCSIview Forces using creation theGroupAll notification event Enable the cluster nodes, both from available target a SCSI Makes HA Plugin javascript debuggingPluginHA javascript Failover State is stickyHA debugging for javascript or disables Enables the failover transfers during the failover If enabled, Property . Properties 5-1: Global Cluster Table Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com

SSH SSH file on Regular /etc/hosts . About Auto-Sync Replication in the HA Cluster in the HA Cluster About Replication Auto-Sync appliance and eachnode of the usingHA Cluster volume service fails over to the healthy node and and node the healthy to over service fails volume lect a shared volume as destination. Local volumes Local volumes destination. volume as lect a shared me under other HA Cluster control, or a regular a regular or control, me under other HA Cluster from an HA volume to another HA volume, all nodes nodes all volume, HA another to an HA volume from ation, create bindings for VIPs on every node. VIPs bindings for create ation, dataset located on the shared-volume or the whole the or on the shared-volume located dataset Auto-Sync does not interrupt replication. replication. not interrupt does Auto-Sync l address (VIP) of the HA Cluster group. Cluster HA of the (VIP) l address rvices in the HA Cluster environment: environment: the HA Cluster in rvices e Auto-Sync replication interrupts. interrupts. replication e Auto-Sync e in the HA Cluster environment, you must complete additional tasks. tasks. additional complete must you environment, e in the HA Cluster every node. You must add IP address/hostname pairs for all nodes in the in all nodes for pairs IP address/hostname add must You Establishing the SSH-binding an ServiceAuto-Sync for binding option. binding See Bind the non-HA NexentaStor appliance to both nodes of the HA Cluster suing appliance of thethe both Cluster HA nodes to Bind the non-HA NexentaStor . To create an Auto-SyncTo service inthe HA Cluster environment: 1. Note:

hostnames. hostnames. servicereplicates that an Auto-Sync create If you volume the shared down, the node goes and node HA an of name host or the IP address to bind If you th Therefore, unavailable. becomes volume as destination. shared se must you When bindyou an appliance VIP, through must be bind to each other. other. each be bind to must replic volume the HA to HA volume the of case In shared the is unavailable, When HA nodes of the one Therefore, VIP. through available remains VIP. through available not are  NexentaStor volume. NexentaStor

• service, an Auto-Sync select a When create you •virtua the to appliance Bind the non-HA Make sure that you configured your appliance your described in as configured you that sure Make Environment Creating an Auto-Sync Service in the HA Cluster Creating an Auto-Sync Service in the HA Cluster Environment You can create the following Auto-Sync se Auto-Sync following the create can You • control. HA Cluster under volume a to volume NexentaStor a From servic an Auto-Sync configure If you About Auto-Sync Replication in the HA Cluster in the HA Replication Auto-Sync About Environment • a volu to control HA Cluster under a volume From the following: Configure • the non-HA NexentaStor ssh-bindings between Create Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com 28 High Availability Cluster User Guide User High Cluster Availability

29 SSH Bound High Availability Cluster User Guide User High Availability Cluster . Creating an Auto- an Creating ssh-bindingthe public window, in the in window, . e remote High Availability Cluster Cluster High Availability e remote SSH binding option. binding SSH VIP user with root credentials. user with root tion. When you create an create tion. When you . to the authorized key file appliance. on the local key the authorized to . checkbox. ng the shared-volume as destination. as ng the shared-volume Summary Network Settings Summary Network Creating an Auto-Sync Service Instance Service Auto-Sync an Creating . Regular SSH-Bind pliance and the HA Cluster group plianceandCluster the HA , Section e to the VIP using the VIP using the e to . on the remote node. on the remote Bidirectionally sed Authentication, is an easy way to make two High Availability High Availability two make to way an easy is Authentication, sed field, type the IP address of th of address the IP field, type field, the name of type the field, select panel, clickpanel, Step 3 Step - . Step 1 Step Network Network type Binding Remote Server Remote User list. Settings > Settings Network Bind Auto-Sync User Guide User Auto-Sync Binding a NexentaStor Appliance to a Virtual to Appliance IP Address Binding a NexentaStor nmc:/$ setup network ssh-bind ssh-bind setup network nmc:/$ Type the IP address or host name of a remote HA Cluster appliance. HA Cluster remote a name of or host IP address the Type password. appliance Super User the remote Type Repeat Click Type: Create an Auto-Sync service specifyi an Auto-Sync Create See See Bind the non-HA applianc non-HA the Bind In the In the In the Type the remote super user password. remote the Type select the Optionally, Click appliance. In the The new SSH-binding displays in the SSH-binding displays The new Hosts 2. 3. 4. To establish an SSH-binding for Auto-Sync service, using NMC: using service, Auto-Sync for SSH-binding an establish To 1. To establish the SSH-binding for Auto-Sync service, the SSH-binding Auto-Sync using for establish NMV: To 1. 3. 2. 3. 4. 2. 8. 7. 6. 5.   Sync Service in the HA Cluster Environment Service in the HA Cluster Sync If you create the SSH-binding to replicate data from or to an HA Cluster group, see an group, HA or to from Cluster data the SSH-binding to replicate If you create SSH-binding, aka SSH Public Key Ba SSH PublicSSH-binding,Key aka connec secure through communicate appliances Cluster is written appliance HA Cluster remote the from key Establishing the SSH-binding for an Auto-Sync Service the SSH-binding for Establishing You need to create the ssh-bidingthe for: create to need You • replication Local-to-remote • replication Auto-Sync Remote-to-local • ap HA Cluster between Replication Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com

. . About Auto-Sync About Auto-Sync file, which file, which in the HA Cluster Environment in the HA Cluster /etc/hosts/ to access the file.) access to . . Creating an Auto-Sync Service an Auto-Sync Creating VIP is section, read and complete the steps in the steps complete and is section, read and SSH-Bind . About Auto-Sync Replication in the HA Cluster Environment in the HA Cluster Replication Auto-Sync About field, select panel, clickpanel, setup appliance hosts appliance setup . You must first bind this NexentaStor appliance to each node of the HA Cluster as as the HA Cluster node of each to appliance this NexentaStor bind first must You in described contains the Internet host table. table. host the Internet contains Type If ssh-binding the manuallyIf can configure you fails, Network Network type Binding Settings > Settings Network Bind Click In the In the virtualSelect IP address. a Click To bind a NexentaStor appliance to a virtual IP address: virtual a to appliance NexentaStor a bind To 1. 3. 5. 2. 4. Note: Note:



Before you complete the steps describe in th in describe the steps complete you Before Environment in the HA Cluster Replication Binding a NexentaStor Appliance to a Virtual IP Address Appliance Binding a NexentaStor Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com 30 High Availability Cluster User Guide User High Cluster Availability

31 6

show volume show operations. There is a set a set is There operations. System Operations System volume if its run on the active node. runon the active if its volume th a context sensitive menu, accessed by right by right menu, accessed sensitive a th context lp you with daily cluster daily cluster with you lp . Shared Volume Shared command,the command that only displays Although both cluster nodes can access a shared volume using the volume shared a access can nodes cluster both Although Settings > Settings HA Cluster nmc:/$ help keyword cluster keyword help nmc:/$ Type: Click clicking over the status row of the volume you want to control. to want you of the volume row status the clicking over To check the status of the Cluster, using NMV: using Cluster, of the status the check To  To see a list of available commands, using NMC: using availablecommands, of list a see To  Note: Viewing Support Logs with a Operations Checking the Status of a Cluster Checking the Status Viewing Status HA Cluster the Failover a Triggering Manually Status Volume Shared Verifying About System Operations About System   • • • • • • • their views.now own tabbed tables have and heartbeat status The cluster • down menu wi drop the action replaced Nexenta • Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com You can check the status of the Cluster at any time. any at the Cluster of the status check can You You can use NMV to check the overall cluster status. cluster the overall check use NMV to can You HA Cluster has many changes in this version: in changes many has HA Cluster There are a variety of commands and GUI screens to he to commands and GUI screens of a variety are There NMC. supplement to commands cluster-specific of a Cluster Checking the Status of About System Operations This section includes the following topics: topics: the following section includes This

. > cluster_name . Heartbeat Status Heartbeat and Status panel, click Cluster Status Cluster Cluster Settings nmc:/$ show group rsf-cluster < rsf-cluster show group nmc:/$ Example: HA-Cluster rsf-cluster show group nmc:/$ response: System VALUE PROPERTY : HA-Cluster name NodeB] : [NodeA appliances : {"NodeA":"XXXXXXXXXX","NodeB":"YYYYYYYYYY"} machinesigs NodeB:NodeA: : NodeA:NodeB: hbipifs : 1 netmon HA-Cluster : Nexenta info : 1 generation : 1375745271.30001 refresh_timestamp : rsf-cluster type 2014 22:34:50 : Jan 8 creation SHARED VOLUME: ha-vol : ha-vol svc-ha-vol-shared-vol-name NodeB:e1000g0 NodeA:e1000g0 : ha-vol svc-ha-vol-ipdevs : svc-ha-vol-ipdevs-IPv6 : svc-ha-vol-attached-vols : NodeA svc-ha-vol-main-node : 20 svc-ha-vol-inittimeout : 8 svc-ha-vol-runtimeout : n svc-ha-vol-mhdc-disable : svc-ha-vol-monitor {"NodeA":{"monitor":"","ipdevs":{"e1000g0":""}},"NodeB":{"monitor":"", "ipdevs":{"e1000g0":""}}} NodeB:c1t1d0 : NodeA:c1t3d0 svc-ha-vol-resdisks nmc:/$ show group rsf-cluster nmc:/$ show group Type: In the Click the tabs toview Type: To view the HA Cluster configuration, using NMC: using configuration, Cluster HA the view To 1. To view the HA Cluster configuration, using NMV: using configuration, HA Cluster the view To 1. To check the status of the Cluster, using NMC: using Cluster, the of status the check To  2.

  

You can view the status of the HA Cluster and heartbeats at any time. any at and heartbeats the HA Cluster of status the view can You HA-Cluster STATUS: HA CLUSTER NodeA: Viewing the HA Cluster Status Viewing the Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com 32 High Availability Cluster User Guide User High Cluster Availability

33 High Availability Cluster User Guide User High Availability Cluster , you must replace the replace must , you failover stop on the current appliance, appliance, the current on stop DEGRADED Otherwise, failover may take long time or take may Otherwise, failover en needed. Performing a failover from the current the current from a failover Performing en needed. . appliance. Additionally, the volume exports to the to exports volume the Additionally, appliance. Status in healthy state by typing: by state in healthy causes the volume sharing service the volume to causes panel, click You must first set all cluster operations to manual mode. manual to operations cluster all set first must You faulted drive(s) before executing failover. failover. executing before drive(s) faulted freeze. may system your If any disk drive from the shared volume is in state state is in volume the shared from disk drive If any Cluster Settings nmc:/$ setup group rsf-cluster rsf-cluster setup group nmc:/$ nmc:/$ zpool status Example: pool: ONLINE state: requested none scan: config: CKSUM WRITE READ STATE NAME 0 0 0 ONLINE 0 0 0 ONLINE mirror-0 0 0 0 ONLINE c1t8d0 0 0 0 ONLINE c1t5d0 0 0 0 ONLINE mirror-1 0 0 0 ONLINE c1t9d0 0 0 0 ONLINE c1t6d0 data errors No known errors: In the Type: Verify that shared volume is volume shared that Verify To view the status of a shared volume, using NMV: using volume, shared a of status the view To 1. 2. To manually trigger a failover, using NMC: using a failover, trigger manually To 1. Note: Warning:   Verifying Shared Volume Status serviceusing NMV or NMC. volume on the shared status the Verify appliance to the specified appliance the specified to appliance Manually Triggering a Failover Manually Triggering wh systems between manually trigger failover a can You ha-vol running auto unblocked ha-vol e1000g0 20 8 20 e1000g0 ha-vol unblocked auto running ha-vol NodeB: 8 20 e1000g0 ha-vol unblocked auto stopped ha-vol and the opposite actions take place on the passive passive place on the take actions opposite and the node. other Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com

log ble for a shared volume through NMV. However, they they However, NMV. through volume a shared ble for

   

Operations with a Shared Volume not availa are operations volume the standard Some of NMC. in through available are Theinclude: operations attaching, detaching, growin Viewing Support Logs even HA Cluster about the information Gather a LU and replacing removing a LUN, online in NMC. volume a shared grow how to describes example The following HA CLUSTER STATUS: HA-Cluster STATUS: HA CLUSTER nodeA: manual stopped vol1-114 unblockednodeB: 10.3.60.134 unblocked auto running vol1-114 20 e1000g0 10.3.60.134 8 20 e1000g0 8 Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com 34 High Availability Cluster User Guide User High Cluster Availability

35 High Availability Cluster User Guide User High Availability Cluster grow if you want to add additional disks. to add want if you finish to adding new disks. y n . y Type Type • response: System Continue adding devices to the volume 'performance1'? (y/n) 'performance1'? to the volume devices adding Continue • (y/n) 'performance1'? Grow volume nmc:/$ setup volume volume nmc:/$ setup response: System selection) multiple for (Use SPACEBAR : devices Group of type : Group redundancy spare log cache pool Type Select disk(s). Select response: System type. Select the redundancy response: System 4. 3. 2. Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com High Availability Cluster User Guide

This page intentionally left blank

37 7

High Availability Cluster Upgrading Upgrading Upgrading HA Cluster HA Upgrading . (s) e their firmware to version C023 before upgrading upgrading before C023 version to e their firmware ed description of how to do the tasks follows this of description dohow follows theto ed tasks ed to use the sTec Device Manager (SDM) tool kit (SDM) tool Device Manager the sTec to use ed (s) hared Volume hared from Version 3.1.x to 4.0.3 3.1.x to Version from the S the e chapter to upgrade the HA Cluster. the HA Cluster. upgrade to chapter . hared Volume hared Status of Status sTec the S the . Shared Volume(s) Shared Status of Status High Availability Cluster Verifying the Verify that all shared volumes are in healthy state. state. in healthy are volumes shared all that Verify See Upgrade the NexentaStor node on which shared volumes are exported. are volumes on which shared node NexentaStor the Upgrade 1. 2. verview of the Upgrade Process verview of the Upgrade Upgrading Upgrading About SAS SSD Firmware Upgrad SSD Firmware About SAS O Verifying the Over Failing Sample Network Architecture Network Sample About Upgrading HA Cluster HA Upgrading About • • • • • • • Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com See Also: See section. Overview of the Upgrade Process detail A process. the upgrade describes below The list NexentaStor to 3.1.6 and later to version 4.x. You ne 4.x. You version to later 3.1.6 and to NexentaStor the upgrade. for sdmcmd-2.0.0.134-b002 version contact information, more For About SAS SSD Firmware Upgrade About SAS SSD Firmware upgrad must you SSDs, SAS ZeusRAM™ use STEC If you Nexenta recommends that you schedule the upgrade for a maintenance period. maintenance a for upgrade the schedule you that recommends Nexenta see: 4.0.1, to 3.1.x NexentaStor upgrade to how on instructions For 4.0.3 3.1.x to Version from About Upgrading HA Cluster About Upgrading HA in this described the procedure Use This section includes the following topics: topics: the following section includes This

shared-volume shared-volume , you failover the shared volume back to the newly newly the to back volume the shared failover , you if any, with healthy disk drives. disk with healthy any, if volume to the node you are not upgrading. After After upgrading. not are the node you to volume before you start the upgrade. If one or more shared shared more or If one the upgrade. start you before ONLINE DEGRADED, e , the upgrade will fail. , the upgrade manual DEGRADED nmc:/$ setup group rsf-cluster rsf-cluster nmc:/$ setup group nmc:/$ zpool status nmc:/$ zpool status Example: pool: ONLINE state: requested none scan: config: STATE NAME CKSUM READ WRITE ONLINE 0 0 0 ONLINE mirror-0 0 0 0 0 0 0 ONLINE c1t8d0 0 0 0 ONLINE c1t5d0 0 0 0 ONLINE mirror-1 0 0 0 ONLINE c1t9d0 0 0 0 ONLINE c1t6d0 data errors No known errors: Set the failover manual. mode Type: to the failover Set Replace disk drives with state state with disk drives Replace Type: To failover the active node to the passive node: passive the the active node to failover To 1. 2. To verify the state of a shared volume, using NMC: using volume, shared of a state the verify To 1. About SAS SSDSAS AboutUpgrad Firmware About Upgrading HA Cluster Upgrading About About Upgrading HA Cluster Upgrading About

 • NexentaStor Installation Guide Installation • NexentaStor •  • •

See Also: upgraded node. upgraded upgrading the node and before upgrading the other upgrading node the node and before upgrading To protect the shared volume, you failover the shared failover you volume, shared the protect To Failing Over the Shared Volume(s) Verifying the Status of the Shared Volume(s) Status of the Shared Verifying the are volumes all shared that verify must You volume(s) are in state state in are volume(s) Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com 38 High Availability Cluster User Guide User High Cluster Availability

High Availability Cluster User Guide User High Availability Cluster may require a reboot or stopping the HA Cluster. Cluster. HA the stopping or reboot a require may datastore availability while working on a node in while on a working availability datastore (s) at the HA Cluster functions correctly. functions the HA Cluster at upgrade during a maintenance window. window. maintenance a during upgrade e High Availability Cluster hared Volume hared cludes the following steps: steps: the following cludes the S Status of to confirm mode change. confirm to Y the nmc:/$ setup group rsf-cluster rsf-cluster nmc:/$ setup group failover Type node: passive the to over it fail to following type the node and the active in to Log 2. 3. Verifying • NexentaStor Installation Guide Installation • NexentaStor • Before starting the upgrade, verify th verify upgrade, the starting Before Verifying the Status of th To prevent user impact, execute the execute user impact, prevent To Overview of the Rolling Upgrade Procedure in procedure upgrade The rolling • environment of existing the health the Verify •node primary to volumes all Move •manual mode to operation cluster Set • 3.1.6 secondary node to Upgrade • secondary to node volumes all shared over Fail • primary 3.1.6 node to Upgrade • of cluster the health the Verify • 4.0.3 secondary node to Upgrade • primary to secondary from volumes Move • 4.0.3 secondary node to Upgrade • status cluster Verify •Automatic mode to failover Cluster High Availability the Set • normal operation to back volumes Move The rolling upgrade procedure allows you to maintain maintain to you allows procedure upgrade The rolling Upgrading High Availability Cluster from Version 3.1.x to Upgrading High Availability 4.0.3 that occur issue may unforeseen an However, the cluster. See Also: Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com

, proceed to to , proceed failover manual . scp /volumes/.config. Verifying the Status of the High Availability Cluster Availability the High of the Status Verifying

nmc:/$ setup group rsf-cluster rsf-cluster nmc:/$ setup group repository setup appliance nmc:/$ :!wq response: System configuration? 'APT' service Re-read nmc:/$ setup group rsf-cluster rsf-cluster group setup nmc:/$ nmc:/$ show volume nmc:/$ show auto-sync nmc:/$ show auto-snap nmc:/$ show auto-tier disable auto-sync setup nmc:/$ disable auto-snap setup nmc:/$ disable nmc:/$ setup auto-tier nmc:/$ show faults ssh-binding show network nmc:/$ save configuration appliance nmc:/$ setup The system backup files are stored in stored files are backup The system Type Set the HA Cluster mode to manual: mode to HA Cluster the Set node: on this imported are no volumes that Verify node: primary to nodes all Failover sources: the repository Update If any of the jobs are running, disable them: running,disable are the jobs of If any Verify that all faults have been cleared: have all that faults Verify communicate can of HA Cluster nodes the that Verify a copy of the configuration: cluster Save using location a safe files backup to Copy system Auto-Snap and jobs: Tier, Auto- Auto-Sync, of status the Verify 4. 2. 3. To upgrade the secondary node, using NMC: NMC: using node, secondary the upgrade To 1. 7. 6. 4. 5. 2. 3. To verify the status of the High Availability Cluster: Availability High the of status the verify To 1.

 

Upgrading the Secondary Node Afteryou complete steps describedin upgrading the HA Cluster nodes. nodes. Cluster HA the upgrading Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com 40 High Availability Cluster User Guide User High Cluster Availability

41 and High Availability Cluster User Guide User High Availability Cluster . manual

. Step 9 Step Yes de process, the final question will request willrequest the final question process, de isting configuration, verifies the state of the state verifies configuration, isting . Verifying the Status of the High Availability Cluster Availability the High of the Status Verifying ve checkpoint by typing: checkpoint ve , upgrade the primarynode node. Primary on which node is the HA Cluster upgrade , Upgrading Primary Node Primary Upgrading Secondary Node Upgrade to version 3.1.6: version to Upgrade nmc:/$ show share nmc:/$ show appliance version show appliance nmc:/$ rsf-cluster nmc:/$ show group rsf-cluster group setup nmc:/$ failover rsf-cluster nmc:/$ setup group nmc:/$ show volume nmc:/$ setup appliance upgrade -v upgrade setup appliance nmc:/$ upgrade nexentastor setup nmc:/$ up the ex backs NexentaStor upgrade the During and appliance, on the NexentaStor installed software party 3rd any whether verifies syspool, 4.0.3. version to upgrades during the upgra questions interactive the Follow reboot. to enable acti Whenprompted, nmc:/$ setup appliance nms property srvpool_cnt_intial 0 srvpool_cnt_intial property nms setup appliance nmc:/$ version show appliance nmc:/$ to number is 3.1.6, proceed version If the NexentaStor Verify the NexentaStor version: the Verify NexentaStor Verify that the status of the HA Cluster node is available. node is available. HA Cluster the of status the that Verify Manual: mode to the failover Set node: secondary to node primary from volume(s) the shared Failover this node: on imported are volumes shared that Verify folders: shared the NexentaStor to connect can clients that Verify to Proceed View theversion: NexentaStor 4.0.3: version to Upgrade Type y. configuration. the sources repository re-reads NexentaStor NMS: Multi Disable the 8. 10. 11. 15. 16. 17. 13. 14. 12. 9. 7. 5. 6. Upgrading Upgrading Upgrading Primary Node in described the steps complete you After shared volume(s) was initially imported. initially imported. was volume(s) shared Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com

on primary node. Secondary Node the Upgrading Upgrading in join the Microsoft Active Directory domain. Directory Active join the Microsoft Step 12 Step to on other node. Step 3 Step Step 3 Step -Upgrade Procedures to option expert_mode =1 option expert_mode !bash Step 1 Step apt-get update apt-get upgrade appliance setup apt-key update apt-key # # nmc:/$ nmc:/$ # nmc:/$ show appliance sysinfo show appliance nmc:/$ nmc:/$ show volume Failover the shared volumes to the to primary node. volumes the shared Failover version the Verify NexentaStor Repeat Log in to bash: bash: Log in to Type: plugins typing: by the required Upgrade Log in to the upgraded secondary node. the upgraded inLog to automatic. to node secondary on the mode failover the that Set NMC. using node, primary to in Log all that exported: volumes Verify are from steps Complete 6. 7. 4. 2. 3. To upgrade NexentaStor plugins, using NMC: using plugins, NexentaStor upgrade To 1. 5. 2. 3. 4. To upgrade theprimary node, usingNMC: To 1.

  Remounting a System Volume System a Remounting Upgrading NexentaStor Plugins Re-joiningMicrosoft WindowsActive Directory

Re-joining Microsoft Windows Active Directory Active Windows Microsoft Re-joining un node may HA Cluster During the upgrade, • Plugins NexentaStor Upgrading in this section described on both HA steps the plugins, complete otherNexentaStor upgrade to need If you nodes. Cluster Completing the Post including: procedures post-upgrade perform to need may you nodes HA Cluster both upgrade you After • • Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com 42 High Availability Cluster User Guide User High Cluster Availability

43 High Availability Cluster User Guide User High Availability Cluster . View every boot, add the following line: line: the following add boot, every . file for editing: file for pane, click Active Directory, using NMC: Active Directory, /etc/vfstab option expert_mode =1 expert_mode option !bash Settings > Appliance Settings mount -f zfs syspool /syspool syspool -f zfs mount mkdir /syspool vi /etc/vfstab Upgrade Checkpoints Upgrade # -syspool /syspool Example: - - yes zfs nmc:/$ nmc:/$ # # Type: the Open on is mounted syspool ensure To Log in to NMC. Log in to Switch to bash. Step 2 Step Use to separate entries. entries. separate to Use 3) 4) 5) In NMV, click In NMV, In the If you do not see any upgrade checkpoints: upgrade any not see do If you 1) 2) You should see the list of upgrade checkpoints. checkpoints. upgrade of should see the list You nmc:/$ setup network service cifs-server join_ads cifs-server service setup network nmc:/$ Log in to one of the HA Cluster nodes. HA Cluster the of one in to Log Type. node. other HA Cluster inLog to Repeat is mounted: volume system that Verify 1. 2. 3. To remount a system volume: remount a system To 1. 2. 3. 4. To re-join the Microsoft Windows Microsoft the re-join To 1. Note:   #device device mount FS fsck mount mount fsck FS mount device #device options at boot pass type point to fsck #to mount syspool - /syspool zfs - yes - Remounting a System Volume a System Remounting to need may You it. remount must you the upgrade, during unmounted becomes volume system If a nodes. Cluster on both HA procedure this perform Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com

on the other node, if needed. node, if on the other :!wq Save the changes: Save Step 1 Step 6) Repeat 2.

45 8

rver. The service cannot rver. up SMTP configuration and test so that that so test and configuration SMTP up the server, but it was stopped safely and you and you safely stopped but it was the server, ng servicethe on the se their state. If and when failover occurs (or any (or any occurs If and when failover state. their ared volume must be exported on both nodes. on both exported be must volume ared Testing and Troubleshooting and Testing clude the information about the shared volume. volume. shared about the information the clude NexentaStor sends an email to the administrator describing the event. sends an email the administrator to NexentaStor on other node. other on you can receive emails from the appliance. emailsfrom receive can you During the NexentaStor installation, you set you installation, During the NexentaStor Manually verify and troubleshoot the volume before marking the state as repaired. repaired. as marking the state before the volume and troubleshoot Manually verify of thedata to volume in anddo so cross-mounting to lead could result Failure corruption. Step 1 Step Execute the volume repair operation: repair the volume Execute nmc:/$ zpool status zpool nmc:/$ in should not The output repair shared-volume group rsf-cluster setup nmc:/$ process. repair the runs and initiates This Verify that volume is exported on both HA Cluster nodes by typing: by nodes on both HA Cluster that volumeisexported Verify Repeat run on any other server in the cluster untilis repaired. it in the cluster server other any run on 3. To repair a shared volume which is in broken state, using NMC: using state, whichbroken volume is in a shared repair To 1. A problem occurred while starting the serviceon while starting occurred A problem elsewhere. it run can stoppi or while starting occurred problem A fatal 2. Note: Replacing a Faulted Node a Faulted Replacing Repairing a Broken Cluster Service Cluster a Broken Repairing Warning:  • • Broken_Safe • Broken_Unsafe • Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com service state), changes a broken to the import forces command repair service. The cluster a for command the repair execute can You the sh Therefore, volume. a shared of operation NexentaStor tracks various appliance components, and and components, appliance various tracks NexentaStor Repairing a Broken Cluster Service Repairing a Broken Cluster states: broken two are There This section includes the following topics: topics: the following section includes This

46 out command command High Availability Cluster User Guide User High Availability Cluster . replace_node . Rolling Back to a Checkpoint Back to Rolling Plugins de in a HA Cluster, in case the state changes to to changes the state in case de in a HA Cluster, command if you reinstalled the software on a new on a the software reinstalled you if command ware failure. You can use this command to repair the repair to this command use can You failure. ware . , section . , section the old one is replaced by a new one. a new by one is replaced the old setup group rsf-cluster restore-node rsf-cluster setup group NexentaStor Installation Guide Installation NexentaStor NexentaStor Installation Guide Installation NexentaStor NexentaStor Installation Guide Installation NexentaStor NexentaStor Installation Guide Installation NexentaStor NexentaStor versions versions NexentaStor settings Network bindings SSH Heartbeats . There is no need to delete the cluster group on another node and reconfigure it and all it on another nodeand reconfigure group thedelete cluster to is no need There . nmc:/$ setup group rsf-cluster rsf-cluster group setup nmc:/$ Install the NexentaStor software on the faulted node. on the faulted software the NexentaStor Install See Set up the same configuration on a new hardware. Verify that the following components are are components following the that Verify hardware. on a new configuration same the up Set nodes: NexentaStor new on the old and identical • nodes. HA Cluster both for versions NexentaStor identical use must You • • • Type: Alternatively, rollback NexentaStor to a previously saved checkpoint. a previously saved to NexentaStor rollback Alternatively, See Apply the NexentaStor license key if required. if license key Apply the NexentaStor See Install the HA Cluster plugin. the HA Cluster Install See Restore the SSH-binding configuration. Restore To restore a faulted node: node: a faulted restore To 1. To replace a faulted node, using NMC: using node, a faulted replace To 1. 2. 2. 3. 4. 5.   that assist inc repairing an HA Cluster node after soft HA Cluster configuration if you had to reinstall or restore from a backup the NexentaStor and HA Cluster HA Cluster and the NexentaStor a backup from or restore reinstall to had if you configuration Cluster HA use this cannot You same hardware. the on software hardware. NexentaStor provides the the provides NexentaStor After executing the command, the system asks you to choose which node to exclude from the cluster and cluster the from exclude choose which node to to you asks the system the command, executing After match they if and node new the of parameters host checks NexentaStor instead. use to node new which group, the cluster of requirements the Restoring a faulted node NexentaStor provides a capability to restore a failed no a failed restore to a capability provides NexentaStor Replacing a Faulted Node a Faulted Replacing of service services. cluster of the Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com

47 High Availability Cluster User Guide User High Availability Cluster Binding the Nodes using SSH using Nodes the Binding nmc:/$ setup group rsf-cluster restore-node rsf-cluster nmc:/$ setup group On the healthy HA Cluster node, run: HA Cluster the healthy On page. Status Cluster HA on the Cluster HA the of status Verify See 7. 6. Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED ALL RIGHTS Systems, Nexenta 2014 © Copyright www.nexenta.com High Availability Cluster User Guide

This page intentionally left blank

Copyright © 2014 Nexenta Systems, ALL RIGHTS RESERVED 48 www.nexenta.com fibre channel failover mode failover F exclusive access E defining D configuring configuration data cluster C binding B additional resources adding A www.nexenta.com NexentaCopyright ©2014 Systems, RESERVED ALL RIGHTS failover configuring setting configuring iSCSI Targets to storage netmasks fibre channel cluster checking status nodes hostname triggering storage service NFS/CIFS iSCSI Targets HA cluster configuration data IP virtual address shared volume hostname address failover manually failover

23 3 2

25 2 5

, 17 18

9 33

31 4

25 15

17 support storage SSH shared volume shared disk setting S nodes failover NFS/CIFS network architecture netmasks N modifying misc options information mapping M implementing I hostname heartbeat mechanism heartbeat interfaces HA cluster H guide G contact failover nodes the binding tocluster adding failover mode binding defining heartbeat interfaces implementing misc options adding modifying configuring audience status sample

13 3

6 v 2 8

, 26 16

9 23 26

3 16

13 6

14 virtual IPaddress virtual V upgrading U triggering T system operations adding HAC failover

High Availability Cluster Availability High User Guide 37

Index 31

49 High Availability Cluster User Guide

This page intentionally left blank

Nexenta EMEA Headquarters Camerastraat 8 1322 BC Almere Netherlands

Nexenta Systems Italy Via Vespucci 8B 26900 Lodi Italy

Nexenta Systems China Room 806, Hanhai Culture Building, Chaoyang District, Beijing, China 100020

Nexenta Systems Korea Chusik Hoesa 3001, 30F World Trade Center 511 YoungDongDa-Ro GangNam-Gu, 135-729

3000-hac4.x-000012-B