Upgrading a multi Infrastructure EMC ControlCenter Environment from version 6.0 to 6.1 UB7 on a Windows 2003 Cluster Kevin F Atkin
EMC Proven Professional Knowledge Sharing 2010
Kevin F Atkin SAN Support Engineer Thomson Reuters [email protected] Contents
1 OVERVIEW ...... 4 1.1 HOW TO USE THIS ARTICLE ...... 4 2 INSTALLATION PRE-REQUISITES ...... 4 2.1 EMC CONTROLCENTER SERVER INFRASTRUCTURE BUILD ...... 4 2.2 HP DL380 SERVERS ...... 6 2.2.1 Install Windows 2003 Enterprise Edition ...... 6
2.3 HP DL360 SERVERS ...... 6 2.3.1 Install Windows 2003 Standard Edition ...... 6 2.3.2 Install HBA cards ...... 6 2.3.3 Install Powerpath version 5.3 ...... 7 2.3.4 Zone and Mask Servers into the SAN...... 7
2.4 USER ACCOUNTS ...... 8 2.4.1 Windows Cluster and User Accounts ...... 8 2.4.2 User Rights Assignment for the Clusteradmin Service Account ...... 9 2.4.3 ControlCenter Users...... 9
2.5 NETWORKING ...... 10 2.5.1 Private LAN Network Card Configuration ...... 10
2.6 SERVICE LAN NETWORK CARD CONFIGURATION ...... 15 2.7 ADVANCED NETWORK CONFIGURATION ...... 19 2.8 DNS NAME RESOLUTION ...... 20 2.9 INSTALL CONTROLCENTER IN A CLUSTERED ENVIRONMENT ...... 20 2.9.1 Create the Cluster ...... 20 2.9.2 Create the Cluster Resources ...... 24 2.9.3 Create Cluster Node Names ...... 25 2.9.4 Create the EMC CONTROLCENTERRG Resource Group ...... 25 2.9.5 Create the shared physical disk resource ...... 26 2.9.6 Create the IP Address Resource ...... 27 2.9.7 Create the NetworkName Resource ...... 29 2.9.8 Install Java on all Servers ...... 32 2.9.9 Install MDAC 2.8 on the third HP DL 380 Server ...... 32 2.9.10 Patch 4014 (Pre 6.1 StorageScope user fix) ...... 32
2.10 PREPARE HOSTS FOR CONTROLCENTER INSTALLATION ...... 32 2.10.1 Upgrade Preparations ...... 34 2.10.2 Install ControlCenter onto the First Cluster Node ...... 36 2.10.3 Upgrade Remaining Infrastructure Store Hosts ...... 43 2.10.4 Upgrade StorageScope Host ...... 44
2010 EMC Proven Professional Knowledge Sharing 2 2.10.5 Install the ControlCenter 6.1 Console ...... 48 2.10.6 ControlCenter 6.1 Post Upgrade Cluster checks...... 51 2.10.7 Edit EMC ControlCenter_server.ini File ...... 52 2.10.8 Edit the Master Agent file ctg.ini ...... 53 2.10.9 Edit the EMC ControlCenter_store.ini file ...... 53 2.10.10 ControlCenter Patching ...... 53 2.10.11 Patch 4667 (EMC CONTROLCENTER Repository) ...... 53 2.10.12 Patch 4668 (StorageScope Repository) ...... 53 2.10.13 Solutions Enabler 7.1.0.0 ...... 54 2.10.14 Install Solutions Enabler 7.1 MRLK (CC4762) on the Active node of the cluster...... 54 2.10.15 After EMC CONTROLCENTER 6.0 has Upgraded to EMC CONTROLCENTER 6.1 ...... 54 2.10.16 Update Bundle 7 (Patch 4673) ...... 57 2.10.17 Patch 4744 ...... 58 2.10.18 Patch 4754 ...... 58 2.10.19 Add local ORA_DBA Group to each Cluster Node ...... 58 2.10.20 Post UB7 Cluster Preparation ...... 59 2.10.21 Confirm Master Agents are at the same patch level ...... 60 2.10.22 Agent Patching ...... 60
2.11 INSTALL CONTROLCENTER LICENCES ...... 61 2.12 CREATE CONTROLCENTER RESOURCES FOR THE CLUSTER SERVICE ...... 62 2.12.1 Load StorageScope ...... 68 2.12.2 Install the WLA Archive Agent ...... 69 2.12.3 Install the NAS Agent ...... 70 2.12.4 Add Gatekeepers to Symmetrix® Agent Host ...... 71
Disclaimer: The views, processes, or methodologies published in this compilation are those of the authors. They do not necessarily reflect EMC Corporation’s views, processes, or methodologies.
2010 EMC Proven Professional Knowledge Sharing 3
1 Overview
1.1 How to use this Article
I wrote this article based on my experience building an EMC ControlCenter® Cluster in the Thomson Reuters Data Center environments. Use this article in conjunction with the EMC Documentation available on PowerLink; I refer directly to EMC documentation in part, where I describe the installation process.
This article s aimed at the Upgrade of EMC CONTROLCENTER 6.0 to 6.1 UB7 but includes the steps required to build the initial cluster etc. For existing EMC CONTROLCENTER 6.0 clustered installations, start from Section 2.9 to prepare the upgrade.
2 Installation Pre-Requisites
2.1 EMC CONTROLCENTER Server Infrastructure Build
This article is based on a six server (two clustered) ControlCenter Infrastructure, but also applies to any number of hosts depending upon requirements. In this example, I have chosen the following server hardware. This procedure works equally well on HP Blade Servers too.
• The EMC ControlCenter Server hardware will comprise 3 x HPDL380 Servers and 3 x HPDL360 Servers.
• All servers that participate in EMC ControlCenter require the Windows 2003 Server. The HPDL380 Servers will require Windows 2003 Enterprise Edition for the purposes of clustering and to utilise the 8GB of Memory that they will be installed with. The HPDL360 Servers will be built using Windows 2003 Standard Edition.
2010 EMC Proven Professional Knowledge Sharing 4 • The ControlCenter Infrastructure will be installed as a 2 Node Cluster on HPDL380 Server hardware in a distributed configuration. The remaining HPDL380 Server will be used for Storage Scope and WLA (Work Load Analyser) reporting. The HPDL360 Servers will communicate with the EMC CONTROLCENTER Cluster to provide extra Console and Agent connectivity.
• A reliable Windows Domain infrastructure and a Cluster Service Account that exists in the domain.
• Three network connections in each clustered server (one for the Service LAN and two for Cluster Private LAN communication) with enough IP addresses to support all required resources: 1 x Server IP Address per node 2 x Cluster Private IP Addresses per Node 1 x Cluster IP Address 1 x Resource Group (Virtual Server) IP Address per Resource Group
• Shared SAN storage that the clustered servers have access to and that are configured to EMC best practices. This will comprise a 1x 68GB LUN and a 500MB Lun for the Quorum disk. • Shared and Non shared 68GB LUNs will be assigned as follows: A) 1 x 68GB LUN assigned to the Cluster servers for Repository and the ControlCenter Server. This is will be assigned the drive letter R b) 1 x 68GB LUN assigned to the Storage Scope Repository on the third HPDL380 Server. This will be assigned the drive letter S C) 1x 68GB LUN assigned to the fourth server (HP DL360) for Work Load Analyzer file storage. This will be assigned the drive letter T Name the drive mappings as in Table 1
Q: SharedQ Quorum
R: SharedR EMC ControlCenter Repository and Server
S: StorageScope Storage Scope™ Repository … …
T: WLArchive WLA Archives
Table 1
2010 EMC Proven Professional Knowledge Sharing 5 2.2 HP DL380 Servers
2.2.1 Install Windows 2003 Enterprise Edition
Load Windows 2003 Enterprise Edition onto the HP DL380 Servers and name each server as follows:
Site id-emccc01a Site id-emccc01b Site id-emccc02
Complete Table 2 for IP address details
Hostname IP Address Subnet Mask Gateway Site-EMCCC01a SITE-EMCCC01b SITE-EMCCC02 SITE-EMCCC03 SITE-EMCCC04 SITE-EMCCC05 SITE-EMCCC01-VIP1 SITE-EMCCC01-VIP2
Table 2
Join all 3 servers to a Windows 2003 Domain. In this guide the Domain will be known as EMC ControlCenter
2.3 HP DL360 Servers
2.3.1 Install Windows 2003 Standard Edition
Load Windows 2003 Standard Edition onto the HP DL360 Servers and name each server as follows:
Site id-emccc03 Site id-emccc04 Site id-emccc05
Join all 3 servers to a Windows 2003 Domain.
2.3.2 Install HBA cards
The EMC ControlCenter servers will each have dual port Emulex LP11002-E cards installed.
2010 EMC Proven Professional Knowledge Sharing 6
2.3.3 Install Powerpath version 5.3 Run the installation of Powerpath and accept the default settings.
For the clustered servers only, you will need to set the Mode on the first HBA to Active and the second HBA to Standby. Simply right click the HBA and change the setting. Note this setting may no longer need to be applied as there was an issue with Power Path that made this change necessary.
2.3.4 Zone and Mask Servers into the SAN
Zone each EMC ControlCenter Server to both SAN Fabrics. Select an appropriate VSAN in the case of a Cisco MDS switch Fabric. The Cluster nodes will be masked to 1 x 68GB Lun and the 0.5GB Quorum disk.
The third HPDL380 Server will be masked to 2 x 68GB LUNs. The remaining HPDL360 Servers will not be using SAN attached storage.
2010 EMC Proven Professional Knowledge Sharing 7 2.4 User Accounts
2.4.1 Windows Cluster and User Accounts Create a Domain Cluster Service account called clusteradmin in the Windows Domain that the EMC ControlCenter Servers will be member servers of. The clusteradmin Domain account must be a member of the local Administrators group on the cluster.
Create a Global Group in the Domain called EMC ControlCenter Administrators. Create user accounts in the Domain for all EMC ControlCenter users and add them to the Domain Global Group EMC ControlCener Administrators. Next, add the EMC ControlCenter Administrators Group to the Local Administrators Group on each EMC ControlCenter Server. Use the clusteradmin Domain account when creating the cluster, as discussed in section 2.7.
2010 EMC Proven Professional Knowledge Sharing 8
2.4.2 User Rights Assignment for the Clusteradmin Service Account
Open the Administrative tools folder and Navigate to:
Security Settings -> Local Policies -> User Rights
The Cluster Service account must be given the following User Rights:
• Act as part of the operating system • Back up files and directories • Restore files and directories • Adjust memory quotas for a process • Log on as a service • Increase scheduling priorities Consult the Microsoft Technet Website for any further settings that may need to be applied under Security Settings -> Local Policies -> Security Options
2.4.3 ControlCenter Users Initially, a Local user account, EMC ControlCenteradmin, must be created on each EMC ControlCenter Server host. This EMC ControlCenteradmin account is used to install ControlCenter .This user will have complete access rights to all ControlCenter objects including ControlCenter administrative rights, which allows the user to add new ControlCenter users and to set the permissions that govern a user’s access rights within ControlCenter.
The user accounts to administer ControlCenter will be Domain accounts that are members of the Global Group ControlCenter Administrators. Once ControlCenter has been installed, the accounts are added to the EMC ControlCenter Users folder in the format:
Domain\User
2010 EMC Proven Professional Knowledge Sharing 9 2.5 Networking
Server clustering requires a very specific network configuration in order to function. They can be grouped into two specific connections to simplify the design:
• The Service LAN
• Cluster Private LAN
This is depicted in the following generic diagram. Please note that only Node 1 and Node 2 are clustered in ControlCenter.
2.5.1 Private LAN Network Card Configuration
The Cluster Private LAN and Cluster Service LAN NICs have quite specific network settings applied.
2010 EMC Proven Professional Knowledge Sharing 10 Verify the Bindings are as follows on the Private LAN:
Select only “Internet Protocol (TCP/IP)”.
Select “Internet Protocol (TCP/IP)” and click the Properties button.
2010 EMC Proven Professional Knowledge Sharing 11
Enter the required IP address of the Private LAN
Ensure there is no Default Gateway specified.
Ensure there are no DNS Servers specified.
Click the Advanced… button, and choose the DNS tab.
2010 EMC Proven Professional Knowledge Sharing 12
Verify that the “DNS suffix for this connection:” entry is blank. Ensure that the “Register this connection’s addresses in DNS” is NOT selected. Select the WINS tab.
2010 EMC Proven Professional Knowledge Sharing 13
Ensure there are NO WINS servers listed. Select “Disable NetBIOS over TCP/IP” Click OK, Then click OK again. Repeat this process for both NICs that are to be used for Cluster Private LAN connections.
2010 EMC Proven Professional Knowledge Sharing 14
2.6 Service LAN Network Card Configuration
The network connection to be used for Service LAN Communications needs to be renamed to Service LAN.
:
Open the Properties for each of the NICs:
Verify the Bindings are as follows:
2010 EMC Proven Professional Knowledge Sharing 15 . Select “Internet Protocol (TCP/IP)” and click the Properties button. Enter the IP Addressing as shown in the example as appropriate.
Click the Advanced… button, and choose the DNS tab
2010 EMC Proven Professional Knowledge Sharing 16
Verify that the “DNS suffix for this connection:” entry is correct. Ensure that the “Register this connection’s addresses in DNS” is selected. Next, select the WINS tab.
2010 EMC Proven Professional Knowledge Sharing 17
Add WINS servers (if used).
Ensure that “Enable NetBIOS over TCP/IP” is selected. Click OK, And click OK again.
Repeat this process for all NICs that are to be used for Service LAN connections.
2010 EMC Proven Professional Knowledge Sharing 18
2.7 Advanced Network Configuration
In the Network Connections window, select the menu option Advanced and choose Advanced Setting.
Ensure the network connections are listed in the Connections: list in the following order:
2010 EMC Proven Professional Knowledge Sharing 19
Rename the network adapters on the two HP DL380 servers that are taking part in clustering as shown; disable the unused adapters.
2.8 DNS Name Resolution
Name resolution will be provided by DNS. Each ControlCenter (EMC ControlCenter) Host will register into DNS when joining the Windows Domain. Verify that both forward and reverse DNS lookup is properly configured and active
2.9 Install ControlCenter in a Clustered Environment
2.9.1 Create the Cluster
The shared SAN storage should only be masked to the first Node of the cluster before the cluster is created. Once the cluster is online, the remaining node can be masked into the shared storage, and the new Node added to the cluster using the clusteradmin utility.
Create the cluster using Microsoft’s Cluster Administrator; select the Domain that is relevant to where ControlCenter is located. Give the Cluster a generic name in the Cluster name field. This name identifies the cluster. An example follows.
2010 EMC Proven Professional Knowledge Sharing 20
Click Next
2010 EMC Proven Professional Knowledge Sharing 21 When prompted for an IP address, enter the Virtual IP address of the Cluster. This Virtual IP is the first of 2 Virtual IP addresses used in this installation.
Click Next,
Enter the Domain cluster account, in this example, clusteradmin
This account requires special permissions as described previously.
2010 EMC Proven Professional Knowledge Sharing 22
Click Next, and the Wizard will check the validity of the cluster
This completes the cluster build.
2010 EMC Proven Professional Knowledge Sharing 23 Certain prerequisites need to be in place before ControlCenter can be installed into a Microsoft Cluster. These are outlined in the following sections.
2.9.2 Create the Cluster Resources
The Cluster comprises a Cluster Group and a Cluster Resource Group called EMC CONTROLCENTERRG. Each Group must have an IP Address (Virtual), a Name, and a shared Disk resource. Note that the Cluster Resource Group shown in the graphic is labelled as ECCRG
The screen shot below shows the Cluster Group that formed when the cluster is created. The ControlCenter components will be added to a new group called EMC CONTROLCENTERRG. This is created by right clicking on the Groups folder and selecting New Group.
2010 EMC Proven Professional Knowledge Sharing 24
2.9.3 Create Cluster Node Names The nodes should be named in the format site-emccc01a, Site-emccc01b
For Example:
Cluster Node #1: dtcs-emccc01a
Cluster Node #2: dtcs-emccc01b
2.9.4 Create the EMC CONTROLCENTERRG Resource Group Using Cluster Administrator create a new Resource Group called EMC CONTROLCENTERRG. To do this, right click on the Cluster and select New Group. The EMC CONTROLCENTERRG Group will be where the ControlCenter Repository and Server will come under cluster control.
The screen shot below shows the EMC CONTROLCENTERRG and the associated IPAddress, NetworkName, and Disk resources. Note: Generic Services are added AFTER ControlCenter has been upgraded. There should be no Generic services created for ControlCenter until after the full upgrade completes.
2010 EMC Proven Professional Knowledge Sharing 25
2.9.5 Create the shared physical disk resource
Click and highlight the Disk Group menu tree icon to show the Disk resource group.
Drag the desired physical disk from the Disk Group to the EMC CONTROLCENTERRG Resource group. When asked to confirm that the resource should be moved and the group should be changed for the resource, reply Yes. MSCS creates a resource for the shared physical disk (Disk R)
2010 EMC Proven Professional Knowledge Sharing 26
2.9.6 Create the IP Address Resource The screen shot shows the General tab of the IP Address resource
The next screen shot shows the Dependencies tab of the IP Address resource where the IP Address shows a dependancy on the shared Disk R.
2010 EMC Proven Professional Knowledge Sharing 27
The next screen shot shows the Paramaters tab (where the IP Address is input). This address is a Virtual IP. This is the address that ControlCenter runs on. When a Move Group is executed, the IP address will transfer to the other node. It is important that DNS is working correctly for this to happen.
2010 EMC Proven Professional Knowledge Sharing 28
2.9.7 Create the NetworkName Resource The screen shot shows the General tab of the NetworkName Resource
On the Dependencies screen, since NetworkName is dependent on IP Address, select Modify. Then move the IP Address resource from the Available Resources column to the Resource Dependencies column.
Then, click next.
This automatically establishes dependencies on all previously configured generic resources.
The Dependencies tab is shown follows.
2010 EMC Proven Professional Knowledge Sharing 29
The Parameters tab shows the name that ControlCenter uses. This name is input into ControlCenter Console when connecting.
2010 EMC Proven Professional Knowledge Sharing 30 Right click on each Cluster Private Lan under the Networks folder in Cluster Administrator. Select the Internal cluster communications only (private network) radio button.
This is shown below:
Select the “All communications” (mixed network) radio button for the Service LAN. Bring the cluster resource group(s) online.
2010 EMC Proven Professional Knowledge Sharing 31 In the MSCS tree panel, right click the cluster resource groups to be brought online (in our example, EMC CONTROLCENTERRG is the only resource group).
And click Bring Online.
Move each Physical Disk resource on to all nodes in the cluster (via the Move Group command in the Cluster Administrator). This is done so that you may verify that the drive letters are assigned correctly on all nodes. Use the Disk volumes labeled in Table 1 to assist in this verification.
2.9.8 Install Java on all Servers Install Java jre-6u2-windows-i586-p.exe on each server. Run the executable and accept the defaults.
2.9.9 Install MDAC 2.8 on the third HP DL 380 Server Download the latest version of MDAC from Microsoft and install on the third HPDL 380 Server which will be site-emccc02.
2.9.10 Patch 4014 (Pre 6.1 StorageScope user fix) Follow the readme for this patch. It may not be required.
2.10 Prepare hosts for ControlCenter installation
Ensure the following for hosts where infrastructure components (EMC CONTROLCENTER Server, Repository, and Store) are to reside:
• The fully-qualified domain names (FQDNs) and hostnames do Not contain underscores. • No applications are currently running (to prevent interference With the installation process, close any Windows Explorer windows and active command prompts). • All Oracle software is removed • Terminal Services is not running • Disable Anti Virus Software
2010 EMC Proven Professional Knowledge Sharing 32 • Stop and Disable DTC Service (Distributed Transaction Coordinator). This should not be re-enabled after EMC CONTROLCENTER is installed. • Stop all HP Services and set to manual • PowerPath™ version 5.3 installs EMC PowerPath RSA Encryption Service 5.3.0. This should be disabled. • Verify ControlCenter Server Secure Listening Port 10799 is OPEN to ensure proper communication between agents and the infrastructure.
Edit the path statement in the Environment Variables section of the System Applet in Control Panel and remove the following path variables:
Variable name: R_SHLIB_LD_LIBRARY_PATH
Variable Value: D:\Program Files\EMC\RSA\Rkm_Client\lib
• If Cisco Device Manager is installed, Stop and set to Manual PostgreSQL service. • The ControlCenter Repository upgrade executable requires the use of two DLL’s ….libeay32.dll and ssleay32.dll. Other applications may also use these DLL’s which can conflict with ControlCenter upgrade.
Note the existing settings for the system %PATH% variable, and then make sure the following path exists at the beginning in the Environment variables section BEFORE the upgrade commences:
%RAMBOH%\admin\Ramb_scripts;R:\EMC CONTROLCENTER\Repository\bin
Once the upgrade has completed, revert back to the original %PATH%
2010 EMC Proven Professional Knowledge Sharing 33
2.10.1 Upgrade Preparations Quoting from the EMC documentation, “Ensure that all ControlCenter Repository maintenance tasks are stopped prior to starting the upgrade." Run the ramb_jobstatus.bat job located in the R:\EMC CONTROLCENTER\Repository\admin\Ramb_scripts folder to determine which maintenance jobs has already been run. Open the ramb_jobstatus.log file located in the same folder as the batch file, to check for any maintenance jobs that have not been run yet and run any additional maintenance jobs you feel are necessary”
• Copy the regutil.exe and regutil.pl from the Regutil\Regutil600 directory, found of where the EMC CONTROLCENTER6.1 disks have been copied as discussed in Sect 2.8.2 of this document.
Copy these files to: R:\EMC CONTROLCENTER\Tools\Utils folder. Do this on ALL Infrastructure hosts.
• Disable all WLA DCP’s as per EMC Upgrade guide discussed earlier.
• Uninstall all Control Center 6.0 Consoles using Add / Remove programs in Windows Control Panel.
• Backup StorageScope Repository. Run an ETL from within StorageScope and run emcsts_export_db.bat
• Backup ControlCenter Repository. Run clustersave.bat from R:\EMC CONTROLCENTER\tools\Utils folder. Next, copy the EMC CONTROLCENTER folder to R:\Rollback\EMC CONTROLCENTER
• If this is an upgrade to an existing cluster, remove ALL EMC CONTROLCENTERRG Generic cluster resources using clusteradmin. The only resources that should be under cluster control prior to the upgrade are: Disk R , EMC CONTROLCENTERRG IPaddress and EMC CONTROLCENTERRG Networkname
2010 EMC Proven Professional Knowledge Sharing 34
From the cluster A node server and the StorageScope server Services window, Stop and set to Manual the following Services:
• Disable Terminal Services • Distributed Transaction Coordinator service • Anti-Virus Software service(s) • EMC ControlCenter API Server service • EMC ControlCenter Master Agent service • EMC ControlCenter StorageScope Server service • EMC ControlCenter Web Server service • EMC ControlCenter WebConsole Server service • EMC ControlCenter Store service • EMC ControlCenter Server service
Note: ControlCenter services will already be set to manual if the clustersave.bat file has been run
From the ControlCenter Repository host (Node A) Services window, Stop and Restart:
• OracleEMC CONTROLCENTERREP_HOMETNSListener • OracleServiceRAMBDB
Note: Stop both the OracleEMC CONTROLCENTERREP_HOMETNSListener service and the OracleServiceRAMBDB service. After both services are listed as stopped, re-start the OracleEMC CONTROLCENTERREP_HOMETNSListener service and the OracleServiceRAMBDB service in that order.
Verify that all the ControlCenter processes listed above have stopped From Task Manager.
2010 EMC Proven Professional Knowledge Sharing 35 From the StorageScope Repository host Services window, Stop and Restart:
• OracleEMC CONTROLCENTERREP_HOMETNSListener • OracleServiceEMCSTSDB
Note: Stop both the OracleEMC CONTROLCENTERREP_HOMETNSListener service and the OracleServiceEMCSTSDB service. After both services are listed as stopped, re-start the OracleEMC CONTROLCENTERREP_HOMETNSListener service and the OracleServiceEMCSTSDB service in that order.
2.10.2 Install ControlCenter onto the First Cluster Node
Download the latest A04 EMC CONTROLCENTER 6.1 ISO images from the EMC Powerlink website. The most convenient way to install or upgrade EMC CONTROLCENTER is to first copy all the ISO images and Patches (currently UB7 shown here as CC_4673) to a shared network drive as shown below:
2010 EMC Proven Professional Knowledge Sharing 36 Once all the pre-requisites discussed in this document and in EMC documentation have been fulfilled, the upgrade is ready to proceed. Make sure that the current EMC CONTROLCENTER folder is backed up as per EMC documentation!
Using the clusteradmin utility, PAUSE the second node of the cluster by right-clicking on Node B and selecting Pause Node.
On the first node of the cluster (Node A), run setup.exe from the shared location. The Primary Network Name appears automatically if DNS is correctly set up.
Select Next,
2010 EMC Proven Professional Knowledge Sharing 37
If prompted for a Secondary Network, Select None.
Accept the installation path of R:\EMC CONTROLCENTER (in this example), and Click Next.
Specify the existing ControlCenter Repository, and Click Next
2010 EMC Proven Professional Knowledge Sharing 38
Select Install
The following screenshots will be observed as the upgrade progresses:
2010 EMC Proven Professional Knowledge Sharing 39
2010 EMC Proven Professional Knowledge Sharing 40
2010 EMC Proven Professional Knowledge Sharing 41
The Installation of the Repository can take up to an hour or more! This will be dependant upon the size of the database being upgraded.
2010 EMC Proven Professional Knowledge Sharing 42
Select, “Yes I want to Restart my Computer now” then select Finish. This completes the initial (pre patching) part of the Repository upgrade.
When the Server has rebooted, check that the Cluster Service is running (the B Node should still be Paused) so that the cluster R drive is made available.
Check that ALL EMC ControlCenter Services can be started. They will be set to Manual at this point.
2.10.3 Upgrade Remaining Infrastructure Store Hosts
Run setup.exe against all remaining EMC CONTROLCENTER hosts that have a 6.0 Store installed, so that the Store upgrades to EMC CONTROLCENTER 6.1
2010 EMC Proven Professional Knowledge Sharing 43
2.10.4 Upgrade StorageScope Host Make sure that all ControlCenter services are running before going onto the StorageScope server (site-emccc02) and running setup.exe from the shared drive where EMC CONTROLCENTER6.1 disks were copied to.
Follow the screen prompts
2010 EMC Proven Professional Knowledge Sharing 44
The above Window will pop up and requires an answer. Select No if you have already run the emcsts_export_db.bat script found in: S:\EMC CONTROLCENTER\Repository\admin\emcsts_scripts folder
Once the upgrade completes, perform the following validation tasks:
2010 EMC Proven Professional Knowledge Sharing 45 Validate access to the ControlCenter Repository and StorageScope Repository databases
• From the Infrastructure Server (Node A), run EMC CONTROLCENTER_GETVERSION and ensure the correct version displays as follows:
R:\EMC CONTROLCENTER\Repository\admin\Ramb_scripts>EMC ControlCenter_getversion.bat EMC CONTROLCENTER_INSTALL_ROOT environment variable found is ==> R:\EMC CONTROLCENTER return code = 0
EMC CONTROLCENTER Repository Version ------
EMC CONTROLCENTER Repository version information
External Version : Version 6.1.0 Internal Version : 6.1.0.4.71 KitSpec : GA6 Build Label : EMC CONTROLCENTER61040.16MAY2009.1146.224 Build Name : EMC CONTROLCENTER61040.16MAY2009.1146.224 Release Date : Sat May 16 14:34:00 2009 Install Date : Thu Sep 24 10:59:16 2009
• Run the following on the StorageScope server and check the result is displayed as below:
S:\EMC CONTROLCENTER\Repository\admin\emcsts_scripts>emcsts_getversio n.bat Value of userName got as input: sys
2010 EMC Proven Professional Knowledge Sharing 46 Initializing Lockbox in:S:\EMC CONTROLCENTER\lockbox\WEB.lb Value of UserName passed to SQL: sys Success! EMCSTS is open Get Schema Version from Single-Site StorageScope Repository Database . . . Value of userName got as input: emcsts Initializing Lockbox in:S:\EMC CONTROLCENTER\lockbox\WEB.lb Value of UserName passed to SQL: emcsts Success!
SQL*Plus: Release 10.2.0.4.0 - Production on Tue Oct 20 16:07:46 2009
Copyright (c) 1982, 2007, Oracle. All Rights Reserved.
Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Production With the Partitioning, OLAP, Data Mining and Real Application Testing options
GETVERSIONINFO ------
EMC CONTROLCENTER Repository version information
External Version : Version 6.1.0 Internal Version : 6.1.0.4.71 KitSpec : GA6 Build Label : EMC CONTROLCENTER61040.16MAY2009.1146.224 Build Name : EMC CONTROLCENTER61040.16MAY2009.1146.224 Release Date : Sat May 16 14:34:00 2009
2010 EMC Proven Professional Knowledge Sharing 47
Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 – Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options S:\EMC CONTROLCENTER\Repository\admin\emcsts_scripts>
DO NOT Apply the Update Bundle if the above commands do not return the correct version. Contact EMC for support.
Open the Services applet on the A Node and verify that the Oracle database services have started.
• Make sure the following services are running (if not then Start them) • EMC ControlCenter Key Management Server • EMC ControlCenter Repository • EMC ControlCenter Server • EMC ControlCenter Store • EMC ControlCenter Web Server • EMC ControlCenter API Server • EMC StorageScope Server
2.10.5 Install the ControlCenter 6.1 Console
Once ControlCenter has upgraded to version 6.1 then install a Console so that the Master Agent can be upgraded as detailed in section 2.8 of this document.
Open a browser and connect to the cluster node name. For example: https://dtcs-emccc01:30002/webinstall
Note that it may be necessary to use the IP address of the cluster if the cluster name cannot be resolved at this point. This solution to this comes after the cluster resources have been re instated later in this document.
2010 EMC Proven Professional Knowledge Sharing 48 Install the Console 6.1.0 from the on screen menu as shown below:
When the Console has installed, enter the Cluster name in the Server Host Name as shown here.
It may be necessary to use the IP address of the cluster if an error appears stating that EMC CONTROLCENTER server is offline at this point. This is addressed later in this document.
2010 EMC Proven Professional Knowledge Sharing 49
Log into the ControlCenter Console.
Use a Domain account that has previously been added to the EMC CONTROLCENTER Users in ControlCenter. If this was a new install then the only account available would be the local EMC ControlCenteradmin account “eccadmin”
Note that the Console view shown here is from a UB5 instance of ControlCenter
Once logged in:
Select the Agents View, and then right click on the main Repository server (Node A) in the Agents view window.
Select Agents in the drop down list followed by Upgrade.
Upgrade the Master Agent to ControlCenter 6.1 level before beginning the next procedure.
2010 EMC Proven Professional Knowledge Sharing 50 2.10.6 ControlCenter 6.1 Post Upgrade Cluster checks Before installing the Update Bundle, prepare the cluster nodes to enable successful failover of the ControlCenter resources.
Execute the following procedure on the A node where the original ControlCenter installation was done.
The Repository is automatically started and online after installing ControlCenter. The Repository must be shut down prior to performing the Move Group command in the next step. Shut down any Repository services that might be running:
In the Services window, select the following services:
OracleServiceEMCSTSDB OracleEMC CONTROLCENTERREP_HOMETNSListener OracleServiceRAMBDB
And select Stop
1. Change directory to R:\EMC CONTROLCENTER\tools\utils 2. Run clustersave.bat.5 Upgrading ControlCenter 5.2 to 6.1 3. Using the clusteradmin utility, initiate a manual failover of the EMC CONTROLCENTERRG resource group to the B node. Right-click the resource group EMC CONTROLCENTERRG and choose Move Group.
From the B node, do the following:
1. Change directory to R:\EMC CONTROLCENTER\tools\utils 2. Run clusterupgrade.bat. 3. Reboot the B node for the upgrade to take effect.
The EMC CONTROLCENTERRG ControlCenter resources need to be recreated in the case of an upgrade, or created if installing for the first time, but only after applying the Update Bundles. This will be discussed later in this document.
The following do not need to be recreated:
2010 EMC Proven Professional Knowledge Sharing 51 EMC CONTROLCENTERRGPhysicalDisk EMC CONTROLCENTERRGIPAddress EMC CONTROLCENTERRGNetworkName
Use the Move Group command again to bring the Cluster Resources under the A node again.
2.10.7 Edit EMC ControlCenter_server.ini File
To enable correct name resolution when logging into the EMC CONTROLCENTER Console after upgrading the cluster configuration, re-enter the IP aliases in the EMC ControlCenter_server.ini file.
Locate the EMC ControlCenter_server.ini file on the A node in
Where in this instance the
Make a copy of the EMC ControlCenter_server.ini file to a backup location.
Then open the EMC ControlCenter_server.ini file and search for the line that contains the string name server aliases and then add the physical IP addresses (not logical) of Node A and Node B respectively as shown in the example below: name server aliases = 10.20.2.4,10.20.2.74
2010 EMC Proven Professional Knowledge Sharing 52
2.10.8 Edit the Master Agent file ctg.ini
Locate the ctg.ini file in the R:\EMC CONTROLCENTER\exec
Add the following entry:
[Default] cluster = true
2.10.9 Edit the EMC ControlCenter_store.ini file Locate the file where each Store is installed.
For example in S:\EMC CONTROLCENTER\EMC ControlCenter_inf\data\EMC ControlCenter_inf_hostname\data
Add the following entry: use fixed port = true
2.10.10 ControlCenter Patching The following sections outline the necessary patching that is required to complete the upgrade, starting with the Oracle OCPU patches.
2.10.11 Patch 4667 (EMC CONTROLCENTER Repository)
This is an Oracle OCPU patch. Follow the readme document that is supplied with this patch. It can take quite some time to load!
2.10.12 Patch 4668 (StorageScope Repository)
This is an Oracle OCPU patch. Follow the readme document that is supplied with this patch. It can take quite some time to load!
2010 EMC Proven Professional Knowledge Sharing 53 2.10.13 Solutions Enabler 7.1.0.0 Update Bundle 7 (UB7) requires SE 7.1.0.0 is installed into the media repository, the next section describes how this is achieved.
2.10.14 Install Solutions Enabler 7.1 MRLK (CC4762) on the Active node of the cluster. This enables Solutions Enabler to be deployed to other hosts
Read the readme that comes with Patch CC 4762 and then run the EMC CONTROLCENTER_SE71.exe file. Make sure the patch installs successfully.
Select Next
The MRLK install installs Solutions Enabler 7.1.0.0 into the media repository, so that it can be deployed to other hosts during Agent installation.
2.10.15 After EMC CONTROLCENTER 6.0 has Upgraded to EMC CONTROLCENTER 6.1 Download Patch 4563 from EMC and apply to the following Infrastructure hosts:
2010 EMC Proven Professional Knowledge Sharing 54
EMC ControlCenter Server EMC ControlCenter Repository EMC ControlCenter Store(s) EMC ControlCenter StorageScope
.
Before running this patch, check for the existence of the registry string MA_ROOT on each of the above hosts found in:
HKEY_LOCAL_MACHINE\SOFTWARE\EMC CONTROL CENTER\6.1
This string contains the value used to specify the ControlCenter installation path of the Master Agent. If the key is not present, add it.
Consult the readme documentation for Patch 4563 on how to apply this patch
Run Patch 4563 on the above hosts. Once the patch is installed you will see the HF4563 folder appear. Run the backup.bat file on each server, found in the HF4563 folder, and if successful, reboot the server.
R:\EMC CONTROLCENTER\HF4563\backup.bat (Repository Server)
2010 EMC Proven Professional Knowledge Sharing 55 S:\EMC CONTROLCENTER\HF4563\backup.bat (StorageScope and Store Server)
T:\EMC CONTROLCENTER\HF4563 (Store Server)
After rebooting, copy the EMC CONTROLCENTER folder to another location.
2010 EMC Proven Professional Knowledge Sharing 56 Example: R:\EMC CONTROLCENTER copy to D:\Rollback-PreUB7\EMC CONTROLCENTER S:\EMC CONTROLCENTER copy to D:\Rollback-PreUB7\EMC CONTROLCENTER T:\EMC CONTROLCENTER copy to D:\Rollback-PreUB7\EMC CONTROLCENTER
I recommend using a third party tool such as Robocopy, or EMCopy to ensure that all files copy to the new backup location successfully.
Example: emcopy R:\EMC CONTROLCENTER D:\Rollback-PreUB7\EMC CONTROLCENTER /s /d /o /r:0 /w:1 /log+:c:\EMC ControlCentercopy.txt
2.10.16 Update Bundle 7 (Patch 4673)
Execute patch6107254_4673.exe on the A node host first.
2010 EMC Proven Professional Knowledge Sharing 57 Click Next,
Follow the on screen prompts. This will take around an hour or so to complete!
2.10.17 Patch 4744 Consult the EMC readme file before applying this patch. You must have EMC ControlCenter 6.1.0.7.254 installed before you apply Patch 4744 which upgrades the following ControlCenter 6.1 components. . • (v,6.1.0.7.289) ControlCenter Server • (v,6.1.0.7.289) ControlCenter Store • (v,6.1.0.7.289) ControlCenter console
2.10.18 Patch 4754
This patch fixes an issue with the Ramb_hotback.bat script, which failed to delete archive files older than 2 days. Follow the readme document supplied with this patch. Make sure the patch completes successfully.
2.10.19 Add local ORA_DBA Group to each Cluster Node
Add a local Group called ORA_DBA to each node. This is a solution to a problem described in EMC203001
2010 EMC Proven Professional Knowledge Sharing 58
2.10.20 Post UB7 Cluster Preparation The cluster nodes need to be made ready for ControlCenter to be able to run on either node. Perform the following procedure on the active node:
In the Services window, select the following services:
OracleServiceEMCSTSDB OracleEMC CONTROLCENTERREP_HOMETNSListener OracleServiceRAMBDB
Select Stop.
1. Change directory to R:\EMC CONTROLCENTER\tools\utils 2. Run clustersave.bat.5Upgrading ControlCenter 5.2 to 6.1 3. Using the clusteradmin utility, initiate a manual failover of the EMC CONTROLCENTERRG resource group to the B node. Right-click resource group EMC CONTROLCENTERRG and choose Move Group.
2010 EMC Proven Professional Knowledge Sharing 59 From the B node, do the following:
1. Change directory to R:\EMC CONTROLCENTER\tools\utils 2. Run clusterupgrade.bat. 3. Reboot the B node for the upgrade to take effect.
Use the Move Group command again to bring the Cluster Resources back under the A node again.
2.10.21 Confirm Master Agents are at the same patch level Check that the Master Agent on both Nodes of the Cluster are at the same version. You will see that the Active node of the cluster will show the Master Agent as Active, but it will be Inactive on the B Node. This is expected. If the Master Agent on the B Node has not upgraded to 6.1 UB7, then do the following:
On the Node that is not active (the Master Agent is not active on that node), rename all files with .COMFILE and .lck extensions found in the C:\Windows\EMC CONTROLCENTER directory.
Fail over the Cluster to restart the Master Agent on that node. The .COMFILE and .lck files will be recreated by the Master Agent when it starts. The Master Agent should now be active against that Node in the ControlCenter Console.
On the other Node (the Node that is now not active), rename all files with .COMFILE and .lck extensions in the C:\Windows\EMC CONTROLCENTER directory.
Fail over the Cluster again back to the A node, and the Master Agent will create the .COMFILE and .lck files again. The Master Agent should now be at the same patch level on both nodes.
2.10.22 Agent Patching
Patch the Agents using the ControlCentre Console. The Master Agent must be patched first! Select the hosts where the Agents are to be installed. As a general rule do not install the FCC Agent on the same host as the NAS agent because of port conflicts. Note that the Cluster nodes Do Not have any Agents installed apart
2010 EMC Proven Professional Knowledge Sharing 60 from the Master and Host agents. The following table shows what I consider to be a good spread of agents:
Host Server Agent NAS Agent, Clariion Agent (requires Naviseccli), Symmetrix Agent, SDM Agent, Master Agent, Host Agent; IntegrationGateway (if sending alerts to third party Site-emccc02 management system)
Site-emccc03 Symmetrix Agent, SDM Agent, Master Agent, Host Agent,
Site-emccc04 FCC Agent, Master Agent, Host Agent Site-emccc05 FCC Agent, Master Agent, Host Agent
The Agents should upgrade to version 6.1.0.7.254
2.11 Install ControlCenter Licences
Add the new EMC CONTROLCENTER 6.1 Licenses using the Console Install Menu.
2010 EMC Proven Professional Knowledge Sharing 61 2.12 Create ControlCenter Resources for the Cluster Service
Create the EMC CONTROLCENTERRG resources shown below. Use the Generic Service. Each has a dependency on its predecessor. It is good practice to number these Resources in the order in which they start. The Resources listed from 01 to 02 as shown below will already be created at this stage.
The Dependencies of each Resource are shown graphically below:
2010 EMC Proven Professional Knowledge Sharing 62
2010 EMC Proven Professional Knowledge Sharing 63
2010 EMC Proven Professional Knowledge Sharing 64
2010 EMC Proven Professional Knowledge Sharing 65
2010 EMC Proven Professional Knowledge Sharing 66 Note the API server is not generally required so this may be omitted.
2010 EMC Proven Professional Knowledge Sharing 67
The screen shot below shows the ControlCenter resources in the EMC CONTROLCENTERRG in an Online state. Make sure this is the case.
This now completes the installation of the cluster. Execute a Move Group and check that the EMC CONTROLCENTERRG moves to the other Node. Make sure all EMC Services come online. You will have to restart the Store(s) manually after a Move Group is executed.
2.12.1 Load StorageScope
Connect to the StorageScope Server via a web browser as follows: https://site-emccc02:30002/srm
2010 EMC Proven Professional Knowledge Sharing 68
2.12.2 Install the WLA Archive Agent
If performing an upgrade to EMC CONTROLCENTER6.1 from 6.0, this will already be in place.
Install the WLA Archiver Agent on the third HPDL 380 Server. Configure the WLA Archive and WLA Reports as folders on the T: drive.
2010 EMC Proven Professional Knowledge Sharing 69
2.12.3 Install the NAS Agent If required, install the NAS Agent on the same server as the Storage Scope server, in this instance this will be dtcp-emccc02
Once installed, navigate to the agent folder CNN610 and change the entry in the CNN.INI file under the [SSH] heading from Version = 1 to version = 2
Restart the NAS Agent.
Make sure the NAS is discovered using the Assisted Discovery wizard in ControlCenter. Use the emcplink.exe utility if there are connection problems. This can be found in the CNN610 Agent folder.
2010 EMC Proven Professional Knowledge Sharing 70 Next, run the following commands: emcplink -setpolicy +EMC_SSH_KEY_SECURITY_NONE emcplink -ssh -interactive -2 -pw nasadmin nasadmin@
Where nasadmin is the user logon, and password of nasadmin. Substitute the correct values if different. This will set up an ssh session with the Control Station and log in. Disconnect from the session.
Retry the Assisted Discovery and the Celerra™ should now discover.
2.12.4 Add Gatekeepers to Symmetrix® Agent Host
Zone in and mask 6 different Gatekeeper devices for each Symmetrix Array, to each HBA on the Symmetrix Agent hosts.
References and Related Documentation
Document / Material Location (Hyperlink/URL) 1 ControlCenter 6.1 Planning https://powerlink.emc.com/nsepn/weba Guide pps/btg548664833igtcuup4826/km/live 1/en_US/Offering_Technical/Technical_ Documentation/300-006- 387.pdf?mtcs=ZXZlbnRUeXBlPUttQ2xpY 2tDb250ZW50RXZlbnQsZG9jdW1lbnRJZ D0wOTAxNDA2NjgwNDVjMDlmLGRvY3V tZW50VHlwZT1wZGYsbmF2ZU5vZGU9M GIwMTQwNjY4MDM1YWM5Yl9Hcmlk
2010 EMC Proven Professional Knowledge Sharing 71