Cloning of Oracle Homes on Oracle RAC 11.2 Cluster Nodes After Linux Upgrade Andreas Becker, Principal Member Technical Staff Oracle Server Technologies, SAP Dev

1 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 Cloning of Oracle Homes on Oracle RAC 11.2 Cluster Nodes after Linux Upgrade Andreas Becker, Principal Member Technical Staff Oracle Server Technologies, SAP Dev. Program Agenda

. The Project . The Plan . The Big Picture . DelNode/AddNode vs. Oracle Home Cloning . Summary

5 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

. SAP System on Oracle RAC . Oracle RAC Cluster with 4 nodes . Oracle Release 11.2.0.3 . Oracle Linux Release 6.1 on SAP Application Server . Oracle Linux Release 5.8 on Oracle RAC nodes . ASM + ACFS . Target: Oracle Linux Release 6.1 on all Oracle RAC nodes

SAP User

SAP Instance 1 SAP Instance 2 SAP Instance 3 SAP Instance 4 SAP Instances

OL6.1 OL6.1 OL6.1 OL6.1 SAP Oracle S1 S2 S3 RAC Instance S4 RAC Instances RAC Instance RAC Instance RAC Instance Services ASCS ERS ASM / ACFS OL5.8 RAC Nodes Node #1 OL5.8 Node #2 OL5.8 Node #3 OL5.8 Node #4 OL5.8 OL6.1

RAC Database Database

11 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 Linux OS Upgrade Oracle Linux In-place upgrading from a major version of Oracle Linux 5 or earlier is not supported. Although Anaconda provides an option to perform an upgrade, fresh installation is strongly recommended.

Oracle® Linux Release Notes for Release 6.4 http://docs.oracle.com/cd/E37670_01/E39522/html/ol _sup_64rn.html

12 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 Linux OS Upgrade Red Hat Linux Red Hat does not support in-place upgrades between any major versions of Red Hat Enterprise Linux.

Red Hat Enterprise Linux 6 Installation Guide https://access.redhat.com/site/documentation/en- US/Red_Hat_Enterprise_Linux/6/html/Installation_Guid e/ch-upgrade-x86.html

. Technical restrictions – Linux upgrade requires fresh installation of all RAC nodes

. SAP System HA requirements – SAP application must remain available, no longer downtimes – Downtimes due to SAP application server restarts are acceptable – 2 of the 4 RAC nodes are sufficient run the SAP system

. Project goal: Linux OS Upgrade – No other requirements or changes – No hardware replacement – No host name change – No network configuration change, no IP change

. 2 RAC nodes are sufficient to run the SAP load (for a while) .  2-phases – Phase 1: „OS upgrade“ of RAC nodes 1 and 2 – Phase 2: „OS upgrade“ of RAC nodes 3 and 4

. Between Phase 1 and Phase 2 the system would run with different OS versions (for a while) MOS 220970.1 For Oracle Database 11g Release 2, Oracle Clusterware and ASM binaries are combined into a single ORACLE_HOME called the grid infrastructure home. This home fully supports rolling upgrades for patches, bundles, patchsets and releases. (If you are upgrading ASM from Oracle Database 10g to 11g Release 2, you will not be able to upgrade ASM in a rolling fashion.)

The Oracle Clusterware and Oracle Real Application Clusters both support rolling upgrades of the OS software when the version of the Oracle Database is certified on both releases of the OS (and the OS is the same, no Linux and Windows or AIX and Solaris, or 32 and 64 bit etc.). This can apply a patch to the operating system, a patchset (such as EL4u4 to EL4u6) or a release (EL4 to EL5).

. Oracle® Database SAP® Bundle Patch 11.2.0.3.7 - 201309 for Grid Infrastructure

If you restrict installation of GI-PSU 11.2.0.3.7 and this SBP to the grid homes and the non-SAP homes as described above, you can also keep the SAP databases on the cluster and the SAP applications running on these available. However, due to limitations in Oracle's connection handling such a RAC-rolling grid home SBP installation is not transparent to the SAP applications connected against the databases of the cluster. To avoid transaction or other errors in the SAP applications when installing this SBP in a rolling manner, follow the additional instructions in section 2.4, "Maintaining SAP Applications during a RAC-Rolling Patch Installation".

…To avoid transaction or other errors in the SAP applications when installing GI-PSU 11.2.0.3.7 and the Grid Infrastructure patches of this SBP in a rolling manner, you must follow the installation instructions below on each of the cluster nodes in turn:

All connections from SAP applications to SAP database instances running on the cluster node must be terminated by the SAP application before installing GI-PSU 11.2.0.3.7 and the Grid Infrastructure patches of this SBP on the cluster node.

Any SAP application component (for example, the Enqueue Server) which is dependent on the Oracle Clusterware on a cluster node must be stopped by the SAP application before installing GI-PSU 11.2.0.3.7 and the Grid Infrastructure patches of this SBP on the cluster node. …

…To avoid transaction or other errors in the SAP applications… 1. All connections from SAP applications to SAP database instances running on the cluster node must be terminated by the SAP application. … 2. Any SAP application component (for example, the Enqueue Server) which is dependent on the Oracle Clusterware on a cluster node must be stopped by the SAP application…

SAP User

SAP Instance 1 SAP Instance 2 SAP Instance 3 SAP Instance 4

OL6.1 OL6.1 OL6.1 OL6.1 SAP Oracle S1 S2 S3 RAC Instance S4 RAC Instance RAC Instance RAC Instance Services ASCS ERS

RAC Node #1 OL6.1OL5.8 OL5.8OL6.1 OL6.1OL5.8 OL6.1 Node #2 Node #3 Node #4 OL5.8OL6.1 Nodes

OCR RAC Voting Database

SAP User

SAP Instance 1 SAP Instance 2 SAP Instance 3 SAP Instance 4 SAP Oracle S1 S2 S3 RAC Instance S4 RAC Instance RAC Instance RAC Instance Services ASCS ERS

RAC Node #1 OL5.8 OL5.8 OL5.8 OL6.1 Node #2 Node #3 Node #4 OL5.8 Nodes

OCR RAC Voting Database

28 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 The Big Picture 1. Block SAP instances 1 and 2 for logon and disable batch jobs Preparations 2. Inform SAP user to reconnect 3. Inform SAP user about SAP instance restart 4. Relocate database services 5. Restart SAP instance(s) 1 and 2 6. Allow logon on SAP instances 1 and 2 and enable batch jobs again SAP Instance 1 SAP Instance 2 SAP Instance 3 SAP Instance 4 SAP Oracle S1 S2 S3 RAC Instance S4 RAC Instance RAC Instance RAC Instance Services ASCS ERS

RAC Node #1 OL5.8 OL5.8 OL5.8 OL6.1 Node #2 Node #3 Node #4 OL5.8 Nodes

OCR RAC Voting Database

29 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 The Big Picture 1. Block SAP instances 1 and 2 for logon and disable batch jobs Preparations 2. Inform SAP user to reconnect 3. Inform SAP user about SAP instance restart 4. Relocate database services 5. Restart SAP instance(s) 1 and 2 6. Allow logon on SAP instances 1 and 2 and enable batch jobs again SAP Instance 1 SAP Instance 2 SAP Instance 3 SAP Instance 4 SAP Oracle S1 S2 S3 RAC Instance S4 RAC Instance RAC Instance RAC Instance Services ASCS ERS

RAC Node #1 OL5.8 OL5.8 OL5.8 OL6.1 Node #2 Node #3 Node #4 OL5.8 Nodes

OCR RAC Voting Database

30 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 The Big Picture 1. Block SAP instances 1 and 2 for logon and disable batch jobs Preparations 2. Inform SAP user to reconnect 3. Inform SAP user about SAP instance restart 4. Relocate database services 5. Restart SAP instance(s) 1 and 2 6. Allow logon on SAP instances 1 and 2 and enable batch jobs again SAP Instance 1 SAP Instance 2 SAP Instance 3 SAP Instance 4 $ srvctl relocate service… $ srvctl relocate service… SAP

S1 S2 Oracle S1 S2 S3 RAC Instance S4 RAC Instance RAC Instance RAC Instance Services ASCS ERS

RAC Node #1 OL5.8 OL5.8 OL5.8 OL6.1 Node #2 Node #3 Node #4 OL5.8 Nodes

OCR RAC Voting Database

31 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 The Big Picture 1. Block SAP instances 1 and 2 for logon and disable batch jobs Preparations 2. Inform SAP user to reconnect 3. Inform SAP user about SAP instance restart 4. Relocate database services 5. Restart SAP instance(s) 1 and 2 6. Allow logon on SAP instances 1 and 2 and enable batch jobs again SAP Instance 1 SAP Instance 2 SAP Instance 3 SAP Instance 4 S3 S1 S2 S4 SAP

S1 S2 Oracle S3 RAC Instance S4 RAC Instance RAC Instance RAC Instance Services ASCS ERS

RAC Node #1 OL5.8 OL5.8 OL5.8 OL6.1 Node #2 Node #3 Node #4 OL5.8 Nodes

OCR RAC Voting Database

32 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 The Big Picture 1. Block SAP instances 1 and 2 for logon and disable batch jobs Preparations 2. Inform SAP user to reconnect 3. Inform SAP user about SAP instance restart 4. Relocate database services 5. Restart SAP instance(s) 1 and 2 6. Allow logon on SAP instances 1 and 2 and enable batch jobs again SAP Instance 1 SAP Instance 2 SAP Instance 3 SAP Instance 4 SAP

S1 S2 Oracle S3 RAC Instance S4 RAC Instance RAC Instance RAC Instance Services ASCS ERS

RAC Node #1 OL5.8 OL5.8 OL5.8 OL6.1 Node #2 Node #3 Node #4 OL5.8 Nodes

OCR RAC Voting Database

SAP Instance 1 SAP Instance 2 SAP Instance 3 SAP Instance 4 SAP

S1 S2 Oracle S3 RAC Instance S4 RAC Instance RAC Instance RAC Instance Services ASCS ERS ERS

RAC Node #1 OL5.8 OL5.8 OL5.8 OL6.1 Node #2 Node #3 Node #4 OL5.8 Nodes

OCR RAC Voting Database

SAP Instance 1 SAP Instance 2 SAP Instance 3 SAP Instance 4 SAP

S1 S2 Oracle S3 RAC Instance S4 RAC Instance RAC Instance RAC Instance Services ASCS ASCS ERS

RAC Node #1 OL5.8 OL5.8 OL5.8 OL6.1 Node #2 Node #3 Node #4 OL5.8 Nodes

OCR RAC Voting Database

SAP Instance 1 SAP Instance 2 SAP Instance 3 SAP Instance 4 SAP

S1 S2 Oracle S3 RAC Instance S4 RAC Instance RAC Instance RAC Instance Services ASCS ERS ERS

RAC Node #1 OL5.8 OL5.8 OL5.8 OL6.1 Node #2 Node #3 Node #4 OL5.8 Nodes

OCR RAC Voting Database

SAP Instance 1 SAP Instance 2 SAP Instance 3 SAP Instance 4 SAP

S1 S2 Oracle S3 RAC Instance S4 RAC Instance RAC Instance RAC Instance Services ASCS ERS

RAC Node #1 OL5.8 OL5.8 OL5.8 OL6.1 Node #2 Node #3 Node #4 OL5.8 Nodes

OCR RAC Voting Database

37 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 The Big Picture 1. Stop RAC instances on nodes 1 and 2 $ srvctl stop instance –i -d Preparations 2. Stop CRS on nodes 1 and 2 (root) # crsctl stop crs

SAP Instance 1 SAP Instance 2 SAP Instance 3 SAP Instance 4 SAP

S1 S2 Oracle S3 RAC Instance S4 RAC Instance RAC Instance RAC Instance Services ASCS ERS

RAC Node #1 OL5.8 OL5.8 OL5.8 OL6.1 Node #2 Node #3 Node #4 OL5.8 Nodes

OCR RAC Voting Database

SAP Instance 1 SAP Instance 2 SAP Instance 3 SAP Instance 4 SAP

S1 S2 Oracle S3 RAC Instance S4 RAC Instance Services ASCS ERS

RAC Node #1 OL5.8 OL5.8 OL5.8 OL6.1 Node #2 Node #3 Node #4 OL5.8 Nodes

OCR RAC Voting Database

. The Project . The Plan . The Big Picture . DelNode/AddNode vs. Oracle Home Cloning . Summary

1. Backup Grid home, database home(s), oratab, host configuration files, user home directories, user configuration, /etc/hosts, /etc/passwd, … 2. Stop database instances and CRS 3. Install the new Linux version and configure the node exactly as before hostname, IP, accounts, user homes, …). 4. Ensure that installation prerequisites are met 5. Restore backup of Grid home and database home(s) 6. Clone Grid home and database home(s) 7. Start CRS and database instances

SAP Instance 1 SAP Instance 2 SAP Instance 3 SAP Instance 4 SAP

S1 S2 Oracle S3 RAC Instance S4 RAC Instance Services ASCS ERS

RAC Node #1 OL5.8 OL5.8 OL5.8 OL6.1 Node #2 Node #3 Node #4 OL5.8 Nodes

Node 1 Node 2 Node 3 OCR RAC Node 4 Voting Database

47 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 The Alternatives 1. Backup Grid home and database home on each node. Oracle home Cloning Approach 2. Backup /etc/oraInst.loc, /etc/oratab 3. Backup OS configuration files like /etc/hosts, /etc/groups, /etc/passwd 4. ASM: mapping of ASM disks

SAP Instance 1 SAP Instance 2 SAP Instance 3 SAP Instance 4 SAP

S1 S2 Oracle gi_home_node1.tar gi_home_node2.tar S3 S4 RAC Instance RAC Instance db_home_node1.tar db_home_node2.tar Services

$ tar cvf … $ tar cvf … ASCS ERS

RAC Node #1 OL5.8 OL5.8 OL5.8 OL6.1 Node #2 Node #3 Node #4 OL5.8 Nodes

Node 1 Node 2 Node 3 OCR RAC Node 4 Voting Database

SAP Instance 1 SAP Instance 2 SAP Instance 3 SAP Instance 4 SAP

S1 S2 Oracle S3 S4 RAC Instance RAC Instance Install Linux OS Services ASCS ERS

RAC Node #1 OL5.8OL6.1 OL5.8OL6.1 OL5.8 OL6.1 Node #2 Node #3 Node #4 OL5.8 Nodes

Node 1 Node 2 Node 3 OCR RAC Node 4 Voting Database

SAP Instance 1 SAP Instance 2 SAP Instance 3 SAP Instance 4 SAP

S1 S2 Oracle gi_home_node1.tar gi_home_node2.tar S3 S4 RAC Instance RAC Instance db_home_node1.tar db_home_node2.tar Services

$ tar xvf … $ tar xvf … ASCS ERS $ clone.pl $ clone.pl RAC Node #1 OL5.8OL6.1 OL5.8OL6.1 OL5.8 OL6.1 Node #2 Node #3 Node #4 OL5.8 Nodes

Node 1 Node 2 Node 3 OCR RAC Node 4 Voting Database

Cloning the Grid home $ su root -c "$OHGRID/crs/install/rootcrs.pl -unlock“ $ /usr/bin/perl $OHGRID/clone/bin/clone.pl ORACLE_BASE=$ORACLE_BASE ORACLE_HOME=$OHGRID -defaultHomeName INVENTORY_LOCATION=/oracle/oraInventory -O'"CLUSTER_NODES={}"' -O'"LOCAL_NODE="' CRS=false - O"SHOW_ROOTSH_CONFIRMATION=false“ su root -c "$OHGRID/rdbms/install/rootadd_rdbms.sh" su root -c "$OHGRID/crs/install/rootcrs.pl -patch" su root -c "chmod 755 /opt/oracle“ Cloning the Database home $ $ORACLE_HOME/clone/bin/clone.pl ORACLE_HOME=$ORACLE_HOME ORACLE_BASE=$ORACLE_BASE -defaultHomeName $ $ORACLE_HOME/oui/bin/runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES={}" # $ORACLE_HOME/root.sh

For details, see My Oracle Support note MOS 1136544.1.

51 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 The Alternatives 1. Start CRS on new nodes 2. Start RAC instances on new nodes Oracle home Cloning Approach Next: Rebalance

SAP Instance 1 SAP Instance 2 SAP Instance 3 SAP Instance 4 SAP

S1 S2 Oracle S3 RAC Instance S4 RAC Instance RAC Instance RAC Instance Services ASCS ERS

RAC Node #1 OL5.8OL6.1 OL5.8OL6.1 OL5.8 OL6.1 Node #2 Node #3 Node #4 OL5.8 Nodes

Node 1 Node 2 Node 3 OCR RAC Node 4 Voting Database

52 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 The Alternatives 1. Rebalance to complete phase 1  Phase 1 complete Oracle home Cloning Approach Next steps: 1. Start phase 2 2. Rebalance to complete phase 2

SAP Instance 1 SAP Instance 2 SAP Instance 3 SAP Instance 4 SAP Oracle S1 S2 S3 RAC Instance S4 RAC Instance RAC Instance RAC Instance Services ASCS ERS

RAC Node #1 OL5.8OL6.1 OL5.8OL6.1 OL5.8 OL6.1 Node #2 Node #3 Node #4 OL5.8 Nodes

Node 1 Zero OCR Node 2 Node 3 OCR RAC updates Node 4 Voting Database

SAP Instance 1 SAP Instance 2 SAP Instance 3 SAP Instance 4 SAP Oracle S1 S2 S3 RAC Instance S4 RAC Instance RAC Instance RAC Instance Services ASCS ERS

RAC Node #1 OL5.8OL6.1 OL5.8OL6.1 OL5.8OL6.1 OL6.1 Node #2 Node #3 Node #4 OL5.8OL6.1 Nodes

Node 1 Zero OCR Node 2 Node 3 OCR RAC updates Node 4 Voting Database

1. Remove a RAC node from the cluster 2. Install the new Linux version 3. Add the node to the cluster again.

Repeat steps 1-3 for all nodes that need to be installed. When finished, backup OCR. . DelNode: /bin/crsctl delete node -n . AddNode: /oui/bin/addNode.sh

SAP Instance 1 SAP Instance 2 SAP Instance 3 SAP Instance 4 SAP

S1 S2 Oracle S3 RAC Instance S4 RAC Instance Services ASCS ERS

RAC Node #1 OL5.8 OL5.8 OL5.8 OL6.1 Node #2 Node #3 Node #4 OL5.8 Nodes

Node 1 Node 2 Node 3 OCR RAC Node 4 Voting Database

SAP Instance 1 SAP Instance 2 SAP Instance 3 SAP Instance 4 SAP

S1 S2 Oracle S3 RAC Instance S4 RAC Instance Services $ crsctl delete node node 1 ASCS ERS

RAC OL5.8 OL5.8 OL6.1 Node #2 Node #3 Node #4 OL5.8 Nodes

OL5.8 Node 2 Node #1 Node 3 Node 4 OCR RAC Voting Database

SAP Instance 1 SAP Instance 2 SAP Instance 3 SAP Instance 4 SAP

S1 S2 Oracle S3 RAC Instance S4 RAC Instance Services ASCS ERS $ crsctl delete node node 2 RAC OL5.8 OL6.1 Node #3 Node #4 OL5.8 Nodes

OL5.8 Node 3 Node #1 2 OCR Node 4 OCR RAC updates Voting Database Node #2 OL5.8

SAP Instance 1 SAP Instance 2 SAP Instance 3 SAP Instance 4 SAP

S1 S2 Oracle S3 RAC Instance S4 RAC Instance Services ASCS ERS

RAC OL5.8 OL6.1 Node #3 Node #4 OL5.8 Nodes

OL5.8OL6.1 Node 3 Node #1n 2 OCR Node 4 RAC updates OCR Voting Database OL6.1 Node #2n OL5.8

60 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 The Alternatives 1. Remove nodes from RAC cluster 2. Install new OS version DelNode/AddNode Approach 3. Add nodes to RAC cluster again

SAP Instance 1 SAP Instance 2 SAP Instance 3 SAP Instance 4 SAP

S1 S2 Oracle S3 RAC Instance S4 RAC Instance Services $ addNode.sh ASCS ERS

RAC Node #1n OL5.8OL6.1 OL5.8 OL6.1 Node #3 Node #4 OL5.8 Nodes

Node 1n 3 OCR Node 3 Node 4 RAC updates OCR Voting Database OL6.1 Node #2n OL5.8

61 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 The Alternatives 1. Remove nodes from RAC cluster 2. Install new OS version DelNode/AddNode Approach 3. Add nodes to RAC cluster again

SAP Instance 1 SAP Instance 2 SAP Instance 3 SAP Instance 4 SAP

S1 S2 Oracle S3 RAC Instance S4 RAC Instance Services $ addNode.sh ASCS ERS

RAC Node #1n OL5.8OL6.1 OL5.8OL6.1 OL5.8 OL6.1 Node #2n Node #3 Node #4 OL5.8 Nodes

Node 1n 4 OCR Node 2n Node 3 OCR RAC updates Node 4 Voting Database

SAP Instance 1 SAP Instance 2 SAP Instance 3 SAP Instance 4 SAP

S1 S2 Oracle S3 RAC Instance S4 RAC Instance RAC Instance RAC Instance Services ASCS ERS

RAC Node #1n OL5.8OL6.1 OL5.8OL6.1 OL5.8 OL6.1 Node #2n Node #3 Node #4 OL5.8 Nodes

Node 1n Node 2n Node 3 OCR RAC Node 4 Voting Database

63 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 The Alternatives 1. Rebalance again: move SAP users, SAP Database Services, and SAP instances DelNode/AddNode Approach back to RAC instances on nodes 1 and 2

SAP Instance 1 SAP Instance 2 SAP Instance 3 SAP Instance 4 SAP Oracle S1 S2 S3 RAC Instance S4 RAC Instance RAC Instance RAC Instance Services ASCS ERS

RAC Node #1n OL5.8OL6.1 OL5.8OL6.1 OL5.8 OL6.1 Node #2n Node #3 Node #4 OL5.8 Nodes

Node 1n Node 2n Node 3 OCR RAC Node 4 Voting Database

64 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 The Alternatives Phase 1 completed. Phase 2: next to start DelNode/AddNode Approach Oracle RAC is running OL 6.1 and OL 5.8 (mix).

SAP Instance 1 SAP Instance 2 SAP Instance 3 SAP Instance 4 SAP Oracle S1 S2 S3 RAC Instance S4 RAC Instance RAC Instance RAC Instance Services ASCS ERS

RAC Node #1n OL5.8OL6.1 OL5.8OL6.1 OL5.8 OL6.1 Node #2n Node #3 Node #4 OL5.8 Nodes

Node 1n 4 OCR Node 2n Node 3 OCR RAC updates Node 4 Voting Database

SAP Instance 1 SAP Instance 2 SAP Instance 3 SAP Instance 4 SAP Oracle S1 S2 S3 RAC Instance S4 RAC Instance RAC Instance RAC Instance Services ASCS ERS

RAC Node #1n OL6.1OL5.8 OL5.8OL6.1 OL6.1OL5.8 OL6.1 Node #2n Node #3n Node #4n OL5.8OL6.1 Nodes

Node 1n 8 OCR Node 2n Node 3n OCR RAC updates Node 4n Voting Database

. The Project . The Plan . The Big Picture . DelNode/AddNode vs. Oracle Home Cloning . Summary

DelNode/AddNode Oracle Home Cloning

• Nodes are removed from cluster and added to • Zero Addnode/DelNode operations cluster • Every ‚node add‘/‘node remove‘ operation changes the OCR and affects the remaining cluster • N-node-RAC  2xN OCR configuration changes • OCR configuration before/after is not identical • OCR/Cluster configuration remains • OCR backup required when finished unchanged

DelNode/AddNode Oracle Home Cloning

• General approach • More specific approach

Suitable for Suitable for • Cluster extensions • OS upgrade / OS fresh installations • Cluster configuration changes (host names, • Rolling GI Home Patching hardware, IP address) • Preparation of new nodes while system is online • Hardware exchange

SAP Notes

. 1696869 - Patching of Oracle Homes with Minimal Downtime http://service.sap.com/sap/support/notes/1696869 . Rolling Patching Oracle® Database SAP® Bundle Patch 11.2.0.3.7 - 201309 for Grid Infrastructure README

MOS Notes

. Oracle Support Document 1136544.1 (Minimal downtime patching via cloning 11gR2 ORACLE_HOME directories) can be found at: https://support.oracle.com/epmos/faces/DocumentDisplay?id=1136544.1 . Oracle Support Document 1262925.1 (How to remove/delete a node from Grid Infrastructure Clusterware when the node has failed) can be found at: https://support.oracle.com/epmos/faces/DocumentDisplay?id=1262925.1 . Oracle Support Document 1332451.1 (How to Add Node/Instance or Remove Node/Instance in 10gR2, 11gR1 and 11gR2 Oracle Clusterware and RAC) can be found at: https://support.oracle.com/epmos/faces/DocumentDisplay?id=1332451.1 . Oracle Support Document 220970.1 (RAC: Frequently Asked Questions) can be found at: https://support.oracle.com/epmos/faces/DocumentDisplay?id=220970.1

Other

. Red Hat Enterprise Linux 6 Product Documentation https://access.redhat.com/site/documentation/en- US/Red_Hat_Enterprise_Linux/6/html/Installation_Guide/ch-upgrade-x86.html . Oracle Linux Release Notes for Release 6.4 Supported Upgrade Path http://docs.oracle.com/cd/E37670_01/E39522/html/ol_sup_64rn.html

78 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 80 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 81 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 © 2013 SAP AG. All rights reserved.

No part of this publication may be reproduced or transmitted in any form or for any purpose Google App Engine, Google Apps, Google Checkout, Google Data API, Google Maps, without the express permission of SAP AG. The information contained herein may be Google Mobile Ads, Google Mobile Updater, Google Mobile, Google Store, Google Sync, changed without prior notice. Google Updater, Google Voice, Google Mail, Gmail, YouTube, Dalvik and Android are trademarks or registered trademarks of Google Inc. Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors. INTERMEC is a registered trademark of Intermec Technologies Corporation. Microsoft, Windows, Excel, Outlook, PowerPoint, Silverlight, and Visual Studio are Wi-Fi is a registered trademark of Wi-Fi Alliance. registered trademarks of Microsoft Corporation. Bluetooth is a registered trademark of Bluetooth SIG Inc. IBM, DB2, DB2 Universal Database, System i, System i5, System p, System p5, System x, System z, System z10, z10, z/VM, z/OS, OS/390, zEnterprise, PowerVM, Power Motorola is a registered trademark of Motorola Trademark Holdings LLC. Architecture, Power Systems, POWER7, POWER6+, POWER6, POWER, PowerHA, Computop is a registered trademark of Computop Wirtschaftsinformatik GmbH. pureScale, PowerPC, BladeCenter, System Storage, Storwize, XIV, GPFS, HACMP, RETAIN, DB2 Connect, RACF, Redbooks, OS/2, AIX, Intelligent Miner, WebSphere, Tivoli, SAP, R/3, SAP NetWeaver, Duet, PartnerEdge, ByDesign, SAP BusinessObjects Explorer, Informix, and Smarter Planet are trademarks or registered trademarks of IBM Corporation. StreamWork, SAP HANA, and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany Linux is the registered trademark of Linus Torvalds in the United States and other countries. and other countries. Adobe, the Adobe logo, Acrobat, PostScript, and Reader are trademarks or registered Business Objects and the Business Objects logo, BusinessObjects, Crystal Reports, Crystal trademarks of Adobe Systems Incorporated in the United States and other countries. Decisions, Web Intelligence, Xcelsius, and other Business Objects products and services Oracle and Java are registered trademarks of Oracle and its affiliates. mentioned herein as well as their respective logos are trademarks or registered trademarks of Business Objects Software Ltd. Business Objects is an SAP company. UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group. Sybase and Adaptive Server, iAnywhere, Sybase 365, SQL Anywhere, and other Sybase Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin products and services mentioned herein as well as their respective logos are trademarks or are trademarks or registered trademarks of Citrix Systems Inc. registered trademarks of Sybase Inc. Sybase is an SAP company. HTML, XML, XHTML, and W3C are trademarks or registered trademarks of W3C®, Crossgate, m@gic EDDY, B2B 360°, and B2B 360° Services are registered trademarks World Wide Web Consortium, Massachusetts Institute of Technology. of Crossgate AG in Germany and other countries. Crossgate is an SAP company. Apple, App Store, iBooks, iPad, iPhone, iPhoto, iPod, iTunes, Multi-Touch, Objective-C, All other product and service names mentioned are the trademarks of their respective Retina, Safari, Siri, and Xcode are trademarks or registered trademarks of Apple Inc. companies. Data contained in this document serves informational purposes only. National product specifications may vary. IOS is a registered trademark of Cisco Systems Inc. The information in this document is proprietary to SAP. No part of this document may be RIM, BlackBerry, BBM, BlackBerry Curve, BlackBerry Bold, BlackBerry Pearl, BlackBerry reproduced, copied, or transmitted in any form or for any purpose without the express prior Torch, BlackBerry Storm, BlackBerry Storm2, BlackBerry PlayBook, and BlackBerry App written permission of SAP AG. World are trademarks or registered trademarks of Research in Motion Limited.