An Oracle Technical White Paper January 2013
Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster Based on the Oracle Optimized Solution for SAP
Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
About this Document ...... 2 Introduction ...... 3 Solution Layout Overview ...... 3 Highly Available Architecture ...... 4 Oracle Solaris Cluster Setup ...... 5 Step 1. Configuring Hostnames Resolution ...... 5 Step 2. Quorum device iSCSI LUN and NFS file systems ...... 7 Step 3. Oracle Solaris Cluster installation ...... 10 Step 4. Configuring the Oracle Solaris 10 GP domains into a cluster ...... 13 Initial Zone Cluster Setup ...... 16 Step 5. Creation of the zone clusters for SAP ...... 16 Step 6. Preparing the zone cluster environment ...... 23 Step 7. SPARC SuperCluster specific tuning ...... 23 Step 8. Creating initial resource groups and logical host resources ...... 23 Step 9. Preparing the NFS file systems for Zone Clusters ...... 25 Step 10. Configuring the plug-in for Oracle Solaris Cluster fencing and creating storage resources ...... 28 SAP Setup ...... 36 Step 11. Installing SAP ...... 36 Step 12. Database Migration to Exadata Storage ...... 92 Step 13. Add RAC VIPs and Listeners on the IPoIB network ...... 103 Step 14. Connect the SAP instances to Oracle RAC ...... 106 Step 15. Move the Connections from 10 GbE to InfiniBand ...... 110 Step 16. SAP Resource Creation ...... 111 Final Zone Cluster Setup ...... 121 Step 17. Finalize the Storage Configuration ...... 121 Step 18. Configure the Oracle External Proxy resource ...... 121 Step 19. Configure the cross-zone clusters dependencies ...... 122 Step 20. Configure CPU resource control for the zone clusters ...... 123 Step 21. Configure Policy-based Failover ...... 123 References and Documentation ...... 125 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
About this Document
This document contains best practice examples for the installation and configuration of Highly Available SAP on the Oracle SPARC SuperCluster T4-4 engineered system. Created and validated during the development of the Oracle Optimized Solution for SAP, these step-by-step instructions show how to prepare and configure the infrastructure, including zone clusters, RAC Framework, or creating users and resource groups for SAP. The examples provided in this document follow and complement the official product documentation and best practices published by SAP and Oracle and do not replace these documents. The References section lists links to recommended additional documentation, whitepapers and best practices related to installing and configuring SAP on a similar infrastructure or its subcomponents. These instructions apply to a fully functional SPARC SuperCluster system, in good operating condition, with all its components in their default pre-configured states as delivered by Oracle service engineers. The intent of this document is neither to diagnose nor to reconfigure the infrastructure. Contact your Oracle service representative for instructions to fix or reconfigure your Oracle systems if necessary. For best results, it is recommended to leverage the Oracle Solution Center for SAP and Oracle Advanced Customer Services (ACS) to deploy this solution. The Oracle Solution Centers provide presales support, including capacity planning, proof of concept (POC) deployments and IT infrastructure optimization, to help customers discover a proven way to architect SAP deployments on Oracle technology. A simple engagement framework provides access to Oracle’s state-of-the-art facilities and Oracle and SAP architects, as well as Oracle Database and hardware support teams. With a global presence and facilities located in Santa Clara (California, USA), Walldorf (Germany), and Tokyo (Japan) the Oracle Solution Center team helps IT organizations assess current SAP deployments and achieve optimal results when transitioning to Oracle engineered systems, new Oracle servers, and other Oracle technology. To learn more about Oracle Solutions Center for SAP, visit http://www.oracle.com/osc. Oracle Advanced Customer Services (ACS) are designed to help customers achieve the optimization of SAP on SPARC SuperCluster architectures. Oracle staff members possess deep product knowledge and, together with the Oracle Solution Center for SAP, are used frequently to supply staffing and deliver content for SAP technical events, as well as provide Oracle database and systems related SAP Notes. Offering a full range of services—Install and Configuration Diagnostic Review, Installation Service, Configuration Service, SAP-specific Oracle-to-Oracle (O2O) and Oracle-to-Oracle Online (Triple O) Migrations, Production Support Readiness, and a Patch Deployment Service—Oracle ACS can help IT organizations get SAP landscapes up and running on engineered systems quickly.
2 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Introduction
This document reproduces the steps that were followed by the Oracle engineering team while configuring SPARC SuperCluster and installing SAP during the development of the Oracle Optimized Solution for SAP. Additional architecture details and test results can be found in the Oracle technical white paper “How to Improve the Efficiency and Performance of an SAP Environment with the Oracle Optimized Solution for SAP” available from: http://www.oracle.com/us/solutions/oos/sap/overview/index.html
Solution Layout Overview For this example, Oracle VM for SPARC was used to divide each SPARC T4-4 server node into two separate domains, the App domain (also called General Purpose domain or GP domain) where SAP is installed and the Database domain (DB domain) running Oracle Database 11g Release 2 and Oracle Real Application Clusters (RAC).
Figure 1. Each SPARC T4-4 node is split in two domains using Oracle VM for SPARC.
3 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Highly Available Architecture This configuration example is for an ABAP-only SAP system "QO1" with Oracle Database.
Figure 2. Virtual clusters ensure high availability of critical SAP services. Oracle Solaris Cluster is leveraged in the GP Domains to ensure maximum availability of the SAP Central Services and of the business-critical application servers. The number of necessary zone clusters will vary depending on the requirements of the implementation. 4 zone clusters were created in our example:
4 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Oracle Solaris Cluster Setup
Oracle Solaris Cluster extends the Oracle Solaris OS to provide enhanced availability to SAP environments. Oracle Solaris Cluster delivers instant support for SAP through specific integration modules called agents that monitor an application to detect whether it is operating correctly, or take action if a problem is detected. The steps in this section are an example of Oracle Solaris Cluster setup for an SAP environment and would have to be repeated for each zone cluster to be created. The ctelnet application allows connecting to multiple nodes using telnet or ssh and issuing commands that will be sent to all nodes, avoiding repetition. For more details see the Oracle Solaris Cluster Reference Manual.
Step 1. Configuring Hostnames Resolution As a best practice, highly available environments should be able recover without dependency to external services, such as name resolution services, therefore any hostname used during the configuration should be resolvable on each node locally using the /etc/hosts file. On each node, add to the global zone file /etc/hosts, all physical and logical hostnames as used for the zone clusters and SAP configurations.
Important considerations regarding hostname resolution All the hosts running SAP instances need to be in /etc/hosts with their fully qualified domain name (FQDN). Hosts with a name starting with an “e” can be accessed externally over the 10 GbE interfaces and should be resolvable by the infrastructure DNS or other name resolution services. It is recommended to include the FQDN for these hosts in the /etc/hosts file. Hosts with a name starting with an “i” are strictly internal (on the InfiniBand ports) and used within the SPARC SuperCluster system only. They do not need to be resolvable by the infrastructure DNS or any other name service outside of SPARC SuperCluster. It is however required to include the FQDN for these hosts in the /etc/hosts file.
The following example shows the file /etc/hosts from one of the node, as used in our example environment. Each node should have an identical configuration.
5 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
# cat >> /etc/hosts # hostnames on the 10GbE network 10.129.184.150 esapdev1 10.129.184.151 esapdev2 10.129.184.152 esapdev3 10.129.184.153 esapdev4
10.129.184.154 escs1 10.129.184.155 escs2 10.129.184.156 escs3 10.129.184.157 escs4
10.129.184.158 eapps1 10.129.184.159 eapps2 10.129.184.160 eapps3 10.129.184.161 eapps4
10.129.184.162 epas1 10.129.184.163 epas2 10.129.184.164 epas3 10.129.184.165 epas4
10.129.184.166 esapdb-lh 10.129.184.167 escs-lh 10.129.184.168 erep-lh 10.129.184.169 ewas-lh 10.129.184.170 eapp1-lh 10.129.184.171 eapp2-lh 10.129.184.185 eapp3-lh 10.129.184.186 eapp4-lh 10.129.184.7 eapp5-lh 10.129.184.8 eapp6-lh 10.129.184.9 eapp7-lh 10.129.184.10 eapp8-lh 10.129.184.172 epas-lh
# hostnames on the IPoIB network 192.168.30.1 etc4sn1-stor etc4sn1-stor.example.domain.com 192.168.30.11 etc4-01-app1-stor etc4-01-app1-stor.example.domain.com 192.168.30.12 etc4-02-app1-stor etc4-02-app1-stor.example.domain.com 192.168.30.13 etc4-03-app1-stor etc4-03-app1-stor.example.domain.com 192.168.30.14 etc4-04-app1-stor etc4-04-app1-stor.example.domain.com 192.168.30.25 etc4-01-ib-vip 192.168.30.26 etc4-02-ib-vip 192.168.30.27 etc4-03-ib-vip 192.168.30.28 etc4-04-ib-vip 192.168.30.31 isapdev1 192.168.30.32 isapdev2 192.168.30.33 isapdev3 192.168.30.34 isapdev4 isapdev4.example.domain.com 192.168.30.35 isapdb-lh 192.168.30.41 iscs1 192.168.30.42 iscs2 192.168.30.43 iscs3 192.168.30.44 iscs4 192.168.30.45 iscs-lh iscs-lh.example.domain.com 192.168.30.46 irep-lh irep-lh.example.domain.com 192.168.30.51 iapps1 192.168.30.52 iapps2 192.168.30.53 iapps3 192.168.30.54 iapps4 192.168.30.55 iapp1-lh iapp1-lh.example.domain.com 192.168.30.56 iapp2-lh iapp2-lh.example.domain.com 192.168.30.57 iapp3-lh iapp3-lh.example.domain.com 192.168.30.58 iapp4-lh iapp4-lh.example.domain.com 192.168.30.66 iapp5-lh iapp5-lh.example.domain.com 192.168.30.67 iapp6-lh iapp6-lh.example.domain.com 192.168.30.68 iapp7-lh iapp7-lh.example.domain.com 192.168.30.69 iapp8-lh iapp8-lh.example.domain.com 192.168.30.61 ipas1 192.168.30.62 ipas2
6 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
192.168.30.63 ipas3 192.168.30.64 ipas4 192.168.30.65 ipas-lh ipas-lh.example.domain.com
Step 2. Quorum device iSCSI LUN and NFS file systems An iSCSI LUN must be configured on the clustered storage for use as the quorum device. A quorum device is a shared storage device or quorum server that is shared by two or more nodes and that contributes votes used to establish a quorum and prevent partitions. This step describes the creation and the configuration of the Quorum device.
2a) Adding a Quorum device Execute the following commands on each Oracle Solaris 10 General Purpose domain: # iscsiadm modify discovery -s enable # iscsiadm list discovery Discovery: Static: enabled Send Targets: disabled iSNS: disabled # iscsiadm list initiator-node Initiator node name: iqn.1986-03.com.sun:01:00144ffae18a.4e5b7811 Initiator node alias: ovmt-1 Login Parameters (Default/Configured): Header Digest: NONE/- Data Digest: NONE/- Authentication Type: NONE RADIUS Server: NONE RADIUS access: unknown Tunable Parameters (Default/Configured): Session Login Response Time: 60/- Maximum Connection Retry Time: 180/- Login Retry Time Interval: 60/- Configured Sessions: 1
# hostname etc4-01-app1 # iscsiadm modify initiator-node -A `hostname`
# hostname etc4-02-app1 # iscsiadm modify initiator-node -N iqn.1986-03.com.sun:01:00144ffae18a.4e5b7812 - A `hostname`
# hostname etc4-03-app1 # iscsiadm modify initiator-node -N iqn.1986-03.com.sun:01:00144ffae18a.4e5b7813 - A `hostname`
# hostname etc4-04-app1 # iscsiadm modify initiator-node -N iqn.1986-03.com.sun:01:00144ffae18a.4e5b7814 - A `hostname` Check back on each node (example for 4 nodes):
7 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Node 1: # iscsiadm list initiator-node Initiator node name: iqn.1986-03.com.sun:01:00144ffae18a.4e5b7811 Initiator node alias: etc4-01-app1 Login Parameters (Default/Configured): Header Digest: NONE/- Data Digest: NONE/- Authentication Type: NONE RADIUS Server: NONE RADIUS access: unknown Tunable Parameters (Default/Configured): Session Login Response Time: 60/- Maximum Connection Retry Time: 180/- Login Retry Time Interval: 60/- Configured Sessions: 1
Node 2: # iscsiadm list initiator-node Initiator node name: iqn.1986-03.com.sun:01:00144ffae18a.4e5b7812 Initiator node alias: etc4-02-app1 Login Parameters (Default/Configured): Header Digest: NONE/- Data Digest: NONE/- Authentication Type: NONE RADIUS Server: NONE RADIUS access: unknown Tunable Parameters (Default/Configured): Session Login Response Time: 60/- Maximum Connection Retry Time: 180/- Login Retry Time Interval: 60/- Configured Sessions: 1
Node 3: # iscsiadm list initiator-node Initiator node name: iqn.1986-03.com.sun:01:00144ffae18a.4e5b7813 Initiator node alias: etc4-03-app1 Login Parameters (Default/Configured): Header Digest: NONE/- Data Digest: NONE/- Authentication Type: NONE RADIUS Server: NONE RADIUS access: unknown Tunable Parameters (Default/Configured): Session Login Response Time: 60/- Maximum Connection Retry Time: 180/- Login Retry Time Interval: 60/- Configured Sessions: 1
Node 4: # iscsiadm list initiator-node Initiator node name: iqn.1986-03.com.sun:01:00144ffae18a.4e5b7814 Initiator node alias: etc4-04-app1 Login Parameters (Default/Configured): Header Digest: NONE/- Data Digest: NONE/- Authentication Type: NONE RADIUS Server: NONE RADIUS access: unknown Tunable Parameters (Default/Configured): Session Login Response Time: 60/- Maximum Connection Retry Time: 180/- Login Retry Time Interval: 60/- Configured Sessions: 1
8 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
2b) Configuring the Quorum Device The iSCSI Quorum Device is configured using the Sun ZFS Storage Appliance graphical user interface. In the Sun ZFS Storage Appliance (ZFSSA) GUI: Go to 7320 SAN Configuration create iSCSI Target "QuorumTarget" choose ipmp1 interface drag and drop it to the iSCSI Target Group to create a new group name it "QuorumGroup" click “Apply”. Write down the Target IQN in our example: iqn.1986-03.com.sun:02:9ab50402-48ee-6c7f-a6fb-db2afe325e57 Select the Initiators headlight create the iSCSI Initiator using the cluster nodes initiator id from above name them with nodes IB subnet internal hostname or 10GbE hostname drag and drop it to the iSCSI Initiator Group to create a new group name it QuorumGroup and select both iSCSI Initiators previously created click “Apply”.
In the shares page Select Project create a new project "QuorumProject" in the app-pool pool. In that project, in the LUNs tab Create a new LUN of 1G size name it QuorumLUN select the drop down menus to put it in the created QuorumGroup Target Group and the QuorumGroup Initiator Group select also LU Number 0 and Fix LU number click “Apply”.
Execute the following on each node: # iscsiadm add static-config iqn.1986-03.com.sun:02:9ab50402-48ee-6c7f-a6fb- db2afe325e57,192.168.30.1
# iscsiadm list static-config # iscsiadm list target -S Target: iqn.1986-03.com.sun:02:9ab50402-48ee-6c7f-a6fb-db2afe325e57 Alias: QuorumTarget TPGT: 2 ISID: 4000002a0000 Connections: 1 LUN: 0 Vendor: SUN Product: ZFS Storage 7320
9 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
OS Device Name: /dev/rdsk/c1t600144F0A67C70FB00004F2CA5A70001d0s2
From one of the nodes use the command format to create a VTOC for the LUN # format
2c) Configure NFS file systems for SAP on the 7320c Using the web management interface of the ZFS Storage appliance, go to the Shares panel and perform the following: Create a new project named "sap" set its "Mountpoint" property to "/export/etc4" In that project, create the following file systems and verify that they have the corresponding mount points: NAME MOUNTPOINT oracle /export/etc4/oracle trans /export/etc4/trans PASshare /export/etc4/PASshare SCSshare /export/etc4/SCSshare sapmnt_QO1 /export/etc4/sapmnt_QO1 APPSshare_QO1 /export/etc4/APPSshare_QO1
Create a new project named "oradb" set its "Mountpoint" property to "/export/etc4" In that project, create the following file systems and verify that they have the corresponding mount points: NAME MOUNTPOINT saplog /export/etc4/saplog orainst /export/etc4/orainst sapdata /export/etc4/sapdata
Note: The above operations for iSCSI Quorum device and NFS file systems could also be performed using the ZFS Storage appliance command line interface.
Step 3. Oracle Solaris Cluster installation In this step we are going to install Oracle Solaris Cluster 3.3 5/11 Release (OSC3.3u1) for SPARC, as well as its latest associated patches and the Oracle External Proxy (OEP).
3a) Initial Steps The next tasks will consist of downloading Oracle Solaris Cluster 3.3u1 and latest patches from the Oracle Technology Network (OTN) and My Oracle Support (MOS). Since the steps will have to be performed on each node, it is recommended, and assumed that the files are copied to an NFS share, read-writable mounted on all Oracle Solaris 10 General Purpose domains. That NFS share could be created on the SPARC SuperCluster ZFS Storage Appliance. We will call the mounted location of that share
10 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
From one Oracle Solaris 10 General Purpose domain, perform the following: # mkdir
Download Oracle Solaris Cluster 3.3 5/11 Release (OSC3.3u1) for SPARC from OTN: http://www.oracle.com/technetwork/server-storage/solaris-cluster/downloads/index.html Save the downloaded file in
Download the following patches for OSC3.3u1 from MOS (support.oracle.com) into the directory
Download patch 12736304 (Generic Platform) from MOS (support.oracle.com) into the directory
11 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
# cd
Download from MOS (support.oracle.com) patch 13987101 for Oracle Solaris on SPARC in to the
3b) Installing Oracle Solaris Cluster From each node, run the JES installer: # pwd
Select “Oracle Solaris Core” “HA for Oracle” “Support for Oracle RAC” “HA for SAP liveCache” “HA for SAPDB” “HA for SAP Web Application Server” “HA for SAP” The summary should look like this: Product: Oracle Solaris Cluster Uninstall Location: /var/sadm/prod/SUNWentsyssc33u1 Space Required: 582.40 MB ------Java DB Java DB Server Java DB Client Oracle Solaris Cluster 3.3u1 Oracle Solaris Cluster Core Oracle Solaris Cluster Manager Oracle Solaris Cluster Agents 3.3u1 Oracle Solaris Cluster HA for Oracle Oracle Solaris Cluster Support for Oracle RAC Oracle Solaris Cluster HA for SAP liveCache Oracle Solaris Cluster HA for SAPDB Oracle Solaris Cluster HA for SAP Web Application Server Oracle Solaris Cluster HA for SAP
3c) Installing the latest patches Once the software installation has completed, install the latest patches: root@sup30: pwd
12 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
# patchadd -M . ./patch_order
3d) Installing the Oracle External Proxy root@sup30: pwd
The generic instructions on how this feature is installed and configured are provided in this file: root@sup30: ls Docs/oracle-external-proxy.pdf Docs/oracle-external-proxy.pdf
Install the Oracle External Proxy (OEP) package on each node: # pwd
3e) Add /usr/cluster/bin to /etc/profile Add /usr/cluster/bin to the PATH environment variable in /etc/profile
3f) Install the Sun ZFS Storage Appliance plugin for Oracle Solaris Cluster # cd
Step 4. Configuring the Oracle Solaris 10 GP domains into a cluster In this step we are going to configure the Oracle Solaris 10 General Purpose domains into a cluster. First we must identify and write down the cluster "transport adapters" by running these commands on the last node: # grep 8501 /etc/path_to_inst "/pci@700/pci@2/pci@0/pci@3/pciex15b3,673c@0/ibport@2,8501,ipib" 7 "ibd" # grep 8502 /etc/path_to_inst "/pci@600/pci@2/pci@0/pci@0/pciex15b3,673c@0/ibport@1,8502,ipib" 1 "ibd" ibd7 and ibd1 are respectively the first and second "transport adapters" to be used in the configuration. From etc4-04-app1 (the last node) # /usr/cluster/bin/scinstall
Use the typical options and provide the above adapters when asked.
13 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Once all the nodes have been rebooted by this configuration task, log back in and check: # clnode status
=== Cluster Nodes ===
--- Node Status ---
Node Name Status ------etc4-01-app1 Online etc4-02-app1 Online etc4-03-app1 Online etc4-04-app1 Online
# clq status
=== Cluster Quorum ===
--- Quorum Votes Summary from (latest node reconfiguration) ---
Needed Present Possible ------3 4 4
--- Quorum Votes by Node (current status) ---
Node Name Present Possible Status ------etc4-01-app1 1 1 Online etc4-02-app1 1 1 Online etc4-03-app1 1 1 Online etc4-04-app1 1 1 Online # clintr status
=== Cluster Transport Paths ===
Endpoint1 Endpoint2 Status ------etc4-01-app1:ibd7 etc4-04-app1:ibd1 Path online etc4-01-app1:ibd1 etc4-04-app1:ibd7 Path online etc4-01-app1:ibd7 etc4-02-app1:ibd7 Path online etc4-01-app1:ibd1 etc4-02-app1:ibd1 Path online etc4-01-app1:ibd7 etc4-03-app1:ibd1 Path online etc4-01-app1:ibd1 etc4-03-app1:ibd7 Path online etc4-02-app1:ibd7 etc4-03-app1:ibd1 Path online etc4-02-app1:ibd7 etc4-04-app1:ibd1 Path online etc4-02-app1:ibd1 etc4-03-app1:ibd7 Path online etc4-02-app1:ibd1 etc4-04-app1:ibd7 Path online etc4-03-app1:ibd1 etc4-04-app1:ibd1 Path online etc4-03-app1:ibd7 etc4-04-app1:ibd7 Path online # scdidadm -L 1 etc4-01-app1:/dev/rdsk/c1t5000C5004311C9AFd0 /dev/did/rdsk/d1 2 etc4-01-app1:/dev/rdsk/c1t5000C500430FCC5Fd0 /dev/did/rdsk/d2 3 etc4-01-app1:/dev/rdsk/c1t50015179596667C7d0 /dev/did/rdsk/d3 4 etc4-01-app1:/dev/rdsk/c1t5001517959697454d0 /dev/did/rdsk/d4 5 etc4-03-app1:/dev/rdsk/c1t600144F0A67C70FB00004F2CA5A70001d0 /dev/did/rdsk/d5 5 etc4-01-app1:/dev/rdsk/c1t600144F0A67C70FB00004F2CA5A70001d0 /dev/did/rdsk/d5 5 etc4-02-app1:/dev/rdsk/c1t600144F0A67C70FB00004F2CA5A70001d0 /dev/did/rdsk/d5 5 etc4-04-app1:/dev/rdsk/c1t600144F0A67C70FB00004F2CA5A70001d0 /dev/did/rdsk/d5 6 etc4-02-app1:/dev/rdsk/c1t50015179596332D6d0 /dev/did/rdsk/d6 7 etc4-02-app1:/dev/rdsk/c1t50015179596974D9d0 /dev/did/rdsk/d7 8 etc4-02-app1:/dev/rdsk/c1t5000C5004327733Fd0 /dev/did/rdsk/d8 9 etc4-02-app1:/dev/rdsk/c1t5000C50043276973d0 /dev/did/rdsk/d9 10 etc4-03-app1:/dev/rdsk/c1t5001517959697498d0 /dev/did/rdsk/d10 11 etc4-03-app1:/dev/rdsk/c1t5000C50043551B2Fd0 /dev/did/rdsk/d11
14 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
12 etc4-03-app1:/dev/rdsk/c1t5000C500432923B7d0 /dev/did/rdsk/d12 13 etc4-03-app1:/dev/rdsk/c1t500151795966904Fd0 /dev/did/rdsk/d13 14 etc4-04-app1:/dev/rdsk/c1t500151795966AFB4d0 /dev/did/rdsk/d14 15 etc4-04-app1:/dev/rdsk/c1t5000C50043276D6Bd0 /dev/did/rdsk/d15 16 etc4-04-app1:/dev/rdsk/c1t50015179596974CAd0 /dev/did/rdsk/d16 17 etc4-04-app1:/dev/rdsk/c1t5000C50043292B67d0 /dev/did/rdsk/d17
If the automatic quorum selection was not selected, add a quorum device now using the iSCSI LUN identified (as d5) above: # clq add d5 # clq status
=== Cluster Quorum ===
--- Quorum Votes Summary from (latest node reconfiguration) ---
Needed Present Possible ------4 7 7
--- Quorum Votes by Node (current status) ---
Node Name Present Possible Status ------etc4-01-app1 1 1 Online etc4-02-app1 1 1 Online etc4-03-app1 1 1 Online etc4-04-app1 1 1 Online
--- Quorum Votes by Device (current status) ---
Device Name Present Possible Status ------d5 3 3 Online
15 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Initial Zone Cluster Setup
Two types of clusters can be configured with the Oracle Solaris operating system: global clusters and zone clusters. Global cluster. The global cluster contains all global zones in a collection of Oracle Solaris hosts. Only the global zone in the global cluster has the privileges needed to perform administrative actions for a zone cluster. Zone cluster. A zone cluster is a virtual cluster composed of one or more non-global zones, which are all of zone brand type cluster. Each cluster node of a zone cluster resides on a different Oracle Solaris host. A zone cluster appears to applications as a cluster dedicated for those applications. This section provides the steps to create the zone clusters and prepare the zone cluster environment for the SAP installation.
Step 5. Creation of the zone clusters for SAP In this step we are going to prepare the storage and create the zone clusters.
5a) Prepare a dedicated ZFS file system for the zone clusters. Execute the following on each node: # zfs create -o mountpoint=/zones BIrpool/zones
# zfs list NAME USED AVAIL REFER MOUNTPOINT BIrpool 138G 136G 107K /BIrpool BIrpool/ROOT 24.5G 136G 31K legacy BIrpool/ROOT/s10s_u10wos_17b 24.5G 136G 24.5G / BIrpool/dump 64.0G 136G 64.0G - BIrpool/export 63K 136G 32K /export BIrpool/export/home 31K 136G 31K /export/home BIrpool/swap 49.5G 186G 16K - BIrpool/zones 31K 136G 31K /zones
5b) Zone clusters creation From node 1: sapdev-zc zone cluster # cat /var/tmp/sapdev-zc-sroot.conf create set zonepath=/zones/sapdev-zc set autoboot=true add sysid set root_password=f/ouCDtWx.rIA set name_service="NIS{domain_name=example.domain.com name_server=ns.example.domain.com (10.129.184.238)}" set nfs4_domain=dynamic set security_policy=NONE set system_locale=C set terminal=xterms set timezone=US/Pacific end add node
16 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
set physical-host=etc4-01-app1 set hostname=isapdev1 add net set address=isapdev1 set physical=ibd3 end add net set address=esapdev1 set physical=ixgbe0 end end add node set physical-host=etc4-02-app1 set hostname=isapdev2 add net set address=isapdev2 set physical=ibd3 end add net set address=esapdev2 set physical=ixgbe0 end end add node set physical-host=etc4-03-app1 set hostname=isapdev3 add net set address=isapdev3 set physical=ibd3 end add net set address=esapdev3 set physical=ixgbe0 end end add node set physical-host=etc4-04-app1 set hostname=isapdev4 add net set address=isapdev4 set physical=ibd3 end add net set address=esapdev4 set physical=ixgbe0 end end set limitpriv="default,proc_priocntl,proc_clock_highres" add net set address=esapdb-lh end add net set address=isapdb-lh end commit # clzc configure -f /var/tmp/sapdev-zc-sroot.conf sapdev-zc # time clzc install sapdev-zc Waiting for zone install commands to complete on all the nodes of the zone cluster "sapdev-zc"... real 3m51.11s user 1m2.43s sys 1m5.37s
# time clzc boot sapdev-zc Waiting for zone boot commands to complete on all the nodes of the zone cluster "sapdev-zc"... real 0m2.36s user 0m0.19s sys 0m0.04s
17 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
scs-zc zone cluster # cat /var/tmp/scs-zc-sroot.conf create set zonepath=/zones/scs-zc set autoboot=true add sysid set root_password=f/ouCDtWx.rIA set name_service="NIS{domain_name=example.domain.com name_server=ns.example.domain.com(10.129.184.238)}" set nfs4_domain=dynamic set security_policy=NONE set system_locale=C set terminal=xterms set timezone=US/Pacific end add node set physical-host=etc4-01-app1 set hostname=iscs1 add net set address=iscs1 set physical=ibd3 end add net set address=escs1 set physical=ixgbe0 end end add node set physical-host=etc4-02-app1 set hostname=iscs2 add net set address=iscs2 set physical=ibd3 end add net set address=escs2 set physical=ixgbe0 end end add node set physical-host=etc4-03-app1 set hostname=iscs3 add net set address=iscs3 set physical=ibd3 end add net set address=escs3 set physical=ixgbe0 end end add node set physical-host=etc4-04-app1 set hostname=iscs4 add net set address=iscs4 set physical=ibd3 end add net set address=escs4 set physical=ixgbe0 end end set limitpriv="default,proc_priocntl,proc_clock_highres" add net set address=escs-lh end
18 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
add net set address=iscs-lh end add net set address=erep-lh end add net set address=irep-lh end commit
# time clzc configure -f /var/tmp/scs-zc-sroot.conf scs-zc real 0m3.70s user 0m0.10s sys 0m0.07s # time clzc install scs-zc Waiting for zone install commands to complete on all the nodes of the zone cluster "scs-zc"... real 2m40.39s user 1m4.25s sys 1m1.82s # time clzc boot scs-zc Waiting for zone boot commands to complete on all the nodes of the zone cluster "scs-zc"... real 0m2.39s user 0m0.19s sys 0m0.04s apps-zc zone cluster # cat /var/tmp/apps-zc-sroot.conf create set zonepath=/zones/apps-zc set autoboot=true add sysid set root_password=f/ouCDtWx.rIA set name_service="NIS{domain_name=example.domain.com name_server=ns.example.domain.com(10.129.184.238)}" set nfs4_domain=dynamic set security_policy=NONE set system_locale=C set terminal=xterms set timezone=US/Pacific end add node set physical-host=etc4-01-app1 set hostname=iapps1 add net set address=iapps1 set physical=ibd3 end add net set address=eapps1 set physical=ixgbe0 end end add node set physical-host=etc4-02-app1 set hostname=iapps2 add net set address=iapps2 set physical=ibd3 end add net
19 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
set address=eapps2 set physical=ixgbe0 end end add node set physical-host=etc4-03-app1 set hostname=iapps3 add net set address=iapps3 set physical=ibd3 end add net set address=eapps3 set physical=ixgbe0 end end add node set physical-host=etc4-04-app1 set hostname=iapps4 add net set address=iapps4 set physical=ibd3 end add net set address=eapps4 set physical=ixgbe0 end end set limitpriv="default,proc_priocntl,proc_clock_highres" add net set address=eapp1-lh end add net set address=iapp1-lh end add net set address=eapp2-lh end add net set address=iapp2-lh end add net set address=eapp3-lh end add net set address=iapp3-lh end add net set address=eapp4-lh end add net set address=iapp4-lh end add net set address=eapp5-lh end add net set address=iapp5-lh end add net set address=eapp6-lh end add net set address=iapp6-lh end add net set address=eapp7-lh end add net set address=iapp7-lh end
20 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
add net set address=eapp8-lh end add net set address=iapp8-lh end commit
# time clzc configure -f /var/tmp/apps-zc-sroot.conf apps-zc real 0m3.75s user 0m0.11s sys 0m0.07s # time clzc install apps-zc Waiting for zone install commands to complete on all the nodes of the zone cluster "apps-zc"... real 2m43.14s user 1m4.84s sys 1m2.02s # time clzc boot apps-zc Waiting for zone boot commands to complete on all the nodes of the zone cluster "apps-zc"... real 0m2.81s user 0m0.20s sys 0m0.04s pas-zc zone cluster # cat /var/tmp/pas-zc-sroot.conf create set zonepath=/zones/pas-zc set autoboot=true add sysid set root_password=f/ouCDtWx.rIA set name_service="NIS{domain_name=example.domain.com name_server=ns.example.domain.com(10.129.184.238)}" set nfs4_domain=dynamic set security_policy=NONE set system_locale=C set terminal=xterms set timezone=US/Pacific end add node set physical-host=etc4-01-app1 set hostname=ipas1 add net set address=ipas1 set physical=ibd3 end add net set address=epas1 set physical=ixgbe0 end end add node set physical-host=etc4-02-app1 set hostname=ipas2 add net set address=ipas2 set physical=ibd3 end add net set address=epas2 set physical=ixgbe0 end
21 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
end add node set physical-host=etc4-03-app1 set hostname=ipas3 add net set address=ipas3 set physical=ibd3 end add net set address=epas3 set physical=ixgbe0 end end add node set physical-host=etc4-04-app1 set hostname=ipas4 add net set address=ipas4 set physical=ibd3 end add net set address=epas4 set physical=ixgbe0 end end set limitpriv="default,proc_priocntl,proc_clock_highres" add net set address=epas-lh end add net set address=ipas-lh end commit
# clzc configure -f /var/tmp/pas-zc-sroot.conf pas-zc # time clzc install pas-zc # time clzc boot pas-zc
# clzc status
=== Zone Clusters ===
--- Zone Cluster Status ---
Name Node Name Zone HostName Status Zone Status ------sapdev-zc etc4-01-app1 isapdev1 Online Running etc4-02-app1 isapdev2 Online Running etc4-03-app1 isapdev3 Online Running etc4-04-app1 isapdev4 Online Running scs-zc etc4-01-app1 iscs1 Online Running etc4-02-app1 iscs2 Online Running etc4-03-app1 iscs3 Online Running etc4-04-app1 iscs4 Online Running apps-zc etc4-01-app1 iapps1 Online Running etc4-02-app1 iapps2 Online Running etc4-03-app1 iapps3 Online Running etc4-04-app1 iapps4 Online Running pas-zc etc4-01-app1 ipas1 Online Running etc4-02-app1 ipas2 Online Running etc4-03-app1 ipas3 Online Running etc4-04-app1 ipas4 Online Running
22 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Step 6. Preparing the zone cluster environment On each node, edit the profile file for each zone cluster: /zones/*/root/etc/profile PATH=$PATH:/usr/cluster/bin
/zones/*-zc/root/etc/ssh/sshd_config #PermitRootLogin no PermitRootLogin yes
# for zc in `clzc list`; do zlogin $zc svcadm restart ssh; done
Add to the zone cluster hosts file all relevant entries in the global zone's hosts file (see step 1 for complete list and recommendations). /zones/*/root/etc/hosts
# cat > /var/tmp/zc-hosts # hostnames on the 10GbE network 10.129.184.150 esapdev1 ... # hostnames on the IPoIB network 192.168.30.1 etc4sn1-stor ... 192.168.30.65 ipas-lh ^D
# cat /var/tmp/zc-hosts >> /zones/sapdev-zc/root/etc/hosts # cat /var/tmp/zc-hosts >> /zones/scs-zc/root/etc/hosts # cat /var/tmp/zc-hosts >> /zones/apps-zc/root/etc/hosts # cat /var/tmp/zc-hosts >> /zones/pas-zc/root/etc/hosts
Step 7. SPARC SuperCluster specific tuning In /etc/system * SSC tuning for SAP set nfs:nfs3_bsize = 131072 set nfs:nfs4_bsize = 131072 set rpcmod:clnt_max_conns = 8 set zfs:zfs_arc_max = 0x1000000000 set rlim_fd_cur=8192 set pg_contig_disable=1
Note: set rlim_fd_cur=8192 (parameter can be deleted after SAP installation) # cat > /etc/rc2.d/S99ndd ndd -set /dev/tcp tcp_xmit_hiwat 1048576 ndd -set /dev/tcp tcp_recv_hiwat 1048576 ndd -set /dev/tcp tcp_cwnd_max 2097152 ndd -set /dev/tcp tcp_max_buf 2097152 # chmod a+rx /etc/rc2.d/S99ndd
Step 8. Creating initial resource groups and logical host resources In this step we create the initial resource groups and the associated logical host resources for the zone clusters.
23 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
8a) scs-zc zone cluster # zlogin scs-zc [Connected to zone 'scs-zc' pts/3] Last login: Sat Feb 4 17:02:48 on console Oracle Corporation SunOS 5.10 Generic Patch January 2005 # ksh -o emacs # clrg create scs-rg # clrslh create -g scs-rg escs-lh # clrslh create -g scs-rg iscs-lh # clrg online -eM scs-rg # clrg create rep-rg # clrslh create -g rep-rg erep-lh # clrslh create -g rep-rg irep-lh # clrg online -eM rep-rg
Note: Once central services are installed and running, additional affinities will have to be configured.
8b) pas-zc zone cluster # zlogin pas-zc [Connected to zone 'pas-zc' pts/3] Last login: Sat Feb 4 17:27:15 on console Oracle Corporation SunOS 5.10 Generic Patch January 2005 # ksh -o emacs # clrg create pas-rg # clrslh create -g pas-rg epas-lh # clrslh create -g pas-rg ipas-lh # clrg online -eM pas-rg # clrs status
=== Cluster Resources ===
Resource Name Node Name State Status Message ------ipas-lh ipas1 Online Online - LogicalHostname online. ipas2 Offline Offline ipas3 Offline Offline ipas4 Offline Offline epas-lh ipas1 Online Online - LogicalHostname online. ipas2 Offline Offline ipas3 Offline Offline ipas4 Offline Offline
8c) apps-zc zone cluster # zlogin apps-zc [Connected to zone 'apps-zc' pts/3] Last login: Sat Feb 4 17:14:16 on console Oracle Corporation SunOS 5.10 Generic Patch January 2005 # ksh -o emacs # clrg create -p failback=true -n 1,2,3,4 D51-rg # clrg create -p failback=true -n 2,3,4,1 D52-rg # clrg create -p failback=true -n 3,4,1,2 D53-rg # clrg create -p failback=true -n 4,1,2,3 D54-rg # clrg create -p failback=true -n 1,2,3,4 D55-rg # clrg create -p failback=true -n 2,3,4,1 D56-rg # clrg create -p failback=true -n 3,4,1,2 D57-rg # clrg create -p failback=true -n 4,1,2,3 D58-rg # clrslh create -g D51-rg eapp1-lh # clrslh create -g D52-rg eapp2-lh # clrslh create -g D53-rg eapp3-lh # clrslh create -g D54-rg eapp4-lh
24 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
# clrslh create -g D55-rg eapp5-lh # clrslh create -g D56-rg eapp6-lh # clrslh create -g D57-rg eapp7-lh # clrslh create -g D58-rg eapp8-lh # clrslh create -g D51-rg iapp1-lh # clrslh create -g D52-rg iapp2-lh # clrslh create -g D53-rg iapp3-lh # clrslh create -g D54-rg iapp4-lh # clrslh create -g D55-rg iapp5-lh # clrslh create -g D56-rg iapp6-lh # clrslh create -g D57-rg iapp7-lh # clrslh create -g D58-rg iapp8-lh # clrg online -eM +
8d) sapdev-zc zone cluster # zlogin sapdev-zc # ksh -o emacs # clrg create sapdb-rg # clrslh create -g sapdb-rg esapdb-lh # clrslh create -g sapdb-rg isapdb-lh # clrg online -eM sapdb-rg
Step 9. Preparing the NFS file systems for Zone Clusters The SAP installer and runtime procedures are all hard-coded to use /usr/sap. Zone clusters are created as sparse-root zones, to minimize the local storage requirement for the zone path and to speed up the zone cluster installation by reducing the number of files to copy. For these reasons, /usr is a read-only file system in the zone cluster, i.e. the user cannot create any new sub directories under /usr. Additionally, /opt is on a private writable file system in the zone cluster. Therefore a symbolic link from /usr/sap to /opt/sap will fulfill both requirements: SAP sees /usr/sap as expected, and from inside the zone cluster, the user can create any new sub directories under /usr/sap. On all nodes, create this symbolic link in the global zone: # ln -s /opt/sap /usr/sap
Log in to each zone cluster to create a mount point and add entries in /etc/vfstab
9a) sapdev-zc zone cluster # zlogin sapdev-zc # mkdir -p /oracle /sapmnt/QO1 /opt/sap/trans # mkdir /orainst /sapdata /saplog
Add to /etc/vfstab: # SAP file systems etc4sn1-stor:/export/etc4/oracle - /oracle nfs - yes rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3,noac,suid etc4sn1-stor:/export/etc4/sapmnt_QO1 - /sapmnt/QO1 nfs - yes rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3
25 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
etc4sn1-stor:/export/etc4/trans - /opt/sap/trans nfs - yes rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3 etc4sn1-stor:/export/etc4/orainst - /orainst nfs - yes rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3,noac,suid etc4sn1-stor:/export/etc4/sapdata - /sapdata nfs - yes rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3,noac,suid,forcedirect io etc4sn1-stor:/export/etc4/saplog - /saplog nfs - yes rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3,noac,suid,forcedirect io
# mountall # df -h -F nfs Filesystem size used avail capacity Mounted on etc4sn1-stor:/export/etc4/orainst 16T 31K 16T 1% /orainst etc4sn1-stor:/export/etc4/oracle 16T 31K 16T 1% /oracle etc4sn1-stor:/export/etc4/sapmnt_QO1 16T 31K 16T 1% /sapmnt/QO1 etc4sn1-stor:/export/etc4/sapdata 16T 31K 16T 1% /sapdata etc4sn1-stor:/export/etc4/trans 16T 31K 16T 1% /opt/sap/trans etc4sn1-stor:/export/etc4/saplog 16T 31K 16T 1% /saplog
9b) scs-zc zone cluster # zlogin scs-zc [Connected to zone 'scs-zc' pts/4] Last login: Sat Feb 4 19:33:25 on pts/3 Oracle Corporation SunOS 5.10 Generic Patch January 2005 # ksh -o emacs # mkdir -p /oracle /sapmnt/QO1 /opt/sap
Add to /etc/vfstab: # SAP file systems etc4sn1-stor:/export/etc4/oracle - /oracle nfs - no rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3,noac,suid etc4sn1-stor:/export/etc4/sapmnt_QO1 - /sapmnt/QO1 nfs - no rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3 etc4sn1-stor:/export/etc4/SCSshare - /opt/sap nfs - no rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3 etc4sn1-stor:/export/etc4/trans - /opt/sap/trans nfs - no rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3
# mount /oracle /sapmnt/QO1 /opt/sap
# mkdir /opt/sap/trans # mount /opt/sap/trans
# df -h -F nfs Filesystem size used avail capacity Mounted on etc4sn1-stor:/export/etc4/oracle 16T 31K 16T 1% /oracle etc4sn1-stor:/export/etc4/sapmnt_QO1 16T 31K 16T 1% /sapmnt/QO1 etc4sn1-stor:/export/etc4/SCSshare 16T 31K 16T 1% /opt/sap etc4sn1-stor:/export/etc4/trans
26 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
16T 31K 16T 1% /opt/sap/trans
9c) pas-zc zone cluster # zlogin pas-zc [Connected to zone 'pas-zc' pts/4] Last login: Sat Feb 4 21:28:16 on pts/3 Oracle Corporation SunOS 5.10 Generic Patch January 2005 # ksh -o emacs # mkdir -p /oracle /sapmnt/QO1 /opt/sap
Add to /etc/vfstab: # SAP file systems etc4sn1-stor:/export/etc4/oracle - /oracle nfs - no rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3,noac,suid etc4sn1-stor:/export/etc4/sapmnt_QO1 - /sapmnt/QO1 nfs - no rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3 etc4sn1-stor:/export/etc4/PASshare - /opt/sap nfs - no rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3 etc4sn1-stor:/export/etc4/trans - /opt/sap/trans nfs - no rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3
# mount /oracle /sapmnt/QO1 /opt/sap
# mkdir /opt/sap/trans # mount /opt/sap/trans
# df -h -F nfs Filesystem size used avail capacity Mounted on etc4sn1-stor:/export/etc4/oracle 16T 31K 16T 1% /oracle etc4sn1-stor:/export/etc4/sapmnt_QO1 16T 31K 16T 1% /sapmnt/QO1 etc4sn1-stor:/export/etc4/PASshare 16T 31K 16T 1% /opt/sap etc4sn1-stor:/export/etc4/trans 16T 31K 16T 1% /opt/sap/trans
9d) apps-zc zone cluster # zlogin apps-zc [Connected to zone 'apps-zc' pts/4] Last login: Sat Feb 4 19:40:37 on pts/3 Oracle Corporation SunOS 5.10 Generic Patch January 2005 # ksh -o emacs # mkdir -p /oracle /sapmnt/QO1 /opt/sap
Add to /etc/vfstab: # SAP file systems etc4sn1-stor:/export/etc4/oracle - /oracle nfs - no rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3,noac,suid etc4sn1-stor:/export/etc4/sapmnt_QO1 - /sapmnt/QO1 nfs - no rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3 etc4sn1-stor:/export/etc4/APPSshare_QO1 - /opt/sap nfs - no rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3 etc4sn1-stor:/export/etc4/trans - /opt/sap/trans nfs - no rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3
27 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
# mount /oracle /sapmnt/QO1 /opt/sap
# mkdir /opt/sap/trans # mount /opt/sap/trans
# df -h -F nfs Filesystem size used avail capacity Mounted on etc4sn1-stor:/export/etc4/oracle 16T 31K 16T 1% /oracle etc4sn1-stor:/export/etc4/sapmnt_QO1 16T 31K 16T 1% /sapmnt/QO1 etc4sn1-stor:/export/etc4/APPSshare_QO1 16T 31K 16T 1% /opt/sap etc4sn1-stor:/export/etc4/trans 16T 31K 16T 1% /opt/sap/trans
Step 10. Configuring the plug-in for Oracle Solaris Cluster fencing and creating storage resources Configure the Sun ZFS Server Appliance plug-in for Oracle Solaris Cluster fencing and create the storage resources oracle> ssh -l root etc4-sn1 Password: Last login: Mon Mar 5 21:58:50 2012 from 10.129.205.70 etc4-sn1:> shares etc4-sn1:shares> select sap etc4-sn1:shares sap> ls Properties: aclinherit = restricted aclmode = discard atime = true checksum = fletcher4 compression = off dedup = false compressratio = 100 copies = 1 creation = Sun Feb 05 2012 05:34:58 GMT+0000 (UTC) logbias = latency mountpoint = /export/etc4 quota = 0 readonly = false recordsize = 128K reservation = 0 rstchown = true secondarycache = all nbmand = false sharesmb = off sharenfs = sec=sys,rw,[email protected]/24 snapdir = hidden vscan = false sharedav = off shareftp = off sharesftp = off sharetftp = pool = ssc canonical_name = ssc/local/sap default_group = other default_permissions = 700 default_sparse = false default_user = nobody default_volblocksize = 8K default_volsize = 0 exported = true nodestroy = false space_data = 373G
28 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
space_unused_res = 0 space_unused_res_shares = 0 space_snapshots = 0 space_available = 15.6T space_total = 373G origin =
Shares:
Filesystems:
NAME SIZE MOUNTPOINT oracle 31K /export/etc4/oracle trans 31K /export/etc4/trans PASshare 31K /export/etc4/PASshare SCSshare 31K /export/etc4/SCSshare sapmnt_QO1 31K /export/etc4/sapmnt_QO1 APPSshare_QO1 31K /export/etc4/APPSshare_QO1
Children: groups => View per-group usage and manage group quotas replication => Manage remote replication snapshots => Manage snapshots users => View per-user usage and manage user quotas etc4-sn1:shares sap> get sharenfs sharenfs = sec=sys,rw,[email protected]/24 etc4-sn1:shares sap> get readonly readonly = false
Identify and write down the IPoIB zone cluster hostnames: 192.168.30.41 iscs1 192.168.30.42 iscs2 192.168.30.43 iscs3 192.168.30.44 iscs4 192.168.30.51 iapps1 192.168.30.52 iapps2 192.168.30.53 iapps3 192.168.30.54 iapps4 192.168.30.61 ipas1 192.168.30.62 ipas2 192.168.30.63 ipas3 192.168.30.64 ipas4 etc4-sn1:shares sap> set sharenfs="sec=sys,[email protected]/24:@192.168.30.41/32:@192.168.30.42/32:@192.1 68.30.43/32:@192.168.30.44/32:@192.168.30.51/32:@192.168.30.52/32:@192.168.30.53/3 2:@192.168.30.54/32:@192.168.30.61/32:@192.168.30.62/32:@192.168.30.63/32:@192.168 .30.64/32,[email protected]/24:@192.168.30.41/32:@192.168.30.42/32:@192.168.30.43/3 2:@192.168.30.44/32:@192.168.30.51/32:@192.168.30.52/32:@192.168.30.53/32:@192.168 .30.54/32:@192.168.30.61/32:@192.168.30.62/32:@192.168.30.63/32:@192.168.30.64/32" sharenfs = sec=sys,[email protected]/24:@192.168.30.41/32:@192.168.30.42/32:@192.168.30.43/3 2:@192.168.30.44/32:@192.168.30.51/32:@192.168.30.52/32:@192.168.30.53/32:@192.168 .30.54/32:@192.168.30.61/32:@192.168.30.62/32:@192.168.30.63/32:@192.168.30.64/32, [email protected]/24:@192.168.30.41/32:@192.168.30.42/32:@192.168.30.43/32:@192.168 .30.44/32:@192.168.30.51/32:@192.168.30.52/32:@192.168.30.53/32:@192.168.30.54/32: @192.168.30.61/32:@192.168.30.62/32:@192.168.30.63/32:@192.168.30.64/32 (uncommitted) etc4-sn1:shares sap> commit etc4-sn1:shares sap> get sharenfs
29 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
sharenfs = sec=sys,[email protected]/24:@192.168.30.41/32:@192.168.30.42/32:@192.168.30.43/3 2:@192.168.30.44/32:@192.168.30.51/32:@192.168.30.52/32:@192.168.30.53/32:@192.168 .30.54/32:@192.168.30.61/32:@192.168.30.62/32:@192.168.30.63/32:@192.168.30.64/32, [email protected]/24:@192.168.30.41/32:@192.168.30.42/32:@192.168.30.43/32:@192.168 .30.44/32:@192.168.30.51/32:@192.168.30.52/32:@192.168.30.53/32:@192.168.30.54/32: @192.168.30.61/32:@192.168.30.62/32:@192.168.30.63/32:@192.168.30.64/32 etc4-sn1:shares sap> get readonly readonly = false etc4-sn1:shares sap> cd ../.. etc4-sn1:> maintenance workflows etc4-sn1:maintenance workflows> ls Properties: showhidden = false
Workflows:
WORKFLOW NAME OWNER SETID ORIGIN workflow-000 Configure for Oracle Solaris Cluster NFS root false Oracle Corporation workflow-001 Unconfigure Oracle Solaris Cluster NFS root false Oracle Corporation workflow-002 Configure for Oracle Enterprise Manager Monitoring root false Sun Microsystems, Inc. workflow-003 Unconfigure Oracle Enterprise Manager Monitoring root false Sun Microsystems, Inc. etc4-sn1:maintenance workflows> select workflow-000 etc4-sn1:maintenance workflow-000> execute etc4-sn1:maintenance workflow-000 execute (uncommitted)> set password=welcome1 password = ******** etc4-sn1:maintenance workflow-000 execute (uncommitted)> set changePassword=false changePassword = false etc4-sn1:maintenance workflow-000 execute (uncommitted)> commit OSC configuration successfully completed. etc4-sn1:maintenance workflow-000> ls Properties: name = Configure for Oracle Solaris Cluster NFS description = Sets up environment for Oracle Solaris Cluster NFS owner = root origin = Oracle Corporation setid = false alert = false scheduled = false etc4-sn1:maintenance workflow-000> cd ../.. etc4-sn1:maintenance> cd .. etc4-sn1:> configuration users etc4-sn1:configuration users> show Users:
NAME USERNAME UID TYPE Oracle Solaris Cluster Agent osc_agent 2000000000 Loc Super-User root 0 Loc
10a) scs-zc zone cluster # zlogin scs-zc [Connected to zone 'scs-zc' pts/22] Last login: Wed Mar 21 07:50:43 from ipas1 Oracle Corporation SunOS 5.10 Generic Patch January 2005
30 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
# ksh -o emacs # getent hosts `clnode list` 192.168.30.41 iscs1 192.168.30.42 iscs2 192.168.30.43 iscs3 192.168.30.44 iscs4 # clnas add -t sun_uss -p userid=osc_agent etc4sn1-stor Enter password: (welcome1)
# clnas set -p "nodeIPs{iscs1}"=192.168.30.41 \ -p "nodeIPs{iscs2}"=192.168.30.42 \ -p "nodeIPs{iscs3}"=192.168.30.43 \ -p "nodeIPs{iscs4}"=192.168.30.44 etc4sn1-stor # clnas find-dir etc4sn1-stor
=== NAS Devices ===
Nas Device: etc4sn1-stor Type: sun_uss Unconfigured Project: ssc/local/QuorumProject Unconfigured Project: ssc/local/oradb Unconfigured Project: ssc/local/sap
# clnas add-dir -d ssc/local/sap etc4sn1-stor # clnas show -v -d all
=== NAS Devices ===
Nas Device: etc4sn1-stor Type: sun_uss userid: osc_agent nodeIPs{iscs4}: 192.168.30.44 nodeIPs{iscs3}: 192.168.30.43 nodeIPs{iscs2}: 192.168.30.42 nodeIPs{iscs1}: 192.168.30.41 Project: ssc/local/sap File System: /export/etc4/oracle File System: /export/etc4/SCSshare File System: /export/etc4/PASshare File System: /export/etc4/sapmnt_QO1 File System: /export/etc4/trans File System: /export/etc4/APPSshare_QO1
# clrg create -S scalmnt-rg # df -h -F nfs Filesystem size used avail capacity Mounted on etc4sn1-stor:/export/etc4/oracle 16T 31K 16T 1% /oracle etc4sn1-stor:/export/etc4/sapmnt_QO1 16T 31K 16T 1% /sapmnt/QO1 etc4sn1-stor:/export/etc4/SCSshare 16T 31K 16T 1% /opt/sap etc4sn1-stor:/export/etc4/trans 16T 31K 16T 1% /opt/sap/trans
# tail -5 /etc/vfstab # SAP file systems etc4sn1-stor:/export/etc4/oracle - /oracle nfs - no rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3,noac,suid etc4sn1-stor:/export/etc4/sapmnt_QO1 - /sapmnt/QO1 nfs - no rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3 etc4sn1-stor:/export/etc4/SCSshare - /opt/sap nfs - no rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3
31 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
etc4sn1-stor:/export/etc4/trans - /opt/sap/trans nfs - no rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3
Mount at boot set to "no" # clrt register ScalMountPoint # clrs create -d -g scalmnt-rg -t ScalMountPoint -x MountPointDir=/oracle \ -x FileSystemType=nas \ -x TargetFileSystem=etc4sn1-stor:/export/etc4/oracle ora-fs-rs
# clrs create -d -g scalmnt-rg -t ScalMountPoint -x MountPointDir=/sapmnt/QO1 \ -x FileSystemType=nas \ -x TargetFileSystem=etc4sn1-stor:/export/etc4/sapmnt_QO1 sapmnt-fs-rs
# clrs create -d -g scalmnt-rg -t ScalMountPoint -x MountPointDir=/opt/sap \ -x FileSystemType=nas \ -x TargetFileSystem=etc4sn1-stor:/export/etc4/SCSshare usrsap-fs-rs # clrs create -d -g scalmnt-rg -t ScalMountPoint \ -x MountPointDir=/opt/sap/trans -x FileSystemType=nas \ -x TargetFileSystem=etc4sn1-stor:/export/etc4/trans \ -p Resource_dependencies_offline_restart=usrsap-fs-rs trans-fs-rs # clrg online -eM scalmnt-rg # clrs status -g scalmnt-rg
=== Cluster Resources ===
Resource Name Node Name State Status Message ------trans-fs-rs iscs1 Online Online iscs2 Online Online iscs3 Online Online iscs4 Online Online usrsap-fs-rs iscs1 Online Online iscs2 Online Online iscs3 Online Online iscs4 Online Online sapmnt-fs-rs iscs1 Online Online iscs2 Online Online iscs3 Online Online iscs4 Online Online ora-fs-rs iscs1 Online Online iscs2 Online Online iscs3 Online Online iscs4 Online Online
10b) pas-zc zone cluster # zlogin pas-zc [Connected to zone 'pas-zc' pts/22] Last login: Sun Mar 25 15:39:58 on pts/9 Oracle Corporation SunOS 5.10 Generic Patch January 2005 You have new mail. # ksh -o emacs # getent hosts `clnode list` 192.168.30.61 ipas1 loghost 192.168.30.62 ipas2 192.168.30.63 ipas3 192.168.30.64 ipas4 # clnas add -t sun_uss -p userid=osc_agent etc4sn1-stor Enter password: (welcome1)
32 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
# clnas set \ -p "nodeIPs{ipas1}"=192.168.30.61 \ -p "nodeIPs{ipas2}"=192.168.30.62 \ -p "nodeIPs{ipas3}"=192.168.30.63 \ -p "nodeIPs{ipas4}"=192.168.30.64 etc4sn1-stor # clnas find-dir etc4sn1-stor
=== NAS Devices ===
Nas Device: etc4sn1-stor Type: sun_uss Unconfigured Project: ssc/local/QuorumProject Unconfigured Project: ssc/local/oradb Unconfigured Project: ssc/local/sap
# clnas add-dir -d ssc/local/sap etc4sn1-stor # clnas show -v -d all
# clrg create -S scalmnt-rg # df -h -F nfs Filesystem size used avail capacity Mounted on etc4sn1-stor:/export/etc4/oracle 16T 31K 16T 1% /oracle etc4sn1-stor:/export/etc4/sapmnt_QO1 16T 31K 16T 1% /sapmnt/QO1 etc4sn1-stor:/export/etc4/PASshare 16T 31K 16T 1% /opt/sap etc4sn1-stor:/export/etc4/trans 16T 31K 16T 1% /opt/sap/trans
# tail -5 /etc/vfstab # SAP file systems etc4sn1-stor:/export/etc4/oracle - /oracle nfs - no rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3,noac,suid etc4sn1-stor:/export/etc4/sapmnt_QO1 - /sapmnt/QO1 nfs - no rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3 etc4sn1-stor:/export/etc4/PASshare - /opt/sap nfs - no rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3 etc4sn1-stor:/export/etc4/trans - /opt/sap/trans nfs - no rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3
=> Mount at boot set to "no" # clrt register ScalMountPoint
# clrs create -d -g scalmnt-rg -t ScalMountPoint -x MountPointDir=/oracle \ -x FileSystemType=nas \ -x TargetFileSystem=etc4sn1-stor:/export/etc4/oracle ora-fs-rs # clrs create -d -g scalmnt-rg -t ScalMountPoint -x MountPointDir=/sapmnt/QO1 \ -x FileSystemType=nas \ -x TargetFileSystem=etc4sn1-stor:/export/etc4/sapmnt_QO1 sapmnt-fs-rs # clrs create -d -g scalmnt-rg -t ScalMountPoint -x MountPointDir=/opt/sap \ -x FileSystemType=nas \ -x TargetFileSystem=etc4sn1-stor:/export/etc4/PASshare usrsap-fs-rs # clrs create -d -g scalmnt-rg -t ScalMountPoint \ -x MountPointDir=/opt/sap/trans -x FileSystemType=nas \ -x TargetFileSystem=etc4sn1-stor:/export/etc4/trans \ -p Resource_dependencies_offline_restart=usrsap-fs-rs trans-fs-rs # clrg online -eM scalmnt-rg # clrs status -g scalmnt-rg
=== Cluster Resources ===
Resource Name Node Name State Status Message ------
33 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
trans-fs-rs ipas1 Online Online ipas2 Online Online ipas3 Online Online ipas4 Online Online usrsap-fs-rs ipas1 Online Online ipas2 Online Online ipas3 Online Online ipas4 Online Online sapmnt-fs-rs ipas1 Online Online ipas2 Online Online ipas3 Online Online ipas4 Online Online ora-fs-rs ipas1 Online Online ipas2 Online Online ipas3 Online Online ipas4 Online Online
# clrg status
=== Cluster Resource Groups ===
Group Name Node Name Suspended Status ------pas-rg ipas1 No Offline ipas2 No Offline ipas3 No Online ipas4 No Offline scalmnt-rg ipas1 No Online ipas2 No Online ipas3 No Online ipas4 No Online
10c) apps-zc zone cluster # zlogin apps-zc [Connected to zone 'apps-zc' pts/22] Last login: Sun Mar 25 15:52:12 on pts/9 Oracle Corporation SunOS 5.10 Generic Patch January 2005 # ksh -o emacs # getent hosts `clnode list` 192.168.30.51 iapps1 192.168.30.52 iapps2 192.168.30.53 iapps3 192.168.30.54 iapps4 # clnas add -t sun_uss -p userid=osc_agent etc4sn1-stor Enter password: # clnas set \ -p "nodeIPs{iapps1}"=192.168.30.51 \ -p "nodeIPs{iapps2}"=192.168.30.52 \ -p "nodeIPs{iapps3}"=192.168.30.53 \ -p "nodeIPs{iapps4}"=192.168.30.54 etc4sn1-stor # clnas find-dir etc4sn1-stor
=== NAS Devices ===
Nas Device: etc4sn1-stor Type: sun_uss Unconfigured Project: ssc/local/QuorumProject Unconfigured Project: ssc/local/oradb Unconfigured Project: ssc/local/sap
# clnas add-dir -d ssc/local/sap etc4sn1-stor # clnas show -v -d all
# clrg create -S scalmnt-rg
34 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
# df -h -F nfs Filesystem size used avail capacity Mounted on etc4sn1-stor:/export/etc4/oracle 16T 31K 16T 1% /oracle etc4sn1-stor:/export/etc4/sapmnt_QO1 16T 31K 16T 1% /sapmnt/QO1 etc4sn1-stor:/export/etc4/APPSshare_QO1 16T 31K 16T 1% /opt/sap etc4sn1-stor:/export/etc4/trans 16T 31K 16T 1% /opt/sap/trans
# tail -5 /etc/vfstab # SAP file systems etc4sn1-stor:/export/etc4/oracle - /oracle nfs - no rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3,noac,suid etc4sn1-stor:/export/etc4/sapmnt_QO1 - /sapmnt nfs - no rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3 etc4sn1-stor:/export/etc4/APPSshare_QO1 - /opt/sap nfs - no rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3 etc4sn1-stor:/export/etc4/trans - /opt/sap/trans nfs - no rw,bg,hard,nointr,rsize=131072,wsize=131072,proto=tcp,vers=3
Mount at boot set to "no" # clrt register ScalMountPoint
# clrs create -d -g scalmnt-rg -t ScalMountPoint -x MountPointDir=/oracle \ -x FileSystemType=nas \ -x TargetFileSystem=etc4sn1-stor:/export/etc4/oracle ora-fs-rs # clrs create -d -g scalmnt-rg -t ScalMountPoint -x MountPointDir=/sapmnt/QO1 \ -x FileSystemType=nas \ -x TargetFileSystem=etc4sn1-stor:/export/etc4/sapmnt_QO1 sapmnt-fs-rs # clrs create -d -g scalmnt-rg -t ScalMountPoint \ -x MountPointDir=/opt/sap/trans -x FileSystemType=nas \ -x TargetFileSystem=etc4sn1-stor:/export/etc4/trans trans-fs-rs # clrs create -d -g scalmnt-rg -t ScalMountPoint \ -x MountPointDir=/opt/sap -x FileSystemType=nas \ -x TargetFileSystem=etc4sn1-stor:/export/etc4/APPSshare_QO1 apps-fs-rs -p Resource_dependencies_offline_restart=usrsap-fs-rs trans-fs-rs # clrg online -eM scalmnt-rg # clrs status -g scalmnt-rg
=== Cluster Resources ===
Resource Name Node Name State Status Message ------trans-fs-rs iapps1 Online Online iapps2 Online Online iapps3 Online Online iapps4 Online Online sapmnt-fs-rs iapps1 Online Online iapps2 Online Online iapps3 Online Online iapps4 Online Online ora-fs-rs iapps1 Online Online iapps2 Online Online iapps3 Online Online iapps4 Online Online apps-fs-rs iapps1 Online Online iapps2 Online Online iapps3 Online Online iapps4 Online Online
35 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
SAP Setup
Step 11. Installing SAP This section provides the steps to download, install and configure SAP ERP 6.0 Enhancement Package 5 on SPARC SuperCluster. Important: The following instructions are provided as example. For the sake of simplicity, default values were accepted for most of the inputs during the installation. Appropriate values and selections should be used instead to match specific configuration requirements. Regularly refer to SAP Note 1693680 “Running SAP Software on Oracle SPARC SuperCluster” and to SAP Note 724713 “Parameter Settings for Solaris 10” to get the latest information. Before installing the SAP software, take the logical hosts up but not DEPRECATED, there are issues with some versions of sapinst. Use the command “ifconfig –a” to double-check. If it needs to be changed, logon to the global zone and run the command ifconfig
After the SAP installation, the logical hosts can be set back to deprecated as required by Oracle Solaris Cluster. After the SAP installation, verify that all the files and environment are identical on the zones in the same zone cluster.
11a) Requirements Prior to starting the installation some Oracle Solaris groups and users need to be created and parameters and resources need to be set.
Add Oracle Solaris groups and users for the SAP installation groupadd -g 1000 sapinst groupadd -g 1001 sapsys groupadd -g 1002 dba groupadd -g 1003 oper useradd -c "SAP System Administrator" -d "/export/home/qo1adm" -u 1000 -s "/bin/tcsh" -g sapsys -G dba,oper,sapinst -m qo1adm useradd -c "SAP Database Administrator" -d "/oracle/QO1" -u 1001 -s "/bin/tcsh" -g dba -G sapinst,oper -m oraqo1 useradd -c "SAP System Administrator" -d "/export/home/sapadm" -u 1002 -s "/bin/tcsh" -g sapsys -G sapinst -m sapadm useradd -c "SAP System Administrator" -d /export/home/daaadm -u 1004 -s "/bin/tcsh" -g sapsys -G sapinst -m daaadm
Set Oracle Solaris Parameters Set the parameter according to SAP Note 724713.
36 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
If running Oracle Solaris 10 Update 10 (Solaris 10 8/11), do not set the "priv" level for process.max-file-descriptor. Use these following kernel parameters instead: set pg_contig_disable=1
Add the project for SAP SID QO1: projadd -p 222 -c "SAP System QO1" \ -U qo1adm,oraqo1,sapadm,daaadm \ -K "process.max-file-descriptor=(basic,65536,deny)" \ -K "process.max-sem-nsems=(priv,2048,deny)" \ -K "project.max-sem-ids=(priv,1024,deny)" \ -K "project.max-shm-ids=(priv,256,deny)" \ -K "project.max-shm-memory=(priv,18446744073709551615,deny)" QO1
In order to define the default project for the SAP system and the database administrator, add the following lines to file /etc/user_attr: qo1adm::::project=QO1 oraqo1::::project=QO1 sapadm::::project=QO1 daaadm::::project=QO1
Set the resources for root Before installing SAP, set resource settings for user root: projmod -s \ -K "process.max-file-descriptor=(basic,65536,deny),(priv,65536,deny)" \ -K "process.max-sem-nsems=(priv,2048,deny)" \ -K "project.max-sem-ids=(priv,1024,deny)" \ -K "project.max-shm-ids=(priv,256,deny)" \ -K "project.max-shm-memory=(priv,18446744073709551615,deny)" \ user.root
As mentioned previously, when using Oracle Solaris 10 Update 10, do not use the priv level for process.max-file-descriptor. After installing SAP, these resource settings for user root can be removed (default settings will apply afterwards): projmod -r \ -K "process.max-file-descriptor=(basic,65536,deny),(priv,65536,deny)" \ -K "process.max-sem-nsems=(priv,2048,deny)" \ -K "project.max-sem-ids=(priv,1024,deny)" \ -K "project.max-shm-ids=(priv,256,deny)" \ -K "project.max-shm-memory=(priv,18446744073709551615,deny)"\ user.root
11b) Required software
Download SAP ERP 6.0 Enhancement Package 5 Download SAP ERP software from the SAP Software Download Center:
37 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
From the URL http://service.sap.com/swdc go to: Installation and Upgrades → A-Z Index → E → SAP ERP → SAP ERP Enhance Package → EHP5 for SAP ERP 6.0 → Installation → DB System based on → ORACLE Database → based on SAP Netweaver 7.3 Download the following software: NW 7.3 Java based SW Comp.s Enablmt. (1 to 6) BS7i2010 Inst.mst. - NW 7.3 Enablmt. (1 to 8) BS7 UC-Kernel 7.20 Solaris on SPARC 64bit BS7i2010 Java - Enablement f. NW 7.3 (1 to 3)
Download the Oracle database If using a ASFU license, download the Oracle Database from the SAP Software Download Center: From the URL http://service.sap.com/oracle-download go to: Oracle 11.2 Software (64-bit) → Installation → Solaris → ORACLE Database Download the following software: ORACLE RDBMS 11.2.0.3 SOL. SPARC 64 (1 to 6) ORACLE Client 11.2.0.3
Installation locations Install the software in the order shown below:
Preinstalled Software in the Database Tier (Database Domain) Oracle Database 11g Release 2 Server In the Application Tier (Application Domain – Application Zone Cluster) SAP ERP 6.0 Enhancement Package 5 The present Installation Master DVD does not support the installation of the Oracle Client 11.2.x.x. As a workaround SAP provides a tool (adaptDvd.sh) to adapt the Installation Master, compare SAP Note 1642058. The tool is attached to the SAP Note. Copy the Installation Master DVD to a local folder and apply the tool as following: # sh adaptDvd.sh
The present kernel DVD does not install the 720_EXT kernel that we chose to use in our example. As a workaround SAP provided SAP Note 1696716 – Creating a modified Solaris kernel medium. Download the latest 720_EXT kernel files from he SAP Software Download Center.
38 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
From the URL http://service.sap.com/swdc go to: Support Packages and Patches → A-Z Index → SAP Kernel 64-Bit Unicode → SAP Kernel 720 EXT 64-Bit UC → Solaris on SPARC 64bit → ORACLE Download the following files: SAPEXEDB
From → #Database Independent Download the following file: SAPEXE
11c) SAP Installation The option “High-Availability System” is required to install the following components on multiple logical hosts: ASCS Instance Database Instance Enqueue Replication Server Instance Central Instance Dialog Instance In the following screenshots the SAP System ID “ERP” is used, the same steps should be followed using ID “QO1”.
Preparation Log on as user root in a zone that belongs to scs-zc (i. e. escs1) Set environment variable DISPLAY Make sure that the logical host escs-lh is running in this zone. If not, start it with cluster command. Make sure that the logical host escs-lh is not DEPRECATED, as there are known issues with some releases of sapinst. Use the command “ifconfig –a” to double-check. If it needs to be changed, logon to the global zone and run the command ifconfig
39 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Installation Start the SAP Installer SAPINST in the directory for the Solaris SPARC OS on the Installation Master DVD. The logical hostname needs to be specified as a suffix: # sapinst SAPINST_USE_HOSTNAME=escs-lh
Select the following option: Enhancement Package 5 for SAP ERP 6.0 → SAP Application Server ABAP → Oracle → High-Availibility System → ASCS Instance
Select “ASCS Instance”
40 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Define the SAP system parameters: Choose the SAP system ID.
Enter the domain name if the installer is not able to detect it. The domain name should be set in the local file /etc/hosts prior to the installation.
41 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Enter the master password for all relevant accounts on operating system and application level.
The prerequisite checker shows at least these two conditions that can be ignored. The parameter rlim_fd_cur was already set as a project parameter but the prerequisite checker expects it explicitly in the file /etc/system. Click “Next”.
42 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Click “Cancel” to continue the installation without repeating the checks.
Indicate the location of the kernel DVD.
43 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
The ASCS instance will be deployed by default as with instance number “00”.
Ensure that the ports to be set for the message server are not already used by another application.
44 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
This option allows installing the SAP cryptographic library if needed. The files can be found on http://service.sap.com/swdc
Leave the defaults to unpack the archive to the proposed destination.
45 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
This summary lists the parameters provided. A parameter can be selected and modified if needed. Click “Next” to start the installation.
After a successful installation a confirmation message box is displayed. Click “OK”.
46 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
11d) Installation of the Oracle 11.2.0.3 Database software
Preparation Log on as user root in a zone that belongs to sapdev-zc, (i.e. esapdev1). The Oracle Database will be temporarily installed in this zone during the SAP installation. After the whole SAP system is installed, the Oracle Database will be migrated to Oracle RAC and ASM in its own Oracle Solaris 11 LDOM. Set the environment variable DISPLAY Make sure that the logical host esapdb-lh is running in this zone. If not, start it with cluster command. Make sure that the logical host esapdb-lh is not DEPRECATED, as there are known issues with some releases of sapinst. Use the command “ifconfig –a” to double-check. If it needs to be changed, logon to the global zone and run the command ifconfig
Installation Start the SAP Installer SAPINST in the directory for the Solaris SPARC OS on the Installation Master DVD. The logical hostname needs to be specified as a suffix: # sapinst SAPINST_USE_HOSTNAME=esapdb-lh
Select the following option: Enhancement Package 5 for SAP ERP 6.0 → SAP Application Server ABAP → Oracle → High-Availibility System → Database Instance
47 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Indicate the location of the installed Java 1.4.2 instance.
Use the default profile directory.
48 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Enter the master password for all users.
Enter the database ID and esapdb-lh as the database host.
49 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Review the results of the prerequisite checker.
Click “Cancel” to continue the installation.
50 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Indicate the location of the kernel DVD.
Indicate the location of the DVD: Installation Export 1 ECC 6.0 EHP5.
51 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Indicate the location of the DVD: Installation Export 2 ECC 6.0 EHP5.
Enter the amount of instance memory for the Oracle database.
52 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Select DB version “112” and “64” bit for DB server and client.
Set the passwords for the Oracle database users.
53 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Indicate the location of the Oracle database RDBMS DVD.
The defaults for the Oracle listener can be accepted if port 1527 is available.
54 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Accept the defaults for the Database load parameters.
Select “Create statistics at the end of the import” and do not change the entry for Program Call.
55 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Indicate the location for the Oracle client DVD.
Leave the defaults to unpack the archives.
56 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
This summary lists the parameters provided. A parameter can be selected and modified if needed. Click “Next” to start the installation.
The installer prepares the Oracle database installation.
57 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
The installer stops for the installation of the Oracle database. Follow the instructions in the message box to start the Oracle “RUNINSTALLER”.
58 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Installation of the Oracle database
Insert a valid email address and your password for “My Oracle Support” to receive communications about security updates.
59 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Select an option to download software updates. In our example we choose "Skip software updates".
Use the default option “Install database software only”.
60 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Use the default option “Single instance database installation”.
Install additional languages if necessary.
61 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Use the default to install the Enterprise Edition of the Oracle Database.
Keep the default installation locations.
62 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Leave the defaults for the inventory directory.
Click “Yes” to confirm the warning to install the inventory directory in /oracle.
63 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Keep the default group names.
During the installation, the warnings related to the Oracle Solaris kernel parameters can be ignored.
64 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Click “Install” to start the installation process.
Installation process.
65 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Follow the instructions to execute configuration scripts as the root user.
After a successful installation, the message “The installation of Oracle Database was successful” appears. Click “Close” to complete the installation process.
66 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
After installing the Oracle Database software, proceed with the database instance installation by choosing “OK” in the dialog box.
After a successful installation a confirmation message box is displayed. Click “OK”.
67 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
11e) Installation of the Enqueue Replication Server Instance The next step is the installation of the Enqueue Replication Server Instance. Logon to the host erep-lh and start the SAP Installer SAPINST in the directory for the Solaris SPARC OS on the Installation Master DVD. The logical hostname needs to be specified as a suffix: # sapinst SAPINST_USE_HOSTNAME=erep-lh
Select the following option: Enhancement Package 5 for SAP ERP 6.0 → SAP Application Server ABAP → Oracle → High-Availibility System → Enqueue Replication Server Instance.
68 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
SAPINST sets the appropriate ownership permissions for the work directory. Accept the changes.
SAPINST sets the appropriate access permissions for the work directory. Accept the changes.
69 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Keep the default location for the SAP system profile directory.
Select the existing SAP system central services instance.
70 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Indicate the location of the kernel DVD.
Select the number of the ERS instance. The default can be accepted.
71 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Check the box “Get the (A)SCS Instance Restarted”.
This summary lists the parameters provided. A parameter can be selected and modified if needed. Click “Next” to start the installation.
72 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
After a successful installation a confirmation message box is displayed. Click “OK”.
11f) Installation of the Central Instance The next step is the installation of the Central Instance. Log on as user root in the host epas-lh and make sure it is not DEPRECATED. Set the environment variable DISPLAY and start the SAP installer in the directory for the Solaris SPARC OS on the Installation Master DVD. The logical hostname needs to be specified as a suffix: # sapinst SAPINST_USE_HOSTNAME=epas-lh
73 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Select the following option: Enhancement Package 5 for SAP ERP 6.0 → SAP Application Server ABAP → Oracle → High-Availibility System → Central Instance.
SAPINST sets the appropriate ownership permissions for the work directory. Accept the changes.
74 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
SAPINST sets the appropriate access permissions for the work directory. Accept the changes.
Choose the directory of the installed Java 1.4.2 instance.
75 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Keep the default profile directory.
Enter the master password for all relevant accounts on operating system and application level.
76 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Review the results of the prerequisite checker.
Click “Cancel” to continue the installation without repeating the checks.
77 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Indicate the location of the kernel DVD.
Make sure that the ports selected for the Oracle listener are not already used by other applications.
78 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Leave the default parameter for the central instance.
Indicate whether the DDIC user has a password different from default.
79 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Indicate the location of the Oracle client DVD.
Leave the defaults to unpack the archives.
80 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Choose the ID for the Diagnostics Agent. The default “DAA” can be accepted.
Indicate the location of the jce policy file.
81 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Leave the default for the instance number of the Diagnostics Agent.
In our example we do not integrate our system in an existing central SLD.
82 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Leave the defaults to unpack the archives.
This summary lists the parameters provided. A parameter can be selected and modified if needed. Click “Next” to start the installation.
83 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Enter the SAP Solution Manager key for your SAP system.
After a successful installation a confirmation message box is displayed. Click “OK”.
11g) Installation of the dialog instances The next step is the installation of the dialog instance(s).
84 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Log on as user root in one of the logical host for the dialog instance, for example eapp1-lh. Make sure that it is not DEPRECATED. Set the environment variable DISPLAY. Start the SAP Installer SAPINST in the directory for the Solaris SPARC OS on the Installation Master DVD. The logical hostname needs to be specified as a suffix: # sapinst SAPINST_USE_HOSTNAME=eapp1-lh
Select the following option: Enhancement Package 5 for SAP ERP 6.0 → SAP Application Server ABAP → Oracle → High-Availibility System → Dialog Instance.
85 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
SAPINST sets the appropriate ownership permissions for the work directory. Accept the changes.
SAPINST sets the appropriate access permissions for the work directory. Accept the changes.
86 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Leave the default location for the profile directory.
Enter the master password for all users.
87 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
The prerequisite checker shows at least these two conditions, which can be ignored. The parameter rlim_fd_cur was already set as a project parameter but the prerequisite checker expects it explicitly in the file /etc/system. Click “Next”.
Click “Cancel” to continue the installation without repeating the checks.
88 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Indicate the location of the kernel DVD.
Make sure that the ports selected for the Oracle listener are not already used by other applications.
89 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
The dialog instance will be deployed by default as Instance Number “02”.
Indicate the location of the Oracle client DVD.
90 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Leave the defaults to unpack the archives.
Finally the installer shows the parameter summary screen. Click “Next” to start the installation.
91 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
After a successful installation a confirmation message box is displayed. Click “OK”. If necessary, repeat the last installation steps to install additional dialog instances. After the SAP installation, ensure that all the files and environment are identical on the zones in the same zone cluster. The cluster console program (package SUNWccon or SUNWcluster) can be used to control multiple terminals simultaneously.
Step 12. Database Migration to Exadata Storage During the previous sapinst installation, the Oracle Database was installed on to the ZFS Storage Appliance. To take advantage of the database acceleration from the Exadata Storage Servers, the database now needs to be moved over to the Exadata Storage Servers. Note: A newer version of the sapinst installer will allow installing SAP with the Database directly on the Exadata Storage Servers, eliminating this extra step. This version was not available at the time this solution was developed.
12a) Migration of the Database from ZFS Storage Appliance to Exadata Storage Nodes
Definitions: original (source) database: SID QO1, service name QO1 duplicated (target) database: SID QO1, service name QO1DUP
92 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Set the environment Logon to host etc4-cn01 as user oracle and set the environment. Example for bash shell: oracle@etc4-cn01:$ export ORACLE_SID=+ASM1 oracle@etc4-cn01:$ export ORACLE_HOME=/u01/app/11.2.0.2/grid oracle@etc4-cn01:$ export PATH=.:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/u01/app/oracle/product/11.2 .0.2/dbhome_1/bin
Password file creation Configuration of a password file for the database instance. oracle@etc4-cn01:$ cd /u01/app/oracle/product/11.2.0.2/dbhome_1/dbs oracle@etc4-cn01:$ orapwd file=orapwQO1 entries=10 password=
SQL*Net configuration on sapdb-lh SQL*Net configuration in source system sapdb-lh listener.ora: no additional entries are necessary ipas1:/oracle/QO1/11203/network/admin# cat listener.ora
################ # Filename...... : listener.ora # Created...... : created by SAP AG, R/3 Rel. >= 6.10 # Name...... : # Date...... : # @(#) $Id: //bas/720_REL/src/krn/tpls/ora/LISTENER.ORA#1 $ ################ ADMIN_RESTRICTIONS_LISTENER = on LISTENER = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = IPC) (KEY = QO1.WORLD) ) (ADDRESS= (PROTOCOL = IPC) (KEY = QO1) ) (ADDRESS = (COMMUNITY = SAP.WORLD) (PROTOCOL = TCP) (HOST = esapdb-lh) (PORT = 1527) ) ) STARTUP_WAIT_TIME_LISTENER = 0 CONNECT_TIMEOUT_LISTENER = 10 TRACE_LEVEL_LISTENER = OFF SID_LIST_LISTENER = (SID_LIST = (SID_DESC = (SID_NAME = QO1) (ORACLE_HOME = /oracle/QO1/112_64) tnsnames.ora: enter the information for the target database (service name QO1DUP) ipas1:/oracle/QO1/11203/network/admin# cat tnsnames.ora
################
93 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
# Filename...... : tnsnames.ora # Created...... : created by SAP AG, R/3 Rel. >= 6.10 # Name...... : # Date...... : # @(#) $Id: //bas/720_REL/src/krn/tpls/ora/TNSNAMES.ORA#1 $ ################ QO1.WORLD= (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (COMMUNITY = SAP.WORLD) (PROTOCOL = TCP) (HOST = esapdb-lh) (PORT = 1527) ) ) (CONNECT_DATA = (SID = QO1) (GLOBAL_NAME = QO1.WORLD) ) ) QO1DUP.WORLD = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (COMMUNITY = SAP.WORLD) (PROTOCOL = TCP) (HOST = 10.129.184.15) (PORT = 1521)) ) (CONNECT_DATA = (SID = QO1) (GLOBAL_NAME = QO1.WORLD) ) ) sqlnet.ora: ipas1:/oracle/QO1/11203/network/admin# cat sqlnet.ora
################ # Filename...... : sqlnet.ora # Created...... : created by SAP AG, R/3 Rel. >= 6.10 # Name...... : # Date...... : # @(#) $Id: //bas/720_REL/src/krn/tpls/ora/SQLNET.ORA#1 $ ################ AUTOMATIC_IPC = ON TRACE_LEVEL_CLIENT = OFF NAMES.DIRECTORY_PATH = (TNSNAMES,EZCONNECT) NAMES.DEFAULT_DOMAIN = WORLD # 05.01.06 unsorported parameter now #NAME.DEFAULT_ZONE = WORLD # 05.01.06 set the default to 10 SQLNET.EXPIRE_TIME = 10 # 05.01.06 set to default #TCP.NODELAY=YES # 05.01.06 set to 32768 DEFAULT_SDU_SIZE=32768
SQL*Net configuration on etc4-cn01 SQL*Net configuration in target system etc4-cn01 listener.ora: enter SID QO1
94 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
oracle@etc4-cn01:$ cd /u01/app/11.2.0.2/grid/network/admin oracle@etc4-cn01:$ cat listener.ora
LISTENER_IB=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_IB)))) # line added by Agent LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))) # line added by Agent LISTENER_SCAN3=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCA N3)))) # line added by Agent LISTENER_SCAN2=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCA N2)))) # line added by Agent LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCA N1)))) # line added by Agent ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON # line added by Agent ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN2=ON # line added by Agent ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN3=ON # line added by Agent ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON # line added by Agent ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_IB=ON # line added by Agent ADMIN_RESTRICTION_LISTENER=ON # # # SID_LIST_LISTENER = (SID_LIST = (SID_DESC = (SID_NAME = QO1) (ORACLE_HOME = /u01/app/oracle/product/11.2.0.2/dbhome_1 ) ) ) tnsnames.ora: enter SID QO1 oracle@etc4-cn01:$ cd /u01/app/oracle/product/11.2.0.2/dbhome_1/network/admin oracle@etc4-cn01:$ cat tnsnames.ora
# tnsnames.ora Network Configuration File: /u01/app/oracle/product/11.2.0.2/dbhome_1/network/admin/tnsnames.ora # Generated by Oracle configuration tools.
+ASM1 = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (COMMUNITY = SAP.WORLD)(PROTOCOL = TCP)(HOST = 10.129.184.15)(PORT = 1521)) ) (CONNECT_DATA = (SID = +ASM1) (GLOBAL_NAME = +ASM1) ) )
QO1 = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (COMMUNITY = SAP.WORLD)(PROTOCOL = TCP)(HOST = esapdb-lh)(PORT = 1521)) ) (CONNECT_DATA = (SID = QO1) (GLOBAL_NAME = QO1) ) )
QO1DUP =
95 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
(DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (COMMUNITY = SAP.WORLD)(PROTOCOL = TCP)(HOST = 10.129.184.15)(PORT = 1521)) ) (CONNECT_DATA = (SID = QO1) (GLOBAL_NAME = QO1) ) )
DBM = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = etc4-scan)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = dbm) ) ) ipas1:/oracle/QO1/11203/network/admin# cat sqlnet.ora ################ # Filename...... : sqlnet.ora # Created...... : created by SAP AG, R/3 Rel. >= 6.10 # Name...... : # Date...... : # @(#) $Id: //bas/720_REL/src/krn/tpls/ora/SQLNET.ORA#1 $ ################ AUTOMATIC_IPC = ON TRACE_LEVEL_CLIENT = OFF NAMES.DIRECTORY_PATH = (TNSNAMES,EZCONNECT) NAMES.DEFAULT_DOMAIN = WORLD # 05.01.06 unsorported parameter now #NAME.DEFAULT_ZONE = WORLD # 05.01.06 set the default to 10 SQLNET.EXPIRE_TIME = 10 # 05.01.06 set to default #TCP.NODELAY=YES # 05.01.06 set to 32768 DEFAULT_SDU_SIZE=32768 sqlnet.ora: oracle@etc4-cn01:$ cd /u01/app/11.2.0.2/grid/network/admin oracle@etc4-cn01:$ cat sqlnet.ora
# sqlnet.ora.etc4-cn01 Network Configuration File: /u01/app/11.2.0.2/grid/network/admin/sqlnet.ora.etc4-cn01 # Generated by Oracle configuration tools.
NAMES.DIRECTORY_PATH= (TNSNAMES, EZCONNECT) ADR_BASE = /u01/app/oracle SQLNET.EXPIRE_TIME=10 NAMES.DEFAULT_DOMAIN = WORLD AUTOMATIC_IPC = ON
Target system setup Add the groups oper, asmoper, asmadmin, asmdba and add user oracle to these groups on all db nodes: root@etc4-cn01:# groupadd -g 703 oper root@etc4-cn01:# groupadd -g 705 asmoper root@etc4-cn01:# groupadd -g 706 asmadmin root@etc4-cn01:# groupadd -g 707 asmdba root@etc4-cn01:# usermod -G +oper,asmoper,asmadmin,asmdba oracle
96 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Create a shell script user “oracle” can source for each Oracle Home that has been installed. Environment for the ASM instance and to run GRID commands: export ORACLE_SID=+ASM1 export ORACLE_HOME=/u01/app/11.2.0.2/grid export PATH=.:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/u01/app/11.2.0.2/grid/bin
Environment for the new database instance: export ORACLE_SID=QO1 export ORACLE_HOME=/u01/app/oracle/product/11.2.0.2/dbhome_1 export PATH=.:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/u01/app/oracle/product/11.2 .0.2/dbhome_1/bin
Create a file initQO1.ora on the target system List the logfiles on the source system: logon to host esapdb-lh as user oraqo1 ierppas1:oraqo1 2% sqlplus / as sysdba
SQL> select member from v$logfile;
MEMBER ------/oracle/QO1/origlogA/log_g11m3.dbf /oracle/QO1/mirrlogA/log_g11m4.dbf /oracle/QO1/origlogB/log_g12m3.dbf /oracle/QO1/mirrlogB/log_g12m4.dbf /oracle/QO1/origlogA/log_g13m3.dbf /oracle/QO1/mirrlogA/log_g13m4.dbf /oracle/QO1/origlogB/log_g14m3.dbf /oracle/QO1/mirrlogB/log_g14m4.dbf Full path of existing logfiles need to be added to pfile initQO1.ora oracle@etc4-cn01:$ cd /u01/app/oracle/product/11.2.0.2/dbhome_1/dbs oracle@etc4-cn01:$ cat initQO1.ora
*.db_name='QO1' *.db_block_size=8192 *.compatible=11.2.0.2 *.audit_trail=none *.log_archive_dest_1='location=+RECO_ETC4/QO1 valid_for=(ALL_LOGFILES,ALL_ROLES) db_unique_name=QO1' db_unique_name='QO1' *.db_create_file_dest='+DATA_ETC4' *.db_create_online_log_dest_1='+DATA_ETC4' *.db_create_online_log_dest_2='+RECO_ETC4' *.log_file_name_convert= '/oracle/QO1/origlogA/log_g11m3.dbf','+DATA_ETC4', '/oracle/QO1/mirrlogA/log_g11m4.dbf','+RECO_ETC4', '/oracle/QO1/origlogB/log_g12m3.dbf','+DATA_ETC4', '/oracle/QO1/mirrlogB/log_g12m4.dbf','+RECO_ETC4', '/oracle/QO1/origlogA/log_g13m3.dbf','+DATA_ETC4', '/oracle/QO1/mirrlogA/log_g13m4.dbf','+RECO_ETC4', '/oracle/QO1/origlogB/log_g14m3.dbf','+DATA_ETC4', '/oracle/QO1/mirrlogB/log_g14m4.dbf','+RECO_ETC4' *.control_files='+DATA_ETC4/QO1/CONTROLFILE/cntrl1.dbf','+RECO_ETC4/QO1/CONTROLFIL E/cntrl2.dbf' *.db_recovery_file_dest='+RECO_ETC4' *.db_recovery_file_dest_size=4G
97 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
*.shared_pool_size=2181038080
Start the target instance into nomount state Logon to the host esapdb-lh as user oraqo1 and connect to the target database: isapdb1:oraqo1 25% sqlplus sys/
SQL*Plus: Release 11.2.0.3.0 Production on Thu Jul 12 13:41:47 2012
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to an idle instance. SQL> startup nomount ORACLE instance started.
Total System Global Area 2941173760 bytes Fixed Size 2162544 bytes Variable Size 2248147088 bytes Database Buffers 536870912 bytes Redo Buffers 153993216 bytes
Database Duplication using RMAN Use Oracle Recovery Manager (RMAN) to duplicate the database from the active database. The source database must be in "mount" or "open" state. Logon to the host esapdb-lh as user oraqo1: isapdb1:oraqo1 42% rman nocatalog
Recovery Manager: Release 11.2.0.3.0 - Production on Thu Jul 12 14:18:52 2012
Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
RMAN> connect target sys/
RMAN> connect auxiliary sys/
RMAN> run { 2> ALLOCATE CHANNEL t1 DEVICE TYPE disk; 3> ALLOCATE CHANNEL t2 DEVICE TYPE disk; 4> ALLOCATE CHANNEL t3 DEVICE TYPE disk; 5> ALLOCATE CHANNEL t4 DEVICE TYPE disk; 6> ALLOCATE CHANNEL t5 DEVICE TYPE disk; 7> ALLOCATE CHANNEL t6 DEVICE TYPE disk; 8> ALLOCATE CHANNEL t7 DEVICE TYPE disk; 9> ALLOCATE CHANNEL t8 DEVICE TYPE disk; 10> ALLOCATE AUXILIARY CHANNEL a1 DEVICE TYPE disk; 11> duplicate target database 12> to QO1 13> from active database 14> nofilenamecheck; 15> } using target database control file instead of recovery catalog allocated channel: t1 channel t1: SID=8 device type=DISK allocated channel: t2
98 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
channel t2: SID=579 device type=DISK ...... datafile 34 switched to datafile copy input datafile copy RECID=33 STAMP=792746259 file name=+DATA_ETC4/qo1/datafile/psapsr3usr.600.792746055 contents of Memory Script: { Alter clone database open resetlogs; } executing Memory Script database opened Finished Duplicate Db at 12-JUL-12 released channel: t1 released channel: t2 released channel: t3 released channel: t4 released channel: t5 released channel: t6 released channel: t7 released channel: t8 released channel: a1
Post duplication tasks Ensure that all files are consistent and no blocks are corrupted: RMAN> validate database;
Starting validate at 10-JUL-12 allocated channel: ORA_DISK_1 channel ORA_DISK_1: SID=246 device type=DISK channel ORA_DISK_1: starting validation of datafile channel ORA_DISK_1: specifying datafile(s) for validation ...... channel ORA_DISK_1: starting validation of datafile channel ORA_DISK_1: specifying datafile(s) for validation including current control file for validation including current SPFILE in backup set channel ORA_DISK_1: validation complete, elapsed time: 00:00:02 List of Control File and SPFILE ======File Type Status Blocks Failing Blocks Examined ------SPFILE OK 0 2 Control File OK 0 1672 Finished validate at 10-JUL-12
Check that the files were copied to the correct ASM disk group: oracle@etc4-cn01:~$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.3.0 Production on Mon Sep 3 07:13:42 2012
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP, Data Mining and Real Application Testing options
99 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
SQL> select name from v$datafile;
NAME ------+DATA_ETC4/qo1/datafile/system.285.775883053 +DATA_ETC4/qo1/datafile/sysaux.283.775883075 +DATA_ETC4/qo1/datafile/sysaux.addbypaul +DATA_ETC4/qo1/datafile/psapsr3.301.775882811 +DATA_ETC4/qo1/datafile/psapsr3.300.775882825 +DATA_ETC4/qo1/datafile/psapsr3.299.775882841 +DATA_ETC4/qo1/datafile/psapsr3.298.775882855 +DATA_ETC4/qo1/datafile/psapsr3.297.775882871 +DATA_ETC4/qo1/datafile/psapsr3.296.775882887 +DATA_ETC4/qo1/datafile/psapsr3.295.775882901 +DATA_ETC4/qo1/datafile/psapsr3.294.775882917
NAME ------+DATA_ETC4/qo1/datafile/psapsr3.293.775882931 +DATA_ETC4/qo1/datafile/psapsr3.292.775882947 +DATA_ETC4/qo1/datafile/psapsr3.291.775882963 +DATA_ETC4/qo1/datafile/psapsr3.290.775882977 +DATA_ETC4/qo1/datafile/psapsr3.289.775882993 +DATA_ETC4/qo1/datafile/psapsr3.288.775883007 +DATA_ETC4/qo1/datafile/psapsr3.287.775883023 +DATA_ETC4/qo1/datafile/psapsr3.284.775883069 +DATA_ETC4/qo1/datafile/psapsr3702.313.775882437 +DATA_ETC4/qo1/datafile/psapsr3702.312.775882473 +DATA_ETC4/qo1/datafile/psapsr3702.311.775882507
NAME ------+DATA_ETC4/qo1/datafile/psapsr3702.310.775882543 +DATA_ETC4/qo1/datafile/psapsr3702.308.775882603 +DATA_ETC4/qo1/datafile/psapsr3702.307.775882639 +DATA_ETC4/qo1/datafile/psapsr3702.306.775882663 +DATA_ETC4/qo1/datafile/psapsr3702.314.775882403 +DATA_ETC4/qo1/datafile/psapsr3702.315.775882367 +DATA_ETC4/qo1/datafile/psapsr3702.309.775882579 +DATA_ETC4/qo1/datafile/psapsr3702.305.775882699 +DATA_ETC4/qo1/datafile/psapsr3702.304.775882725 +DATA_ETC4/qo1/datafile/psapsr3702.303.775882759 +DATA_ETC4/qo1/datafile/psapsr3702.302.775882785
NAME ------+DATA_ETC4/qo1/datafile/psapsr3usr.282.775883083 +DATA_ETC4/qo1/datafile/psapundo_002 +DATA_ETC4/qo1/datafile/psapundo_001 +RECO_ETC4/qo1/datafile/psapundo_003 +RECO_ETC4/qo1/datafile/psapundo_004 +DATA_ETC4/qo1/datafile/sysaux.addbypaul2
39 rows selected.
SQL> select name from v$controlfile;
NAME ------+DATA_ETC4/qo1/controlfile/cntrl1.dbf +RECO_ETC4/qo1/controlfile/cntrl2.dbf
SQL> select member from v$logfile;
MEMBER ------+DATA_ETC4/qo1/onlinelog/group_8.604.792746389 +RECO_ETC4/qo1/onlinelog/group_8.437.792746409
100 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
+DATA_ETC4/qo1/onlinelog/group_7.603.792746349 +RECO_ETC4/qo1/onlinelog/group_7.436.792746369 +DATA_ETC4/qo1/onlinelog/group_6.602.792746309 +RECO_ETC4/qo1/onlinelog/group_6.435.792746327 +DATA_ETC4/qo1/onlinelog/group_5.601.792746267 +RECO_ETC4/qo1/onlinelog/group_5.434.792746287
8 rows selected.
12b) Enable Oracle RAC
Additional database configuration Create undo tablespace: logon to db node 1 (etc4-cn01) as user oracle and set the environment for SID QO1. oracle@etc4-cn01:$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.3.0 Production on Fri Aug 31 05:29:24 2012
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,Data Mining and Real Application Testing options
SQL> create undo tablespace psapundo_001 datafile '+DATA_ETC4/qo1/datafile/psapundo_001' size 10000m reuse; SQL> create undo tablespace psapundo_002 datafile '+DATA_ETC4/qo1/datafile/psapundo_002' size 10000m reuse; SQL> create undo tablespace psapundo_003 datafile '+RECO_ETC4/qo1/datafile/psapundo_003' size 10000m reuse; SQL> create undo tablespace psapundo_004 datafile '+RECO_ETC4/qo1/datafile/psapundo_004' size 10000m reuse;
Logfile configuration for 4 RAC nodes: SQL> alter database add logfile thread 1 group 11 ('+DATA_ETC4/QO1/log1_g11_m1.dbf') size 4000m reuse; SQL> alter database add logfile thread 1 group 12 ('+DATA_ETC4/QO1/log1_g12_m1.dbf') size 4000m reuse; SQL> alter database add logfile thread 1 group 13 ('+RECO_ETC4/QO1/log1_g13_m1.dbf') size 4000m reuse; SQL> alter database add logfile thread 1 group 14 ('+RECO_ETC4/QO1/log1_g14_m1.dbf') size 4000m reuse;
SQL> alter database add logfile thread 2 group 21 ('+DATA_ETC4/QO1/log1_g21_m1.dbf') size 4000m reuse; SQL> alter database add logfile thread 2 group 22 ('+DATA_ETC4/QO1/log1_g22_m1.dbf') size 4000m reuse; SQL> alter database add logfile thread 2 group 23 ('+RECO_ETC4/QO1/log1_g23_m1.dbf') size 4000m reuse; SQL> alter database add logfile thread 2 group 24 ('+RECO_ETC4/QO1/log1_g24_m1.dbf') size 4000m reuse;
SQL> alter database add logfile thread 3 group 31 ('+DATA_ETC4/QO1/log1_g31_m1.dbf') size 4000m reuse; SQL> alter database add logfile thread 3 group 32 ('+DATA_ETC4/QO1/log1_g32_m1.dbf') size 4000m reuse; SQL> alter database add logfile thread 3 group 33 ('+RECO_ETC4/QO1/log1_g33_m1.dbf') size 4000m reuse; SQL> alter database add logfile thread 3 group 34 ('+RECO_ETC4/QO1/log1_g34_m1.dbf') size 4000m reuse;
101 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
SQL> alter database add logfile thread 4 group 41 ('+DATA_ETC4/QO1/log1_g41_m1.dbf') size 4000m reuse; SQL> alter database add logfile thread 4 group 42 ('+DATA_ETC4/QO1/log1_g42_m1.dbf') size 4000m reuse; SQL> alter database add logfile thread 4 group 43 ('+RECO_ETC4/QO1/log1_g43_m1.dbf') size 4000m reuse; SQL> alter database add logfile thread 4 group 44 ('+RECO_ETC4/QO1/log1_g44_m1.dbf') size 4000m reuse;
SQL> alter database switch logfile;
SQL> alter database drop logfile group 1; SQL> drop tablespace PSAPUNDO;
SQL> alter database enable public thread 1; SQL> alter database enable public thread 2; SQL> alter database enable public thread 3; SQL> alter database enable public thread 4;
Update initQO1.ora Edit pfile initQO1.ora, delete information from the migration steps and append additional parameters for the RAC nodes: oracle@etc4-cn01:$ cd /u01/app/oracle/product/11.2.0.2/dbhome_1/dbs oracle@etc4-cn01:$ cat initQO1.ora
*.db_name='QO1' *.db_block_size=8192 *.compatible=11.2.0.2 *.audit_trail=none *.log_archive_dest_1='location=+RECO_ETC4/QO1 valid_for=(ALL_LOGFILES,ALL_ROLES) db_unique_name=QO1' db_unique_name='QO1' *.db_create_file_dest='+DATA_ETC4' *.db_create_online_log_dest_1='+DATA_ETC4' *.db_create_online_log_dest_2='+RECO_ETC4' *.control_files='+DATA_ETC4/QO1/CONTROLFILE/cntrl1.dbf','+RECO_ETC4/QO1/CONTROLFIL E/cntrl2.dbf' *.db_recovery_file_dest='+RECO_ETC4' *.db_recovery_file_dest_size=4G *.shared_pool_size=2181038080
*.remote_login_passwordfile='exclusive' *.remote_os_authent='TRUE' *.undo_management = auto
QO11.instance_number = 1 QO11.thread = 1 QO11.instance_name = QO11 QO11.service_names = (QO1,QO11) QO11.undo_tablespace='psapundo_001'
QO12.instance_number = 2 QO12.thread = 2 QO12.instance_name = QO12 QO12.service_names = (QO1,QO12) QO12.undo_tablespace='psapundo_002'
QO13.instance_number = 3 QO13.thread = 3 QO13.instance_name = QO13 QO13.service_names = (QO1,QO13) QO13.undo_tablespace='psapundo_003'
QO14.instance_number = 4 QO14.thread = 4 QO14.instance_name = QO14
102 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
QO14.service_names = (QO1,QO14) QO14.undo_tablespace='psapundo_004'
Generate spfile Create a binary spfile by using sqlplus oracle@etc4-cn01:$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.3.0 Production on Fri Aug 31 08:26:03 2012
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP, Data Mining and Real Application Testing options
SQL> create spfile='+DATA_ETC4/QO1/SPFILEQO1.ora' from pfile='initQO1.ora';
Register to the grid Register database QO1: oracle@etc4-cn01:$ srvctl add database -d QO1 -o $ORACLE_HOME -p '+DATA_ETC4/QO1/SPFILEQO1.ora' -a "DATA_ETC4,RECO_ETC4"
Register RAC nodes: oracle@etc4-cn01:$ srvctl add instance -d QO1 -i QO11 -n etc4-cn01 oracle@etc4-cn01:$ srvctl add instance -d QO1 -i QO12 -n etc4-cn02 oracle@etc4-cn01:$ srvctl add instance -d QO1 -i QO13 -n etc4-cn03 oracle@etc4-cn01:$ srvctl add instance -d QO1 -i QO14 -n etc4-cn04
Start the database: oracle@etc4-cn01:$ srvctl start database -d QO1
Step 13. Add RAC VIPs and Listeners on the IPoIB network Verify that Clusterware has been configured with all the required networks root@etc4-cn01:~# oifcfg getif bondeth0 10.129.184.0 global public bondib0 192.168.10.0 global cluster_interconnect bondib1 192.168.10.0 global cluster_interconnect ipmpapp0 192.168.30.0 global public
Add the IPoIB network root@etc4-cn01:~# srvctl add network -k 2 -S 192.168.30.0/255.255.255.0/ipmpapp0 root@etc4-cn01:~# srvctl config network Network exists: 1/10.129.184.0/255.255.255.0/bondeth0, type static Network exists: 2/192.168.30.0/255.255.255.0/ipmpapp0, type static root@etc4-cn01:~# olsnodes etc4-cn01 etc4-cn02 etc4-cn03 etc4-cn04
103 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Add RAC VIPs to this IPoIB network root@etc4-cn01:~# srvctl add vip -n etc4-cn01 -A etc4-01-ib-vip/255.255.255.0 -k 2 root@etc4-cn01:~# srvctl add vip -n etc4-cn02 -A etc4-02-ib-vip/255.255.255.0 -k 2 root@etc4-cn01:~# srvctl add vip -n etc4-cn03 -A etc4-03-ib-vip/255.255.255.0 -k 2 root@etc4-cn01:~# srvctl add vip -n etc4-cn04 -A etc4-04-ib-vip/255.255.255.0 -k 2
As the oracle user add the listeners for the IPoIB network oracle@etc4-cn01:~$ srvctl add listener -l LISTENER_IB -k 2 -p 1521 -s oracle@etc4-cn01:~$ srvctl start listener -l LISTENER_IB
Add the following TNS definitions to the DB instances tnsnames.ora file oracle@etc4-cn01:/u01/app/oracle/product/11.2.0.2/dbhome_1/network/admin$ tail -34 tnsnames.ora
LISTENER_IBREMOTE = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = etc4-02-ib-vip)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = etc4-03-ib-vip)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = etc4-04-ib-vip)(PORT = 1521)) ) )
LISTENER_IBLOCAL = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = etc4-01-ib-vip)(PORT = 1521)) ) )
LISTENER_10GBEREMOTE = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = etc4-02-vip)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = etc4-03-vip)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = etc4-04-vip)(PORT = 1521)) ) )
LISTENER_10GBELOCAL = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = etc4-01-vip)(PORT = 1521)) ) ) oracle@etc4-cn02:/u01/app/oracle/product/11.2.0.2/dbhome_1/network/admin$ tail -34 tnsnames.ora
LISTENER_IBREMOTE = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = etc4-01-ib-vip)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = etc4-03-ib-vip)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = etc4-04-ib-vip)(PORT = 1521)) ) )
104 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
LISTENER_IBLOCAL = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = etc4-02-ib-vip)(PORT = 1521)) ) )
LISTENER_10GBEREMOTE = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = etc4-01-vip)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = etc4-03-vip)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = etc4-04-vip)(PORT = 1521)) ) )
LISTENER_10GBELOCAL = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = etc4-02-vip)(PORT = 1521)) ) ) oracle@etc4-cn03:/u01/app/oracle/product/11.2.0.2/dbhome_1/network/admin$ tail -34 tnsnames.ora
LISTENER_IBREMOTE = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = etc4-02-ib-vip)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = etc4-01-ib-vip)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = etc4-04-ib-vip)(PORT = 1521)) ) )
LISTENER_IBLOCAL = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = etc4-03-ib-vip)(PORT = 1521)) ) )
LISTENER_10GBEREMOTE = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = etc4-02-vip)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = etc4-01-vip)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = etc4-04-vip)(PORT = 1521)) ) )
LISTENER_10GBELOCAL = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = etc4-03-vip)(PORT = 1521)) ) ) oracle@etc4-cn04:/u01/app/oracle/product/11.2.0.2/dbhome_1/network/admin$ tail -34 tnsnames.ora
LISTENER_IBREMOTE = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = etc4-02-ib-vip)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = etc4-03-ib-vip)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = etc4-01-ib-vip)(PORT = 1521)) )
105 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
)
LISTENER_IBLOCAL = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = etc4-04-ib-vip)(PORT = 1521)) ) )
LISTENER_10GBEREMOTE = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = etc4-02-vip)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = etc4-03-vip)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = etc4-01-vip)(PORT = 1521)) ) )
LISTENER_10GBELOCAL = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = etc4-04-vip)(PORT = 1521)) ) )
Log in to the DB and configure the instances to register with all listeners of all the networks oracle@etc4-cn01:~$ sqlplus "/as sysddba" SQL> alter system set listener_networks='((NAME=network2)(LOCAL_LISTENER=LISTENER_IBLOCAL)(REMOTE_LISTEN ER=LISTENER_IBREMOTE))', '((NAME=network1)(LOCAL_LISTENER=LISTENER_10GBELOCAL)(REMOTE_LISTENER=LISTENER_10G BEREMOTE))' scope=both;
SQL> alter system register;
Step 14. Connect the SAP instances to Oracle RAC
14a) Define the database services for the SAP application instances Every SAP application instance uses a unique database service for the connection to the database. The creation of the services is done on an Oracle RAC node as user oracle. In our example we logon to node 1 etc4-cn01. Example: Definition of database service (-s) QO1_DVEBMGS for database (-d) QO1 for primary SAP application instance. This database instance is configured to be available on instance (-r) QO11, with the ability to fail over to available instances (-a) QO12, QO13 or QO14. # srvctl add service -d QO1 -s QO1_DVEBMGS -r QO11 -a QO12,QO13,QO14 -P \ BASIC -y AUTOMATIC -q true -j long -e SELECT -m BASIC -z 3 -w 5
Add a service for each application instance (D51 to D58) and put them on different preferred RAC instances. # srvctl add service -d QO1 -s QO1_D51 -r QO11 -a QO12,QO13,QO14 -P \ BASIC -y AUTOMATIC -q true -j long -e SELECT -m BASIC -z 3 -w 5
# srvctl add service -d QO1 -s QO1_D52 -r QO12 -a QO13,QO14,QO11 -P \ BASIC -y AUTOMATIC -q true -j long -e SELECT -m BASIC -z 3 -w 5
106 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
# srvctl add service -d QO1 -s QO1_D53 -r QO13 -a QO14,QO11,QO12 -P \ BASIC -y AUTOMATIC -q true -j long -e SELECT -m BASIC -z 3 -w 5
# srvctl add service -d QO1 -s QO1_D54 -r QO14 -a QO11,QO12,QO13 -P \ BASIC -y AUTOMATIC -q true -j long -e SELECT -m BASIC -z 3 -w 5
# srvctl add service -d QO1 -s QO1_D55 -r QO11 -a QO12,QO13,QO14 -P \ BASIC -y AUTOMATIC -q true -j long -e SELECT -m BASIC -z 3 -w 5
# srvctl add service -d QO1 -s QO1_D56 -r QO12 -a QO13,QO14,QO11 -P \ BASIC -y AUTOMATIC -q true -j long -e SELECT -m BASIC -z 3 -w 5
# srvctl add service -d QO1 -s QO1_D57 -r QO13 -a QO14,QO11,QO12 -P \ BASIC -y AUTOMATIC -q true -j long -e SELECT -m BASIC -z 3 -w 5
# srvctl add service -d QO1 -s QO1_D58 -r QO14 -a QO11,QO12,QO13 -P \ BASIC -y AUTOMATIC -q true -j long -e SELECT -m BASIC -z 3 -w 5
Start the services: # srvctl start service -d QO1
Check the status of the services: # srvctl status service -d QO1 oracle@etc4-cn01:~$ srvctl status service -d QO1 Service QO1_D51 is running on instance(s) QO11 Service QO1_D52 is running on instance(s) QO12 Service QO1_D53 is running on instance(s) QO13 Service QO1_D54 is running on instance(s) QO14 Service QO1_D55 is running on instance(s) QO11 Service QO1_D56 is running on instance(s) QO12 Service QO1_D57 is running on instance(s) QO13 Service QO1_D58 is running on instance(s) QO14 Service QO1_DVEBMGS00 is running on instance(s) QO11
Check the status of the database: # srvctl status database -d QO1 oracle@etc4-cn01:~$ srvctl status database -d QO1 Instance QO11 is running on node etc4-cn01 Instance QO12 is running on node etc4-cn02 Instance QO13 is running on node etc4-cn03 Instance QO14 is running on node etc4-cn04
14b) Modify tnsnames.ora Modify the file /sapmnt/QO1/profile/oracle/tnsnames.ora to add the Oracle database connect descriptors for the SAP instances. Here is an example for QO1_D51 and QO1_D52. The same needs to be done for QO1_D51 to QO1_D58. ################ # Filename...... : tnsnames.ora # Created...... : created by SAP AG, R/3 Rel. >= 6.10 # Name...... : # Date...... : # @(#) $Id: //bas/720_REL/src/krn/tpls/ora/TNSNAMES.ORA#1 $
107 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
# Modified manually for Oracle RAC over Infiniband # Generally we use RAC VIP listeners on IPoIB instead of SCAN ################
QO1.WORLD= (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (COMMUNITY = SAP.WORLD) (PROTOCOL = TCP) (HOST = etc4-01-ib-vip) (PORT = 1521) ) (ADDRESS = (COMMUNITY = SAP.WORLD) (PROTOCOL = TCP) (HOST = etc4-02-ib-vip) (PORT = 1521) ) (ADDRESS = (COMMUNITY = SAP.WORLD) (PROTOCOL = TCP) (HOST = etc4-03-ib-vip) (PORT = 1521) ) (ADDRESS = (COMMUNITY = SAP.WORLD) (PROTOCOL = TCP) (HOST = etc4-04-ib-vip) (PORT = 1521) ) ) (CONNECT_DATA = (SERVICE_NAME = QO1) (GLOBAL_NAME = QO1) (FAILOVER_MODE = (TYPE = SELECT) (METHOD = BASIC) ) ) ) QO11.WORLD= (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP) (HOST = etc4-01-ib-vip) (PORT = 1521)) ) (CONNECT_DATA = (SERVICE_NAME = QO11) (GLOBAL_NAME = QO1) ) ) QO12.WORLD= (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP) (HOST = etc4-02-ib-vip) (PORT = 1521)) ) (CONNECT_DATA = (SERVICE_NAME = QO12) (GLOBAL_NAME = QO1) ) ) QO13.WORLD= (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP) (HOST = etc4-03-ib-vip) (PORT = 1521)) ) (CONNECT_DATA = (SERVICE_NAME = QO13) (GLOBAL_NAME = QO1) ) )
108 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
QO14.WORLD= (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP) (HOST = etc4-04-ib-vip) (PORT = 1521)) ) (CONNECT_DATA = (SERVICE_NAME = QO14) (GLOBAL_NAME = QO1) ) ) QO1_DVEBMGS00.WORLD= (DESCRIPTION = (ADDRESS_LIST = (FAILOVER = on) (LOAD_BALANCE = off) (ADDRESS = (PROTOCOL=TCP)(HOST= etc4-01-ib-vip)(PORT=1521)) (ADDRESS = (PROTOCOL=TCP)(HOST= etc4-02-ib-vip)(PORT=1521)) (ADDRESS = (PROTOCOL=TCP)(HOST= etc4-03-ib-vip)(PORT=1521)) (ADDRESS = (PROTOCOL=TCP)(HOST= etc4-04-ib-vip)(PORT=1521)) ) (CONNECT_DATA = (SERVICE_NAME = QO1_DVEBMGS00) (GLOBAL_NAME = QO1) (FAILOVER_MODE= (TYPE=SELECT) (METHOD=BASIC) ) ) ) QO1_D51.WORLD= (DESCRIPTION = (ADDRESS_LIST = (FAILOVER = on) (LOAD_BALANCE = off) (ADDRESS = (PROTOCOL=TCP)(HOST= etc4-01-ib-vip)(PORT=1521)) (ADDRESS = (PROTOCOL=TCP)(HOST= etc4-02-ib-vip)(PORT=1521)) (ADDRESS = (PROTOCOL=TCP)(HOST= etc4-03-ib-vip)(PORT=1521)) (ADDRESS = (PROTOCOL=TCP)(HOST= etc4-04-ib-vip)(PORT=1521)) ) (CONNECT_DATA = (SERVICE_NAME = QO1_D51) (GLOBAL_NAME = QO1) (FAILOVER_MODE= (TYPE=SELECT) (METHOD=BASIC) ) ) ) QO1_D52.WORLD= (DESCRIPTION = (ADDRESS_LIST = (FAILOVER = on) (LOAD_BALANCE = off) (ADDRESS = (PROTOCOL=TCP)(HOST= etc4-02-ib-vip)(PORT=1521)) (ADDRESS = (PROTOCOL=TCP)(HOST= etc4-03-ib-vip)(PORT=1521)) (ADDRESS = (PROTOCOL=TCP)(HOST= etc4-04-ib-vip)(PORT=1521)) (ADDRESS = (PROTOCOL=TCP)(HOST= etc4-01-ib-vip)(PORT=1521)) ) (CONNECT_DATA = (SERVICE_NAME = QO1_D52) (GLOBAL_NAME = QO1) (FAILOVER_MODE= (TYPE=SELECT) (METHOD=BASIC) ) ) )
[… …]
109 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
To set the connection between a SAP instance and an Oracle RAC instance via services, also add the following parameters in the SAP start profile and instance profile. For example, the SAP application instance DVEBMGS00 connects to Oracle RAC via service QO1_DVEBMGS00: The instance profile QO1_DVEBMGS00_epas-lh: dbs/ora/tnsname = QO1_DVEBMGS00
The start profile START_DVEBMGS00_epas-lh: SETENV_05 = dbs_ora_tnsname=QO1_DVEBMGS00
The SAP application instance D51 connects to Oracle RAC via service QO1_D51: The instance profile QO1_D51_eapp1-lh: dbs/ora/tnsname = QO1_D51
The start profile START_D51_eapp1-lh: SETENV_05 = dbs_ora_tnsname=QO1_D51
Using the same method, add the parameters in the instance profile and start profile for every SAP instance with its own RAC service.
Step 15. Move the Connections from 10 GbE to InfiniBand
15a) Move the ASCS connections to InfiniBand To move ASCS connections to InfiniBand, modify the profile parameter SAPLOCALHOST in QO1_ASCS00_escs-lh and START_ASCS00_escs-lh as follows: SAPLOCALHOST = iscs-lh
Modify the profile parameter in DEFAULT.PFL as follows: SAPGLOBALHOST = iscs-lh rdisp/mshost = iscs-lh enque/serverhost = iscs-lh
15b) Move the Central Instance connections to InfiniBand To move the central instance connections to InfiniBand, modify the profile parameter SAPLOCALHOST in QO1_DVEBMGS00_epas-lh and START_DVEBMGS00_epas-lh as follows: SAPLOCALHOST = ipas-lh
110 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
15c) Move the App server connections to InfiniBand To move the application server connections to InfiniBand, modify the profile parameter SAPLOCALHOST to the InfiniBand logical hosts.
15d) Use FQDNs in the hosts file Double check the file /etc/hosts on all affected zones, making sure that the InfiniBand logical hosts entries have their fully qualified domain name after the simple hostnames. 192.168.30.45 iscs-lh iscs-lh.example.domain.com 192.168.30.46 irep-lh irep-lh.example.domain.com 192.168.30.55 iapp1-lh iapp1-lh.example.domain.com 192.168.30.56 iapp2-lh iapp2-lh.example.domain.com 192.168.30.57 iapp3-lh iapp3-lh.example.domain.com 192.168.30.58 iapp4-lh iapp4-lh.example.domain.com 192.168.30.65 ipas-lh ipas-lh.example.domain.com
15e) Check the connections Restart the SAP system; ensure that the instances are connected among themselves and to the database via InfiniBand. For example, on host iscs-lh, execute the following commands to check that the ASCS instance is connected to the database via InfiniBand. IP addresses in the 192.*.*.* subnet are InfiniBand connections. You may also use the grep command to make sure that there is no connection left on 10 GbE. iscs1:# ps -efa | grep ms.sapQO1_ASCS00 qo1adm 1475 1460 0 Aug 28 ? 1:18 ms.sapQO1_ASCS00 pf=/usr/sap/QO1/SYS/profile/QO1_ASCS00_escs-lh iscs1:/# pfiles 1475 | grep 192 sockname: AF_INET 192.168.30.45 port: 3900 peername: AF_INET 192.168.30.61 port: 42198 sockname: AF_INET 192.168.30.45 port: 3900 peername: AF_INET 192.168.30.61 port: 42298 sockname: AF_INET 192.168.30.45 port: 3900 peername: AF_INET 192.168.30.52 port: 39217 sockname: AF_INET 192.168.30.45 port: 3900 peername: AF_INET 192.168.30.54 port: 61912 sockname: AF_INET 192.168.30.45 port: 3900 peername: AF_INET 192.168.30.54 port: 61913 ….... …....
The process and its open files are communicating on the InfiniBand network.
Step 16. SAP Resource Creation The resource groups and resources configuration are identical for ABAP and Java Systems. The following example is for SAP system QO1, which is an ABAP only ERP system.
16a) Preparation for HA SCS and ERS after Installation The following is performed in the ASCS/ERS zone cluster as user root.
111 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
On all the other zones in the zone cluster where the instance may fail over to, copy the following files from the installation zone: /etc/services
If a script was created on the installation zone to add groups and users, it can be re-used on all other zones that the instances may fail over to. Check that the following files are the identical to the files in the installation zone. /etc/group /etc/passwd /etc/project /etc/user_attr
Note: SAP system users must always have the same uids on all the zones they may run. Set the same password for the users on all zones. Copy all the user environment files from the installation zone to the other zones where the instance may fail over to in the zone cluster. The following command copies the whole /export/home folder. tar cfB - /export/home |ssh root@
Log on as user qo1adm, create wrapper scripts in the home folder of qo1adm for the cluster agent to start and stop the SAP instances. Create these wrapper scripts for every instance. Stop/Start the instances with the wrapper scripts as qo1adm user. Copy these scripts to all the zones where the instance may fail over to. Example wrapper script to start the instance ASCS00: file name: /export/home/qo1adm/sapstartASCS ########################################################### kill -9 `/usr/ucb/ps -auxww|grep QO1|grep ASCS00|grep -v startsap|grep -v grep|awk '{print $2}'` /usr/sap/QO1/SYS/exe/run/startsap r3 ASCS00 escs-lh
Example wrapper script to stop the instance ASCS00: file name: /export/home/qo1adm/sapstopASCS ########################################################### /usr/sap/QO1/SYS/exe/run/stopsap r3 ASCS00 escs-lh kill -9 `/usr/ucb/ps -auxww|grep QO1|grep ASCS00|grep -v stopsap|grep -v grep|awk '{print $2}'` /usr/sap/QO1/SYS/exe/run/cleanipc 00 remove
Switch the logical hosts with resource groups for testing reason. For example: # clrg status # clrs status # clrg switch -n iscs1 scs-rg
112 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Logon the host as user qo1adm where the logical host is running, start and stop the instances with the wrapper script. For example: iscs1:qo1adm % ./sapstartASCS iscs1:qo1adm % ./sapstopASCS
Switch logical host to each zone in the cluster and verify that the SAP instances can be started/stopped correctly by the wrapper scripts. No wrapper scripts are required for the ERS instance(s) since the enrepserver process(es) will be directly started by the cluster agent.
16b) Enable HA for message server, enqueue server and enqueue replication server Stop all the SAP instances, do not stop the database. Log on as user root, register the resource types in the specific local zone. For example in the scs- zc: # clrt register sapenq # clrt register saprepl # clrt register sapscs
Create the message server resource # clrs create -d -g scs-rg -t sapscs \ -p sap_sid=QO1 \ -p sap_instance_number=00 \ -p sap_instance_name=ASCS00 \ -p msg_server_port=3900 \ -p scs_startup_script=/export/home/qo1adm/sapstartASCS \ -p scs_shutdown_script=/export/home/qo1adm/sapstopASCS \ -p resource_dependencies=escs-lh \ abapmsg-rs
Bring all the resources into managed and online state. # clrg online -eM +
Check that all the instance processes are running. Since the enqueue server process will be directly started by the cluster agent, remove the start program of the enqueue server from the start profile /sapmnt/QO1/profile/START_
113 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
#------
Disable the message server resource and enable it again. Verify that the enqueue server process is not restarted. # clrs disable abapmsg-rs # clrs enable abapmsg-rs
Create the enqueue resource # clrs create -d -g scs-rg -t sapenq \ -p enqueue_profile=/sapmnt/QO1/profile/QO1_ASCS00_escs-lh \ -p enqueue_server=/usr/sap/QO1/SYS/exe/run/enserver \ -p sap_user=qo1adm -p enqueue_instance_number=00 \ -p resource_dependencies=escs-lh \ abapenq-rs
Bring all the resources into managed and online state. # clrg online -eM +
Check that the enqueue server process enserver is running # ps -ef | grep -i sap | grep -i enserver
Modify the ERS profile /sapmnt/QO1/profile/QO1_
Modify the ERS start profile /sapmnt/QO1/profile/START_
To make sure that the enqueue server and the enqueue replication sever are automatically reconnected after a network failure, modify the ASCS and ERS instance profiles to add following line: enque/enrep/keepalive_count = 1
114 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Setting the parameter enque/enrep/keepalive to 1 tells the enqueue server to send a keepalive package to the enqueue replication server periodically, and wait for an answer to determine, that the connection to the ERS is not lost unnoticed. Next we need to set resource group affinities to ensure that the enqueue server and replication server run on different nodes and that the enqueue server always failover to the node where the enqueue replication server is running. # clrg set -p rg_affinities=--scs-rg rep-rg # clrg set -p rg_affinities=+rep-rg scs-rg
Create the enqueue replication server resource: # clrs create -d -g rep-rg -t saprepl \ -p replica_profile=/sapmnt/QO1/profile/QO1_ERS10_erep-lh \ -p replica_server=/usr/sap/QO1/SYS/exe/run/enrepserver \ -p sap_user=qo1adm \ -p resource_dependencies=erep-lh,abapenq-rs \ abaprepenq-rs
Bring all the resources into managed and online state. # clrg online -eM +
Check the enqueue replication server process enrepserver is running # ps -ef | grep -i sap | grep -i enrepserver
Switch the resources and check that the processes are failed over to the correct node. The enqueue replication resource group (rep-rg) should not run on the same node as the enqueue resource group (scs-rg). The scs-rg should always failover to the node where rep-rg was running. # clrg switch -n
16c) Preparation for HA for the Primary Application Server (former CI) The following steps are performed in the pas-zc zone cluster (for Primary Application Servers, formerly named Central Instance) as user root unless otherwise stated. On all the other zones in the zone cluster where the instance may fail over to, copy following files from the installation zone: /etc/services
115 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
If a script was created in the installation zone to add groups and users, it can be re-used in all other zones that the instances may fail over to. Ensure that the following files are the same as in the installation zone. /etc/group /etc/passwd /etc/project /etc/user_attr
Note: SAP system users must always have the same uids on all the zones it may run. Set the same password for the users on all zones. Copy all the user environment files from the installation zone to other zones where the instance may fail over to in the zone cluster. The following command copies the whole /export/home folder. tar cfB - /export/home |ssh root@
Log on as user qo1adm and create wrapper scripts in the home folder of qo1adm for the cluster agent to start/stop the SAP instances. Create these wrapper scripts for each instance. Stop/Start the instances with the wrapper scripts as user qo1adm. Copy these scripts to all the zones where the instance may fail over to. Example wrapper script to start the instance DVEBMGS00: file name: /export/home/qo1adm/sapstartPAS ########################################################### kill -9 `/usr/ucb/ps -auxww|grep QO1|grep DVEBMGS00|grep -v startsap|grep -v grep|awk '{print $2}'` /usr/sap/QO1/SYS/exe/run/cleanipc 00 remove /usr/sap/QO1/SYS/exe/run/startsap r3 DVEBMGS00 epas-lh
Example wrapper script to stop the instance DVEBMGS00: file name: /export/home/qo1adm/sapstopPAS ########################################################### /usr/sap/QO1/SYS/exe/run/stopsap r3 DVEBMGS00 epas-lh kill -9 `/usr/ucb/ps -auxww|grep QO1|grep DVEBMGS00|grep -v stopsap|grep -v grep|awk '{print $2}'` /usr/sap/QO1/SYS/exe/run/cleanipc 00 remove
Note the difference between sapstartASCS and sapstartPAS. There can be a cleanipc before starting the instance DVEBMGS00 in sapstartPAS but do not in sapstartASCS, or the lock table in shared memory would be deleted and the enqueue server would not get the lock table data from the enqueue replication server while failing over. Switch the logical hosts with resource groups for testing reason. For example: # clrg status # clrs status # clrg switch -n ipas3 pas-rg
116 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Logon to the host as user qo1adm where the logical host is running, start and stop the instances with your wrapper script. Verify that the instance is started/stopped successfully. For example: ipas3:qo1adm % ./sapstartPAS ipas3:qo1adm % ./sapstopPAS
Switch the logical host to each zone in the cluster and verify that the SAP instances can be started/stopped correctly using the wrapper script.
16d) Enable HA for the Primary Application Server
Stop all the SAP instances, do not stop the database. Log on as user root and register the resource types in the specific local zone. For example in the pas-zc: # clrt register sapwebas
Create the primary application server resource # clrs create -d -g pas-rg -t sapwebas \ -p failover_mode=HARD \ -p sap_sid=QO1 \ -p sap_instance_number=00 \ -p sap_instance_name=DVEBMGS00 \ -p sap_user=qo1adm \ -p resource_project_name=QO1 \ -p sap_instance_type=ABAP \ -p webas_startup_script=/export/home/qo1adm/sapstartPAS \ -p webas_shutdown_script=/export/home/qo1adm/sapstopPAS \ -p sap_logdir=/usr/sap/QO1/DVEBMGS00/work \ -p webas_use_pmf=true \ -p resource_dependencies=epas-lh \ abap-pas-rs
Bring the resources to Managed and Online States Bring all the resources into managed and online state in the zone cluster. # clrg online -eM +
Verify that all the instance processes are running. Get the resource status from the global zone with command “clrg status” Switch the resources and make sure that the processes are failed over to the correct node.
16e) Configure Oracle Solaris Cluster HA for SAP AAS as failover resources The additional application servers use the same resource type as the primary application server. Log on as superuser in one of the nodes in the cluster that hosts the SAP instances.
117 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Create the wrapper scripts /export/home/qo1adm/sapstartD51 and /export/home/qo1adm/sapstopD51 for starting/stopping SAP instance D51. Verify that they are working on all the zones where the instance is going to switch to. iapps1:qo1adm 268% cat sapstartD51 kill -9 `/usr/ucb/ps -auxww|grep QO1|grep D51|grep -v sapstart|grep -v startsap|grep -v grep|awk '{print $2}'` /usr/sap/QO1/SYS/exe/run/cleanipc 51 remove /usr/sap/QO1/SYS/exe/run/startsap R3 D51 eapp1-lh iapps1:qo1adm 269% cat sapstopD51 /usr/sap/QO1/SYS/exe/run/stopsap R3 D51 eapp1-lh kill -9 `/usr/ucb/ps -auxww|grep QO1|grep D51|grep -v sapstop|grep -v stopsap|grep -v grep|awk '{print $2}'` /usr/sap/QO1/SYS/exe/run/cleanipc 51 remove
Create the resource D51-rs in the failover resource group D51-rg. # clrs create -d -g D51-rg -t sapwebas \ -p sap_sid=QO1 \ -p sap_instance_number=51 \ -p sap_instance_name=D51 \ -p sap_user=qo1adm \ -p resource_project_name=QO1 \ -p sap_instance_type=ABAP \ -p webas_startup_script=/export/qo1adm/sapstartD51 \ -p webas_shutdown_script=/export/qo1adm/sapstopD51 \ -p sap_logdir=/usr/sap/QO1/D51/work \ -p webas_use_pmf=true \ D51-rs
Manage the resource group and enable the resources to take the SAP application server instances online. # clrg online -eM D51-rg
Repeat the steps above to create failover resources for all the other SAP application server instances D52 – D58.
16f) Operation and Trouble Shooting
Check the status of the SAP Application Server Instances Log on host apps-zc as root and execute the command “clrs status” to get the status of the instances: # ssh root@eapps1 Password: Last login: Mon Apr 23 07:34:09 2012 from epas3 Oracle Corporation SunOS 5.10 Generic Patch January 2005 # bash iapps1:/# clrs status
118 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Start/Stop a single SAP Application Server Instance Logon on host apps-zc as root and execute command “clrs disable
To shut down instance D57 on node1 iapps1:/# clrs disable D57-rs
To start instance D57 on node1 iapps1:/# clrs enable D57-rs
Get the resource status from the global zone Get the resource status from the global zone with command “clrg status -Z
If a resource group is in “ Error_stop_failed” status, logon to the local zone and fix the problem. Then clear the flag “Stop_failed” of the resource, and restart the resources from the local zone. Use “clrs status” to check the status of the resources. # clrs status # clrs clear
Alternatively, the flag from the global zone can be cleared: etc4-01-app1:/# clrs status -Z scs-zc -g scs-rg etc4-01-app1:/# clrs clear -Z scs-zc abapmsg-rs etc4-01-app1:/# clrg online -Z scs-zc +
If a resource group is in “Error_start_failed” status, restart the resource group on the same node or switch it to another node. The flag will be cleared automatically. To restart the resource group: etc4-01-app1:/# clrg restart -Z scs-zc scs-rg
To switch the resource group: etc4-01-app1:/# clrg switch -n iscs2 -Z scs-zc scs-rg
119 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
At this point all the SAP components have been installed and configured and verified functional in their respective zone clusters.
120 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Final Zone Cluster Setup
Step 17. Finalize the Storage Configuration After the central services are installed and configured under Oracle Solaris Cluster, complete the storage configuration with the following: # clrg set -p RG_affinities+=++scalmnt-rg scs-rg # clrg set -p RG_affinities+=++scalmnt-rg rep-rg # clrs set -p Resource_dependencies_offline_restart+=sapmnt-fs-rs,usrsap-fs- rs,trans-fs-rs abapenq-rs # clrs set -p Resource_dependencies_offline_restart+=sapmnt-fs-rs,usrsap-fs- rs,trans-fs-rs abapmsg-rs # clrs set -p Resource_dependencies_offline_restart+=sapmnt-fs-rs,usrsap-fs- rs,trans-fs-rs abaprepenq-rs
After the primary instance is installed and configured under Oracle Solaris Cluster, complete the storage configuration with the following: # clrg set -p RG_affinities+=++scalmnt-rg pas-rg
# clrs set -p Resource_dependencies_offline_restart+=ora-fs-rs,sapmnt-fs- rs,usrsap-fs-rs,trans-fs-rs abap-pas-rs
After the additional dialogue instances are installed and configured under Oracle Solaris Cluster, complete the storage configuration with the following: # clrg set -p RG_affinities+=++scalmnt-rg D51-rg D52-rg D53-rg D54-rg D55-rg D56- rg D57-rg D58-rg # clrs set -p Resource_dependencies_offline_restart+=ora-fs-rs,sapmnt-fs-rs,trans- fs-rs,usrsap-fs-rs -t SUNW.sapwebas +
Step 18. Configure the Oracle External Proxy resource Configure the Oracle External Proxy resource in the SPARC SuperCluster zone cluster. This step is performed after the database for SAP has been configured as a RAC database running in the database domains, and that the connectivity to the database via InfiniBand has been configured. First add the monitoring user and access privileges to the database in the DB domains, from one database node: -bash-3.00$ sqlplus "/as sysdba" SQL> create user hauser identified by hauser; SQL> grant create session to hauser; SQL> grant execute on dbms_lock to hauser; SQL> grant select on v_$instance to hauser; SQL> create profile hauser limit PASSWORD_LIFE_TIME UNLIMITED; SQL> alter user hauser identified by hauser profile hauser; SQL> exit
In the scs-zc zone cluster, create a /var/opt/oracle/tnsnames.ora file on each node iscs1, iscs2, iscs3 and iscs4, as follows:
121 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
iscs1:/# cat /var/opt/oracle/tnsnames.ora QO1 = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = etc4-01-ib-vip)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = etc4-02-ib-vip)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = etc4-03-ib-vip)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = etc4-04-ib-vip)(PORT = 1521)) ) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = qo1) ) )
Create an encrypted password file on iscs1, iscs2, iscs3 and iscs4, as follows: iscs1:/# dd if=/dev/urandom of=/var/cluster/scoep_key bs=8 count=1 iscs1:/# echo hauser | /usr/sfw/bin/openssl enc -aes128 -e -pass file:/var/cluster/scoep_key -out /opt/ORCLscoep/.oep-rs_passwd
Verify that the encrypted password can be decrypted. iscs1:/# /usr/sfw/bin/openssl enc -aes128 -d -pass file:/var/cluster/scoep_key -in /opt/ORCLscoep/.oep-rs_passwd hauser iscs1:/# chmod 400 /var/cluster/scoep_key iscs1:/# chmod 400 /opt/ORCLscoep/.oep-rs_passwd
Create the Resource group and resource from one zone cluster node. iscs1:/# clrt register -f /opt/ORCLscoep/etc/ORCL.oracle_external_proxy ORCL.oracle_external_proxy iscs1:/# clrg create -S oep-rg iscs1:/# clrs create -g oep-rg -t ORCL.oracle_external_proxy \ -p service_name=qo1 \ -p ons_nodes=etc4-01-ib-vip:6200,etc4-02-ib-vip:6200,etc4-03-ib-vip:6200,etc4- 04-ib-vip:6200 \ -p dbuser=hauser -d oep-rs iscs1:/# clrg online -eM oep-rg iscs1:/# clrs status -g oep-rg
=== Cluster Resources ===
Resource Name Node Name State Status Message ------oep-rs iscs1 Online Online - Service qo1 is UP iscs2 Online Online - Service qo1 is UP iscs3 Online Online - Service qo1 is UP iscs4 Online Online - Service qo1 is UP
Step 19. Configure the cross-zone clusters dependencies Configure the cross-zone clusters dependencies to coordinate availability between domains and zone clusters. In the scs-zc zone cluster
122 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
# clrs set -p Resource_dependencies+=oep-rs abapenq-rs # clrs set -p Resource_dependencies+=oep-rs abapmsg-rs
From the global zone: # clrs set -Z pas-zc -p Resource_dependencies+=scs-zc:oep-rs,scs-zc:abapmsg-rs abap-pas-rs
# for i in 51 52 53 54 55 56 57 58 do set -x clrs set -Z apps-zc -p Resource_dependencies+=scs-zc:oep-rs,scs-zc:abapmsg-rs D"$i"-rs set +x done
Step 20. Configure CPU resource control for the zone clusters In this example, the zone clusters for SCS and PAS would need just 0.5 T4 core (=4 threads) and the Apps would need 4 cores (=32 threads). From the global zone of one node: # clzc configure scs-zc clzc:scs-zc> add dedicated-cpu clzc:scs-zc:dedicated-cpu> set ncpus=4 clzc:scs-zc:dedicated-cpu> set importance=20 clzc:scs-zc:dedicated-cpu> end clzc:scs-zc> exit # clzc configure pas-zc clzc:pas-zc> add dedicated-cpu clzc:pas-zc:dedicated-cpu> set ncpus=4 clzc:pas-zc:dedicated-cpu> set importance=20 clzc:pas-zc:dedicated-cpu> end clzc:pas-zc> exit # clzc configure apps-zc clzc:apps-zc> add dedicated-cpu clzc:apps-zc:dedicated-cpu> set ncpus=32 clzc:apps-zc:dedicated-cpu> set importance=20 clzc:apps-zc:dedicated-cpu> end clzc:apps-zc> exit
# clzc halt apps-zc pas-zc scs-zc # clzc boot scs-zc pas-zc apps-zc # poolstat pset id pool size used load 7 SUNWtmp_apps-zc 32 0.00 0.39 0 pool_default 88 0.00 0.08 3 SUNWtmp_pas-zc 4 0.00 0.00 2 SUNWtmp_scs-zc 4 0.00 0.01
Step 21. Configure Policy-based Failover During availability testing, we observed that overload situations could be created during major failures that impacted the performance of all SAP services, regardless of their level of criticality (See the Availability Test section in the Oracle Technical Whitepaper “How to Improve the Efficiency and Performance of an SAP Environment with the Oracle Optimized Solution for SAP”). During the most dramatic failure scenarios, the post-failure workloads showed severe
123 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
performance degradation caused too many application server instances with too many users running on each surviving node. To remediate the performance degradation it was decided to limit and control the impact on the most business-critical applications by implementing, load-based, application-specific failover policies. The policies were configured as follows: 1. Based on previous testing, a policy was implemented limiting the number of SAP application server instances to a maximum of three per node. 2. A policy was configured that assigned a priority and load capacity to each application instances. The failover policies were configured as follows: Highest priority level (30) applied to application instances D51, D52, and D53 Medium priority level (20) applied to application instance D54 Low priority level (10) applied to application instances D55, D56, D57, and D58
# clnode create-loadlimit -p limitname=cpu_load -p softlimit=24 -p hardlimit=28 + # clrg set -p load_factors=cpu_load@8 -p priority=30 -p preemption_mode=Never D51- rg D52-rg D53-rg # clrg set -p load_factors=cpu_load@8 -p priority=20 -p preemption_mode=Has_cost D54-rg # clrg set -p load_factors=cpu_load@8 -p priority=10 -p preemption_mode=Has_cost D55-rg D56-rg D57-rg D58-rg
124 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
References and Documentation
ORACLE WEB PAGES
The Oracle Optimized Solution for SAP http://www.oracle.com/us/solutions/oos/sap/overview/index. html
Oracle Advanced Customer Support Services (ACS) http://www.oracle.com/us/support/software/advanced- customer-services/index.html
Oracle Database and IT Infrastructure for SAP http://www.oracle.com/sap
Oracle for SAP Resources http://www.oracle.com/us/solutions/sap/database/oracle- sap-resources-303416.html
Oracle IT Infrastructure for SAP Resources http://www.oracle.com/us/solutions/sap/infrastructure/sap- resources-352184.html
Oracle Engineered Systems http://www.oracle.com/us/products/engineered- systems/index.html
Oracle SPARC Systems http://www.oracle.com/us/products/servers- storage/servers/sparc-enterprise/overview/index.html
Oracle Storage and Tape Systems http://www.oracle.com/us/products/storage/overview/index.h tml
Oracle Solaris http://www.oracle.com/us/products/servers- storage/solaris/overview/index.html
Oracle Solaris Cluster http://www.oracle.com/us/products/servers- storage/solaris/cluster/overview/index.html
SAP WEB PAGES
SAP on Oracle http://scn.sap.com/community/oracle
Featured Content for SAP on Oracle http://scn.sap.com/docs/DOC-23461
SAP on Oracle Solaris http://scn.sap.com/community/oracle-solaris
Key SAP Notes for SAP on Oracle http://scn.sap.com/docs/DOC-8664
Key SAP Notes for SAP on Oracle Solaris http://scn.sap.com/docs/DOC-23511
SAP on Oracle Real Application Clusters - RACs http://scn.sap.com/docs/DOC-8835
SAP on Oracle Installation and Upgrade http://scn.sap.com/docs/DOC-7888
SAP on Oracle - Performance http://scn.sap.com/docs/DOC-8000
WHITE PAPERS, ARTICLES, BEST PRACTICES
SAP on Oracle White Papers http://scn.sap.com/docs/DOC-8752
Best Practices for Migrating SAP Environments http://www.oracle.com/technetwork/articles/systems- hardware-architecture/sap-migration-163923.pdf
Configuring Sun Storage 7000 Unified Storage Systems for http://www.oracle.com/technetwork/articles/systems- Oracle® Databases hardware-architecture/ss700-config-oracle-db-163902.pdf
125 Oracle Technical White Paper — Implementation Guide for Highly Available SAP on Oracle SPARC SuperCluster
Optimizing Storage in SAP with Oracle Database 11g Release 2 http://www.oracle.com/technetwork/database/focus- areas/performance/sapstorageoptimization-335322.pdf
High Availability with the Standalone Enqueue Server http://help.sap.com/saphelp_nw04s/helpdata/en/6d/e3a03cc 24efd1de10000000a114084/content.htm
SPARC SuperCluster T4-4 Platform Security Principles and http://www.oracle.com/us/products/servers- Capabilities storage/servers/sparc-enterprise/supercluster/supercluster- t4-4/ssc-security-pac-1716580.pdf
DOCUMENTATION AND GUIDES
Sun Cluster Data Service for SAP Web Application Server Guide http://docs.oracle.com/cd/E19316-01/820-2568/index.html for Solaris OS
Installing and Configuring HA for SAP http://docs.oracle.com/cd/E19680-01/html/821- 1536/behgdfff.html
Installing and Configuring HA for SAP Web Application Server http://docs.oracle.com/cd/E19680-01/html/821- 1538/installconfig-1.html
Oracle Solaris Cluster Concepts Guide http://docs.oracle.com/cd/E19680-01/html/821- 1254/index.html
Oracle Database Quick Installation Guide http://docs.oracle.com/cd/E11882_01/install.112/e24349/toc. 11g Release 2 for Oracle Solaris on SPARC htm
126
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only and Deployment Guide for Highly Available SAP on the contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other Oracle SPARC SuperCluster warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or Original September 2012 fitness for a particular purpose. We specifically disclaim any liability with respect to this document and no contractual obligations are Updated January 2013 formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any Authors: Pierre Reynes ; Gia-Khanh Nguyen ; means, electronic or mechanical, for any purpose, without our prior written permission. Hans-Juergen Denecke ; Xirui Yang Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective Oracle Corporation owners. World Headquarters 500 Oracle Parkway AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. Intel Redwood Shores, CA 94065 and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are U.S.A. trademarks or registered trademarks of SPARC International, Inc. UNIX is a registered trademark licensed through X/Open Company, Ltd. 0110 Worldwide Inquiries: Phone: +1.650.506.7000 Fax: +1.650.506.7200 oracle.com