Exadata Storage Server ‰‰‰ Administration ‰‰‰ Manage (Name=Other, Level =3 Allocation=100)) IO Resource

CON8245 Tips for Successful Oracle Exadata Management with Oracle Enterprise Manager 12c

Ashish Agrawal Consulting Product Manager Oracle Corporation, Product Management Roland Schiller Expert System Engineer, Swiss Reinsurance Company Ltd Akash Guha Database Administration Specialist, eBay Inc September, 2014

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

1 What & How is Exadata Monitored?

2 What's New in Exadata Management?

3 Real World Experience

1 What & How is Exadata Monitored?

2 What's New in Exadata Management?

3 Real World Experience

Database

Storage Server

Infiniband Network

KVM, PDU, ILOM, CISCO SWITCH

ORACLE DATABASE MACHINE OEM Agent deployed on the Compute Node COMPUTE NODE #1

DATABASE SERVER 1

Exadata Plug-in deployed with the Agent Oracle Enterprise Manager 12c Agent 5 OMS 2 Exadata Plug-in Agent communicates with Storage Server and Infiniband Switch targets using ssh 4 3 Other DBM Devices Exadata Plug-in monitors the other DBM Exadata Storage components Server PDU KVM

Exadata Infiniband ILOM CISCO S/W OEM Agent collects data and uploads it to the Switches and Network Oracle Management Server (OMS)

Simplifies monitoring

View all critical issues in the system, monitor performance

Drill down to individual targets

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Exadata Management Integrated View of Hardware and Software • Hardware view • Schematic of hardware components: Storage Servers, Compute Nodes, IB Switches etc • Hardware components status & alerts • Software/system view • Performance, availability, usage by databases, services, clusters • Software Alerts for db, Cluster, ASM • Topology view of DB systems/clusters • Configuration view • Version summary of all components

Storage Cell monitoring and administration support • Cell Home page and performance pages • Actions supported: Start/stop Cell, verify connectivity, setup SSH

Automatic discovery of Exadata cells

Management by Cell Group

• All cells used by a database automatically placed in a group Navigation: Targets ‰‰‰ Exadata ‰‰‰ DB Machine ‰‰‰ • Cell Group level administration operations Exadata Grid Name

Perform Cell Administration tasks • Execute Cellcli commands on a set of cells or all cells

Allocate I/O resources across databases by means of an IORM plan configured on each storage cell IORM plans should be identically configured on each storage cell Advantage with Exadata Plugin:- Implement IORM plan across all cells with one click

CellCLI> ALTER IORMPLAN – dbPlan = ( - (name=DBM, level=1 allocation=60), (name=CRM, level=2 allocation=80), Navigation: Exadata Storage Server ‰‰‰ Administration ‰‰‰ Manage (name=other, level =3 allocation=100)) IO Resource

Infiniband network and switches as GC targets Full monitoring

• Automatic discovery • Alerts (switch generated and EM generated) • Performance metrics, collected every 60 Network home page and performance page seconds • Real time and historical • Configuration metrics – detect and notify configuration changes/best practice Topology view of Network with switch and port level details violations

Perform Infiniband Administration tasks • Enable Port • Disable Port • Clear Performance counters • Clear Error counters

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Integration with the Database Performance Page • Drill down from database Performance page Provides composite view of all health indicators of a cell or cell group

Helps triage

• Load imbalance • ASM related problems • Cell software or hardware failures • Cell configuration issues Navigation: Targets ‰‰‰ Databases ‰‰‰ Target Database ‰‰‰ Database Instance • Network related ‰‰‰Performance ‰‰‰ Performance Home ‰‰‰ I/O failures

Common metrics monitored Specific metrics monitored • Cisco Switch Power supply failure • Configuration change tracking and reporting Fan failure • Unauthorized SNMP access • Keyboard, Video, Mouse (KVM) Temperature out of range • Server connected to KVM added/removed, powered on/off

1 What & How is Exadata Monitored?

2 What's New in Exadata Management?

3 Real World Experience

Full support for Oracle Database Multitenant option 1 CDB level IO workload summary with PDB level 2 details breakdown

Visualization of IO utilization for CDB & PDB 3 IORM for multi-tenancy database (CDB/PDB) environment

Support for Exadata Server Version:11.2.3.3.1,12.1.1.1.1 & Exadata Hardware Navigation: Exadata Storage Server ‰ X4-2,X4-8 Administration ‰‰‰ Manage IO Resource

New Fine-grained Detailed summary of flash space usage performance summary for flash and hard disk

New

I/O time distribution by flash and hard disk.

IORM wait per database

New Usage Statistics Highlights

Flash cache efficiency

Smart Scan efficiency

Usage Statistics by Hard Disk & Flash

Note Utilization, IOPS & MBPS

Provides insight into storage capacity utilization

Numbers will be workload dependent

Any deviation from normal baselines should be investigated

Easier to monitor & compare New usage statistics highlights Flash performance Cache of flash and hard disk & Smart Scan efficiency

New IO time distribution by Flash, Hard disk & IORM New detailed breakdown of Flash Space Usage Wait per DB

Easily identify how Flash Cache is used Visualization for: Visually see Keep, Non-Keep and Unused area I/O time distribution by flash and hard disk IORM wait per database

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 22 What's New in Exadata Management? Storage Cell Performance Page in 12.1.0.6.0 Plug-in Flash & Hard disk Utilization chart to identify if workload is reaching hardware limit New side-by-side comparison of Flash & Hard disk performance charts

Average IO Utilization

Maximum Cell Disk Limit Line For IOPS

Maximum Cell Disk Limit Line For MBPS

Check the Average IO utilization in conjunction with the Maximum Cell Disk Limit Check Datasheet For Exadata hardware for limits Identify if workload reaching hardware limit

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 23 What's New in Exadata Management? Storage Cell Performance Page in 12.1.0.6.0 Plug-in Drill down charts to compare performance between multiple Cells

Performance comparison between multiple Exadata Storage Servers

Identify load imbalance and outliers

Doc ID 1626579.1- Exadata Storage software versions supported by the Oracle Enterprise Manager Exadata plug-in

Roland Schiller, Swiss Re, Sep 2014 Swiss Re is a leading and highly diversified global re/insurance company

• 151 years of experience in providing wholesale re/insurance and risk management solutions

Headquarters, Zurich • We deliver both traditional and innovative offerings in Property & Casualty and Life & Health that meet our clients' needs

• A pioneer in insurance-based capital market solutions, we combine Armonk, New York financial strength and unparalleled expertise for the benefit of our clients

• Our financial strength is currently rated: Standard & Poor’s: AA-/stable; Moody’s: Aa3/stable;

The “Gherkin”, London A.M. Best: A+/stable

Exadata Management with OEM12c | Swiss Re | Sep 2014 27 We enable risk-taking that is essential to enterprise and progress

We identify and evaluate risks Climate change identified as emerging risk almost 20 years ago

Insurance of single risks We select and (eg industrial plants) or entire portfolios of risk take risks (eg motor insurance portfolio)

We transfer and Securitisation of earthquake, trade risks hurricane or life risks

We educate and consult on Over 30 risk-related publications risks during the last 12 months

Exadata Management with OEM12c | Swiss Re | Sep 2014 28 Exadata Strategy until 2015

• General move to Exadata • 150 often highly shared databases supporting 350 different applications • 1PB Storage • Oracle 12c with the Multitenant Option • Active Data Guard

• OEM12c for ° Monitoring + alerting ° Job scheduling ° Backup

Risk DB – 65TB ° Data pump x4-2 2014 ° DB provisioning ° Active Data Guard

DWH DB – 8TB ° GDS x2-2 2012

Exadata Management with OEM12c | Swiss Re | Sep 2014 29 Hundreds of Exadata Metrics to collect, monitor and alert for …all in one place!

Metrics for • Storage cells • Exadata hosts • Databases • Infiniband • PDU • Exachk

Configure • Warning Thresholds • Critical Thresholds • Corrective Actions • Add your own Metric Extensions

Exadata Management with OEM12c | Swiss Re | Sep 2014 30 Metrics by Component => Monitoring Templates for Hosts, DB, ILOM, PDU etc

• You may need different templates per SLA level • Have an Enterprise User own the objects

Exadata Management with OEM12c | Swiss Re | Sep 2014 31 Templates organised by Environment => Template Collections for Prod / Non-Prod

Exadata Management with OEM12c | Swiss Re | Sep 2014 32 Administration Groups Keep it Simple!

• Administration Groups are a special type of group used to fully automate the application of management settings (monitoring settings, compliance standards, and cloud policies) to targets upon joining the group.

Exadata Management with OEM12c | Swiss Re | Sep 2014 33 Set Lifecycle during Discovery -> All Monitoring/Alerting Settings applied automatically

• When a target is added to an administration group, Enterprise Manager automatically applies management settings associated with the group to the newly added target.

Exadata Management with OEM12c | Swiss Re | Sep 2014 34 Dynamic Groups for Exadata Availability and Incident Dashboard

• A dynamic group allows administrators to create a group of targets by specifying its membership criteria instead of adding targets directly into the group. • Membership criteria are based on target properties such as Lifecycle Status, Line of Business, Target Type, etc. • When multiple criteria are defined for a dynamic group, a target must match all criteria before it is automatically added to the dynamic group.

Exadata Management with OEM12c | Swiss Re | Sep 2014 35 Dynamic group "Exadata Production"

Exadata Management with OEM12c | Swiss Re | Sep 2014 36 Drill down into any target member of "Exadata Production" for detailed analysis

Exadata Management with OEM12c | Swiss Re | Sep 2014 37 Compare Database Workload on the Storage Grid

Exadata Management with OEM12c | Swiss Re | Sep 2014 38 Detailed Cell Metrics Compare I/O Throughput for DB on Cell Targets

Exadata Management with OEM12c | Swiss Re | Sep 2014 39 IORM Utilization of the Production Standby Compare DB Keys

Exadata Management with OEM12c | Swiss Re | Sep 2014 40 Support for Swiss Re's Strategic IT Objectives

•Availability •Very short time to •Performance market •SLA compliance

Positive Customer Agility Experience

Cloud Productivity Transformation

•Standardisation •Full Integration with •Simplification by Cloud Self Service removing 'home- (PDBaaS) made' complexity •Centralisation

Exadata Management with OEM12c | Swiss Re | Sep 2014 41 Exadata Management with OEM12c | Swiss Re | Sep 2014 42 Legal notice

©2014 Swiss Re. All rights reserved. You are not permitted to create any modifications or derivative works of this presentation or to use it for commercial or other public purposes without the prior written permission of Swiss Re.

The information and opinions contained in the presentation are provided as at the date of the presentation and are subject to change without notice. Although the information used was taken from reliable sources, Swiss Re does not accept any responsibility for the accuracy or comprehensiveness of the details given. All liability for the accuracy and completeness thereof or for any damage or loss resulting from the use of the information contained in this presentation is expressly excluded. Under no circumstances shall Swiss Re or its Group companies be liable for any financial or consequential loss relating to this presentation.

Exadata Management with OEM12c | Swiss Re | Sep 2014 43 Program Agenda with Highlight

1 What & How is Exadata Monitored?

2 What's New in Exadata Management?

3 Real World Experience

• Company Background

• Architecture

• EM 12c - Real Experience

• Benefits of Oracle Enterprise Manager

• Useful Tips

Monitoring Exadata with Oracle Enterprise Manager 12c 46 Company Background

Company – Ebay Inc: Global commerce platform and payments leader

Who Am I – Akash Guha, Database Engineer, MTS2 • Oracle Certified Professional with many years of experience starting with Oracle 8 • Oracle RAC and Performance Tuning Expert • Working On Exadata for the past 2 years

Monitoring Exadata with Oracle Enterprise Manager 12c 47 PayPal Exadata Architecture Primary Data Center

Mission-critical OLTP Databases

Production Database • 2 X Exadata X2-8 (HP-600GB Disk) • 2 X Full Storage Expansion (HP-600GB Disk) Cluster A Cluster B Cluster C

DR Data Center

Active Dataguard Down Stream Users

DR Database • 2 X Exadata X2-8 (HP- 600GB Disk) • 2 X Full Storage RW DW System Expansion (HP-600GB Disk) Cluster C Analytical Database Cluster A Cluster B • 1X Exadata X2-8 (HC-3TB disk)

Monitoring Exadata with Oracle Enterprise Manager 12c 48 Characteristics Hardware Characteristics • 13 x X2-8 • 12 X Full Storage Expansion • 384 Storage Server with 600GB HP Disk

Database Characteristics • Redo 200MB/s across the clusters • 300k Executions/s • 90k Connections from Application

Application Characteristics • Mix of JDBC and C++ • Over 400 application servers • SLA of 250ms for execution of each SQL

Monitoring Exadata with Oracle Enterprise Manager 12c PayPal OEM/ASR/Ops Center Primary Data Center Primary Data Center DR Site Extranet Proxy DRSite Extranet

To DR OEM LB ASR Oracle GCS Other Assets OEM LB Server ASR Central DBnode/Cell OS OEM RAC1 OEM RAC2 Rules open Exadat ILOMs/Cellcli DL360G8 DL360G8 SR DB SAN Storage a OEM/OC OCR on ASM DB Manual SR s/w repo NAS Dataguar OC Back d to DR Solaris OS Tier Proxy1 T4-1 OpsCenter1/2 EC OC DCN Solaris ILOM 2-node cluster Proxy1 T4- OC Back NOC 2x T4-2 1 Tier Proxy2 s/w repo NAS OC DCN Console T4-1 Proxy2 T4- Back Tier DCN 1

Monitoring Exadata with Oracle Enterprise Manager 12c Enterprise Manager Configuration

• 7000+ targets • RAC cluster for OMR on Primary and Standby • 2x active OMS behind LB on primary, 2x passive OMS behind LB on standby • OpsCenter also on RAC clusters • ASR opening SRs via proxy • Alerts visible on NOC Console for 24x7x365 monitoring

Monitoring Exadata with Oracle Enterprise Manager 12c 51 Exadata Plugin for Enterprise Manager 12c

Monitoring Exadata with Oracle Enterprise Manager 12c 52 Exadata Plugin – Alert

stxxx.payapal.com

Monitoring Exadata with Oracle Enterprise Manager 12c 53 Enterprise Manager 12c – Real Experience Problem Description

• Intermittent Session Spike observed on all three primary clusters

• Application SLA miss by 5% causing significant loss to business

• Sub-optimal Query performance- SQLs taking more than 2 seconds

Triage Using Enterprise Manager 12c

• Ash Analytics

• Metric Extension

Monitoring Exadata with Oracle Enterprise Manager 12c Monitoring Exadata with Oracle Enterprise Manager 12c 55 Enterprise Manager 12c – Metric Extension

Monitoring Exadata with Oracle Enterprise Manager 12c 56 Metric Extension: Monitoring HCA Port Throughput • Measuring amount of data Received and Transmitted through the Host Channel Adaptor Ports. • Implementation: Multi column Host script run by root from OEM (courtesy Oracle)

Imbalance in Traffic between Ports

Monitoring Exadata with Oracle Enterprise Manager 12c 57 Observations

• Ash Analytics page shows Cluster Waits as the top Wait Class

• The Drill down page points out the detailed GC Waits and the top SQLs contributing to it

• Metric Extension on HCA throughput data confirms imbalance in traffic between the InfiniBand cards

• The output of RDS-ping shows high outliers and latencies(via ExaWatcher)

Monitoring Exadata with Oracle Enterprise Manager 12c 58 Solutions Applied

• Adjust Cluster Interconnect parameter to use 4 IB cards

cluster_interconnects=192.168.10.1:192.168.10.2:192.168.10.3:192.168.10.4

• Applied Bundle Patch 22 (pre-requisite for patch 12811198)

• Applied Patch 12811198: ASM ASYNC REBALANCE NOT MAXIMIZING IB THROUGHPUT

Balanced Traffic across all Ports

Monitoring Exadata with Oracle Enterprise Manager 12c 59 Benefits of Oracle Enterprise Manager 12c • Automation of Hardware and Software Monitoring for Exadata

• Standardized Monitoring Template

• Use of ASR and minimize human intervention

• Centralized Repository for database metrics

• Extensible Metric Framework

• Easy Interface for troubleshooting Performance problem

• Rich Reporting Framework and enables comparison between Targets

Monitoring Exadata with Oracle Enterprise Manager 12c 60 Tips from our Experience

• Implement High Availability solutions for Enterprise Manger • Keep Enterprise manager up-to-date with current release to benefit from new features and latest fixes • Configure ASR for fast restoration of Hardware Faults • Implement Monitoring Templates to improve standardization • Use OMS Repository for enterprise wise Metrics data – this enables easy comparison between various Targets and Metrics