Oracle Exadata / Oracle Exalytics Integration for Fast Analytics and an Optimized Data Warehouse

Satyendra Pasalapudi Associate Practice Director, IMS Apps Associates

Satyendra Kumar Pasalapudi Associate Practice Director, IMS at Apps Associates Co-Founder & Vice President of AIOUG 15+ years of experience working with Oracle Products and Technologies

@pasalapudi

Global Reach, Broad Service Profile • Founded in 2002, 600+ employees • US, Europe, India, Middle East • Service Offerings: Applications, CRM, Analytics, EPM, Cloud, Middleware, Application Development, App & Infrastructure Managed Services

Significant Investment in R&D • Cloud (IaaS, PaaS, SaaS) • Business Process & System Integration • Analytics & Big Data

Strategic Partnerships, Certifications, Credentials • Oracle Platinum Partner, Oracle Specialized Across Our Portfolio of Services • AWS Advanced Consulting Partner, Certified Managed Services Provider • Microsoft Certified • CMMI Level 3 & SSAE 16

• Problem Statement – Fast Analytics & Real-Time Reporting • Introduction to Exalytics and Software Features • In-Memory Analytics • InfiniBand Connection to Exadata • Using DBFS in the Exadata Storage Servers for ETL • EHCC • Management and Monitoring • Workloads

Speed, agility, and intelligence are competitive advantages that nearly all organizations seek.

Existing traditional reporting systems provide information after 36 – 48 hours.

To support operational users and influence what should happen next, the data should be available in real time to know what is happening now.

• There are three aspects to this challenge: – Availability – 24/7/365 without service interruption or performance degradation – Reduced latency – At all layers of Business Intelligence – Heterogeneity and IT flexibility – Flexibility to be easily modified and distributed across diverse IT systems

• Companies need a platform that allows business applications to benefit from continuous access to real-time information – without compromising on performance and data integrity

Value Business Event

Data Delivered to DW

Data Analyzed

Action Taken

Time

Value Business Event

Data Delivered to DW Data Analyzed Action Taken

Time

All real-time business intelligence systems have some latency, but the goal is to minimize the time from the business event happening to a corrective action or notification being initiated. Analyst Richard Hackathorn describes three types of latency:

• Data latency; the time taken to collect and store the data • Analysis latency; the time taken to analyze the data and turn it into actionable information • Action latency; the time taken to react to the information and take action

• Cache data in memory from disk to achieve fast query response • For OLTP/ DSS workloads, memory should be big enough to hold as much as possible

SGA 100% cache hit ratio is ideal for OLTP workloads Buffer Cache

Cache Miss Cache Hit

1. Data size is increasing 1. Huge amounts of data 2. Not possible to cache all data

SGA

Buffer cache 2. User volume is increasing

3. Many I/O operations

How can we achieve faster and real-time BI?

OGG – Oracle GoldenGate ODI – Oracle Data Integrator

Extract Transform Load Lookups/Calcs Transform Load

Set-based SQL transforms SQL load typically faster inside DB is always faster

ODI InfiniBand Lookup Bulk Data Movement Stage Prod Data Data Transformation Sources Near Real Time Flexible options for OGG True Real Time OGG real-time data streams

Exalytics X3-4 Exalytics T5-8

2 TB DRAM 4 TB DRAM

40 Intel Cores 128 SPARC T5 Processor Cores

2.4 TB Flash Storage 3.2 TB Flash Storage

5.4 TB Hard Disk Drive 7.2 TB Hard Disk Drive

Oracle Exalytics is the industry's first in- Why Exalytics? memory machine that delivers the fastest performance for business intelligence and New Challenges planning applications Need to analyze/aggregate/display huge volume of data with extreme fast response time.

Solutions by Exalytics • High computing capacity: more CPUs/memory • In-memory analytics: DW/aggregates in memory for extremely high performance • Fast inter-connectivity with InfiniBand • Fully optimized software to utilize the hardware

New Exalytics T5-8

TimesTen for Exalytics 4 TB RAM 128 Processing Cores 3.2 TB Flash Storage Exalytics X3-4 Essbase

2 TB RAM 40 Processing Cores Adaptive In-Memory Tools 2.4 TB Flash Storage In-Memory Analytics In-Memory Analytics Oracle Business Intelligence Foundation Suite Software Hardware

Browser/Mobile Weblogic Server Javahost (Analytics App) (Chart Engine) Timesten DB (In-Memory Analytics)

OBIPS OBIS (Presentation) (Relational OLAP) Essbase Exalytics (Multidimensional OLAP) Webcat Repository

Endeca Information Discovery (Unstructured Data)

Data Warehouse

• JVM Heap size • JVM Heap Size for Trellis Charts • Analytics app Connection Pool • Thread Pool size • DB connection • Analytics app session timeout • Javahost Clustering • Log Level • WLS LDAP connection pool • Temp/log folder location • Slow queries • Log folder location • WLS Min/Max threads

WLS (bi_server) Javahost (Analytics App) (Chart Engine) TimesTen DB Browser/Mobile

OBIPS OBIS • Hardware Acceleration flag (sawserver) (nqserver) • Logical SQLs/Physical SQLs • Init block • Server Thread pool • Hardware Acceleration flag Webcat Repository • DB Gateway connection pool • Chart Thread Pool • OBIS Cache • OBI PS Cache (catalog/account) • Temp/log folder location • Session timeouts • Log level • Cursor Wait Time Exalytics • Memory Quota checking • Temp/log folder location Data Warehouse

Summary advisor scripts to populate TimesTen cache

• Intelligently recommends an optimal list of aggregate tables based on query patterns that will achieve maximum query performance

• Summary advisor generates an aggregate creation script that can be run to create the recommended aggregate tables

Adaptive In-Memory ROLAP Data Management

• Out-of-the-box acceleration for analytics – Process billion-rows per second – Ideal for aggregates and data marts • Store more data in Exalytics Row-based Columnar – No Indexes required – Lower administration cost • Offload analytics processing – Data and operational isolations from production data warehouse

Easy wizard based interface for cache definitions

More options for data load GoldenGate Loader • Real-time data refresh using native GoldenGate Native Batch Loader OBIEE Server integration TimesTen Loader In-Memory • Incremental batch loading with built-in batch loader Database Data optimizations for efficiency and performance • Data type optimizations • Compression heuristics based on data • Selective columns • Load sequencing, parallelization • TimesTen indexing

In-Memory MOLAP

• Multi dimensional OLAP server for analytics applications • Essbase on Exalytics has number of optimizations for in-memory operations • Improvements to overall storage layer performance • Enhancements to parallel operations • Enhanced MDX syntax and high performance MDX query engine • Essbase on Exalytics provides up to 16x faster query execution as well as up to 6x reduction in write back and calculation operations, including batch processes

Exalytics In-Memory Analytics InfiniBand Fabric

Exadata Big Data Appliance OLTP High Performance Data Warehousing Hadoop Engine

An Unbalanced Configuration

<20% Possible Database CPUs Memory HBAs Switches Controllers Disks Efficiency

Exadata Configuration

100% Possible Database CPUs Memory HBAs Switches Controllers Disks Efficiency

• Highest Bandwidth Delivers 40 Gigabit Server Connections Today • Over 3x the Throughput of 10 Gb Ethernet

Lowest Latency • Application-to-Application Latency as Low as 1 µS • Less Than 1/10th the Latency of Ethernet

Unleash Server • Eliminates O/S Intervention in Network and Storage Communication Resources • Frees Up CPU & Memory for Application Processing

Unrivaled • Scales to Thousands of Physical Systems Scalability • Partitionable into Isolated Virtual Networks

Sun QDR InfiniBand Bandwidth Connectivity for Sun Servers and Storage

• 40 GB quad data rate InfiniBand connectivity • Supports rich set of network and storage protocols – Ideal for delivering network services to high-performance clusters • Form factor: Sun InfiniBand dual port 4x QDR PCIe low profile Host Channel Adapter M2 – PCI Express low profile (x8 PCIe Base 2.0) – Two QDR InfiniBand QSFP ports

• Oracle Exalytics has 2-port QDR InfiniBand Host Channel Adapter (HCA) installed in the PCIe slot 6 • Oracle Exadata Database Machine includes two 36-port InfiniBand leaf switches (Sun datacenter QDR InfiniBand switch 36)

These two switches located in the middle of the rack are referred to as leaf switches

Uniform responsiveness over large federated deployments Exadata Exalytics OLTP In-Memory Analytics Data Warehousing • Parallel Processing • Extension to In-Memory Cache / Data Mart

Optimum SQL “Speed of Thought” Enterprise Generation for Exadata Extreme performance via pre-integrated, Business Intelligence Platform optimized Oracle database platform

• Database Machine comes with DBFS shared Linux file system • Shared storage for ETL staging, scripts, reports and other application files • Files are stored as SecureFile LOBs in database tables stored in Exadata • Protected like any database – mirroring, DataGuard, flashback, etc. • 5-7 GB per second file system I/O throughput Load into Database Using External Tables

ETL Files in DBFS ETL More file throughput than high-end NAS filer

Development

Performance Service Level Monitoring & Management Diagnostics OBIEE Essbase Production Test

Hardware & Software Lifecycle Management

• With real customer FIN DB (500 GB), the queries are 2.5x faster FIN DB • With CRM DW (320 GB), the queries are 23% Attributes TT 11.2.2.5 TT 11.2.2.4 faster No. of SQLs 514 514

Response time 12890 32527 Optimizations in seconds (3.5hrs) (9 hrs) • Improved Index Advisor to provide excellent index recommendations CRM DW • Query functions for Analytics replaced with Attributes TT 11.2.2.5 TT 11.2.2.4 faster implementations No. of SQLs 132 132 • Faster Star Join and Merge Join Response time • Faster expression evaluator in seconds 226 297 • Faster Hash index and Tmp index

• 5.7x faster on recommending aggregates • Financial Data Warehouse / Exadata (360 GB) – Optimized Summary Advisor internal queries – Optimized Statistics Logging

BI Build Exalytics 11.1.1.6.10 Time taken to complete summary advisor 44 mins 249 mins No. of aggregates suggested 8 10 • Aggregates in TT improved overall query response time by 2.5x

Exadata DW TT 11.2.2.5 Aggregates Total number of queries 514 514 Total RT in seconds 10817 (3hrs) 4355 (1.2hrs)

Recreate Drilldown Usage Measure & Compare Query Original Timing Exalytics Timing Fiscal Year/Account Level 1 9 sec < 1 sec Fiscal Year/Account Level 2 15 sec < 1 sec Fiscal Year/Account Level 3 11 sec < 1 sec Fiscal Year/Account Level 4 9 sec < 1 sec Fiscal Year/Account Level 5 9 sec < 1 sec Fiscal Year/Account Level 6 9 sec < 1 sec Fiscal Year/Account Level 7 9 sec < 1 sec Fiscal Qtr/Account Level 7 39 sec < 1 sec Fiscal Month/ Account Level 7 28 sec < 1 sec Fiscal Month/Account Level 7/Sub Region Level 2 15 sec < 1 sec

These latencies can be overcome to achieve faster/real-time BI with the combination of software/hardware features of Exadata & Exalytics

Web: www.appsassociates.com Email: [email protected] | [email protected] YouTube: www.youtube.com/user/AppsAssociates LinkedIn: www.us.linkedin.com/company/Apps-Associates Twitter: @AppsAssociates | @pasalapudi Facebook: www.facebook.com/AppsAssociatesGlobal Google+: www.plus.google.com/+AppsAssociatesGlobal/