The Following Is Intended to Outline Our General Product Direction

1 The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

S317423: Deploying PeopleSoft Enterprise Applications on Exadata Tips, Techniques and Best Practices Oracle Development: Database and Applications MAA / Exadata Richard Exley, Darryl Presley – Oracle MAA/Exadata PepleSoft on Exadata

Introduction: MAA and Exadata PeopleSoft on Exada PeopleSoft MAA HR Payroll Test Case Test Case Findings

4 Introduction

5 Introduction MAA/Exadata Team

Committed to delivering a complete hardware and software solution for deploying Oracle applications on Exadata High Performance Scalability High Availability Disaster Recovery Current focus: E-Business Suite PeopleSoft Siebel

6 Exadata Hardware Architecture

Scaleable Grid of industry standard servers for Compute and Storage • Eliminates long-standing tradeoff between Scalability, Availability, Cost

Database Grid • 8 compute servers Storage Grid (1U) • 14 storage servers (2U)

• 64 Intel cores • 112 Intel cores in storage • 100 TB SAS disk, or 336 TB SATA disk InfiniBand Network • Redundant 40Gb/s switches • 5 TB PCI Flash • Unified server & storage net • Data mirrored across storage servers

Exadata Exadata Hybrid Exadata Smart Intelligent Storage Columnar Flash Cache Grid Compression

Storage Server is smart storage, not a DB node Storage remains an independent tier Compute and Memory Intensive Processing Database Servers Perform complex database processing such as joins, aggregation, etc.

Exadata Storage Servers Search tables and indexes filtering out data that is not relevant to a query Cells serve data to multiple databases enabling OLTP and consolidation Data Intensive Simplicity, and robustness of storage appliance Processing

Data is organized and compressed by column Dramatically better compression Speed Optimized Query Mode for Data Warehousing 10X compression typical

Query Runs faster because of Exadata offload! Space Optimized Archival Mode for infrequently accessed data 15X to 50X compression typical

Faster and Simpler Benefits Backup, DR, Caching, Reorg, Clone Multiply

• Exadata has 5 TB of flash • 56 Flash PCI cards avoid disk controller bottlenecks

• Intelligently manages flash • Smart Flash Cache holds hot data • Gives speed of flash, cost of disk

5X More I/Os than • Exadata flash cache achieves: 1000 Disk Enterprise • Over 1 million IO/sec from SQL (8K) Storage Array • Sub-millisecond response times • 50 GB/sec query throughput

Data Guard RAC

ASM ERP CRM Secure Backup Warehouse

Flashback Data Mart HR Online Redefinition

Redundant Hardware Documented and validated Servers, Storage, Network Exadata MAA best practices Database Level HA to achieve the highest Tolerate failures and changes availability

13 PeopleSoft on Exadata Maximum Availability Architecture (MAA)

MAA best practices for PeopleSoft Planned and unplanned outages Integrated Validated Documented

14 PeopleSoft on Exadata Maximum Availability Architecture (MAA)

15 PeopleSoft on Exadata Maximum Availability Architecture (MAA)

Full support for RAC Failover through TAF and FAN Application Server enhanced to support Fast Application Notification (FAN) to expedite application failover as of PeopleTools 8.50.09 Role base services Single Client Access Name (SCAN) support for PeopleSoft application servers utilizing Oracle 11g Release 2 Client

Fast transaction recovery with Flashback technology

16 PeopleSoft on Exadata Maximum Availability Architecture (MAA)

Full support for Oracle Data Guard Physical or logical Recovery Point Objective (RPO) < 5 minutes, 0 if required Data Guard Broker Fast Start Failover - validated Full application stack startup at DR site (example in white paper) PeopleTools 8.51 support for Active Data Guard Utilized to offload queries on standby database Auto-Block Recovery transparent to PeopleSoft Active Data Guard configured as “secondary” database in PeopleSoft application server configuration

17 PeopleSoft on Exadata Maximum Availability Architecture (MAA)

Planned outages: Only minutes of downtime for RAC rolling database upgrades Online database patching Near zero downtime database upgrade Supports Transient Logical Standby

18 Oracle Maximum Availability Architecture

Integrated suite of best-of-breed HA technologies Best Availability AND - Each is scale-out, fully active, data centric Lowest Cost

Online Upgrade Real Application Clusters Upgrade Hardware Data Guard & Clusterware and Software Online Fully Active Fault Tolerant Failover Replica Server Scale-Out

Database Database

Automatic Storage Management Storage Fault Tolerant Storage Scale-Out Storage Recovery Manager & Oracle Secure Backup Flashback Online Redefinition Low Cost High Performance Correct Errors by Redefine Tables Online Data Protection & Archival Moving Back in Time

19 PeopleSoft on Exadata Maximum Availability Architecture (MAA)

PeopleSoft Specific Failover Conditions for application server clients

PeopleSoft Client Operation Behavior Web client user is updating data and PeopleSoft resubmits the update submits or saves the updates during or just after the database failure. Web client user is paging through Pages are rendered from pre-fetched queried data when the database failure result-set. occurs. Web client user is issuing a new query Oracle reconnects and session resume or switching screens just after the the query database failure.

20 PeopleSoft on Exadata Maximum Availability Architecture (MAA)

PeopleSoft Specific Failover Conditions for batch server processes PeopleSoft Batch Operation Behavior Process Scheduler The process scheduler reconnects to the surviving node successfully Application Engine (AE) job submitted AE jobs may fail and show as “No just BEFORE primary failure Success” and require restart Application Engine (AE) submitted AE job completes successfully during or just AFTER primary instance failure COBOL jobs just BEFORE primary Programs with only SELECTs failover instance failure successfully otherwise, restart required. Crystal and SQR reports Same as COBOL programs

PSQUERY, Tree Viewer, XMLP Viewer These components will all failover successfully

21 PeopleSoft on Exadata Maximum Availability Architecture (MAA)

All MAA best practices are supported for PeopleSoft on Exadata PeopleSoft MAA white paper at:

http://www.oracle.com/technetwork/database/features/availability/maa- peoplesoft-bestpractices-134154.pdf

22 PeopleSoft HR Payroll Test Case

23 PeopleSoft on Exadata HR Payroll Test Case

PeopleTools 8.49 (32bit client) PeopleSoft HCM 9.0 PeopleSoft Payroll batch processing: PaySheet PayCalc PayConfirm 500,000 Employees paid 128 Pay Groups 1 Company Multiple run controls/run IDs Single Check = “NO” (Consolidated pay checks) PayCalc can run with multiple concurrent processes

24 PeopleSoft on Exadata HR Payroll Test Case – Application Profile Single row processing - SQL SELECTs / Fetch, INSERTs “sweep” style - very few tables are reread Business logic processing (COBOL) is CPU intensive High IO Rates 15K IOPs/second (Peek) 200MB reads/second (Peek) Index bound Very high network chattiness 12MB/second 37K messages round trips/second

25 PeopleSoft on Exadata HR Payroll Test Case - Lab Setup

Database Server Quarter Rack Exadata V2 Database Machine 2 Compute nodes 3 Storage Cells Oracle Database 11g Release 2 Oracle Enteprise Linux 5 Update 3 Hugepages must be configured for SGA Process Scheduler Co-located with the database, or Separate machine Oracle Enterprise Linux

26 PeopleSoft on Exadata Test Case Findings – note!

Results based on our project work and analysis - your mileage may vary System configuration Application configuration Business requirements Results provided in relative terms so that effects can be assessed for different machines This was an MAA exercise, not a benchmark

27 PeopleSoft on Exadata HR Payroll Test Case – RAC Tuning

Motivation: Scale the workload Exploit both compute nodes Minimize RAC cluster contention Use automatic RAC load balancing

Tuning Method: Table/Index Range Partitioning PAYGROUP range for partition keys 9 Payroll tables were partitioned Most indexes were locally partitioned

28 PeopleSoft on Exadata HR Payroll Test Case – RAC Tuning

Partition Configuration 128 Partitions 1 PAYGROUP per partition Reconfigured run controls to redistribute workload No run control had overlapping paygroups 32 and 64 concurrent payroll processes (streams) Workload redistribution was done for 64 concurrent streams Workload load balanced across both RAC instances

29 PeopleSoft on Exadata HR Payroll Test Case – RAC Tuning

Results weighted average:

Metric 128 Partitions Cluster (%) 8.69 DB CPU (%) 57.0 IO (%) 35.0

RAC cluster wait events are low, so higher throughput and scaling can be achieved

30 PeopleSoft on Exadata HR Payroll Test Case – RAC Tuning

Results: Wait Event (%) PaySheet PayCalc PayConfirm DB CPU % 50.97 44.44 68.54 RAC Cluster % 13.15 8.54 8.30 IO % 36.88 47.48 24.02

Overall elapse time was reduced by 15% when using 128 partitions with 32 concurrent payroll processes More productive work being done

31 PeopleSoft on Exadata HR Payroll Test Case – RAC Tuning

Doubling the workload processes from 32 to 64 reduced overall elapsed time by 20% Workload was not evenly distributed Some PAYGROUPs have fewer paid employees than others Some process streams had more work than others

32 PeopleSoft on Exadata HR Payroll Test Case – RAC Tuning Conclusions

PeopleSoft Payroll can utilize RAC successfully On Exadata, RAC gives you more CPU and IO bandwidth Each RAC node is connected to the storage cells using InfiniBand yielding higher IO bandwidth and Cluster interconnect IOPs requests are distributed between the two RAC nodes Payroll workload does not require affinity to a static RAC instance, so scaling is easily achieved

33 PeopleSoft on Exadata HR Payroll Test Case – Flash Cache

Exadata Smart Flash Cache Reduce Latency 0.7 ms v 7 to 8 ms for conventional IO High access rates > 1,000,000 IOPs in Full Rack database machine No additional requirements to use Flash Cache for PeopleSoft Kicks in automatically CELL_FLASH_CACHE KEEP segment storage attributes for finer grain control

34 PeopleSoft on Exadata HR Payroll Test Case – Flash Cache Results

Results With Flash Cache, overall performance improved by 15% Average IO latency was reduced from 7ms to 3ms with 80% hit ratio Worked automatically – no manual intervention required

35 PeopleSoft on Exadata HR Payroll Test Case – Process Scheduler Placement

PeopleSoft Batch Processes have high number of network round trips (due to single row processing) Placement options: Collocate with database Separate application tier On Exadata Database Machine Can use either TNS or BEQ connection Must consider CPU usage by the COBOL processes and database RAC instance

36 PeopleSoft on Exadata HR Payroll Test Case – Process Scheduler Placement

Not recommended to collocate process scheduler on Exadata compute node where CPU usage may be high Node 1 16 cores, RAC instance only (Green line) Node 2 16 cores, RAC instance + process scheduler running 32 payroll processes (Light blue line)

37 PeopleSoft on Exadata HR Payroll Test Case – Process Scheduler Placement

Process Scheduler placed on separate middle Tier Initial tests were conducted on GigE 2-hops away from Database Machine Latency: 0.264ms Results (PayConfirm) 43% client side 57% database

38 PeopleSoft on Exadata HR Payroll Test Case – Process Scheduler Placement

Process Scheduler placed on separate middle tier Move to a mid tier one-hop away on GigE Latency: 0.121ms Results (PayConfirm) 27% client side 73% database Of total elapse time SQL Net message waits reduction 76K sec to 36K sec Aggregated across all payroll processes

39 PeopleSoft on Exadata HR Payroll Test Case – Process Scheduler Placement

Waits in Seconds and Elapse Time

40 PeopleSoft on Exadata HR Payroll Test Case – Process Scheduler Placement

Consider placing the process scheduler on separate middle tiers with minimal network latency Configure multiple process schedulers on multiple application servers Configure queues to support 2 or more process schedulers Allows for scale-out for driving the Database Machine For HA, configure Master Scheduler InfiniBand can be considered for lower latency and wide bandwidth to share with other applications 4GB/sec and 0.064ms round trip latency

41 PeopleSoft on Exadata HR Payroll Test Case – Process Scheduler Placement

Other factors to considered for process scheduler placement Number and speed of CPU/cores 32 concurrent COBOL process consume 25 – 30% CPU of an 8 core server

42 Summary

43 Summary

Exadata is a fully integrated hardware and software stack Peoplesoft benefits from the performance and HA capabilities of Exadata PeopleSoft products with integrated MAA features now leverage MAA solutions for data protection, high availability and scalability

44 45 46