Copyright © 2013, Oracle And/Or Its Affiliates. All Rights Reserved. 1

2 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions.

The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

. Benchmarks – What do we measure today? . Benchmarketing – Art or Science? . Avoiding the Scalability Trap

. 800 HP . 19500 RPM . Speed 600 km/h . Load 1t

. 400 HP . 2500 RPM . Speed 100km/h . Load 40t

. System Admin . High CPU utilization . DBA . High “Rows/sec” scan rate . DWH User . Short query response time . DataCenter Manager . Number of systems consolidated per Rack Unit

. Single Core CPU speed . SPECint . Multi Core CPU speed . SPECint_rate . Transactional DB performance . TPC-C . Datawarehouse performance . TPC-H . Ideal application server . SPECjbb2013 . SAP OLTP system . SAP SD . Storage performance . SPC-1/SPC-2 . My custom built application . My own test center

A System is more than a CPU

. CPU & Cache → SPECint . Memory throughput → Stream . CPU & Memory – SPECjbb2013 – SAP SD2 . CPU & Memory & IO – TPC-C – TPC-H

. Throughput Performance . Single Thread Performance

. SPECcpu2006 – Focus mainly on CPU, limited use for system evaluation . SPECjbb2005 – Although a Java benchmark, it has a small memory footprint and focuses mainly on CPU. Can be partitioned to hide scalability issues. . Linpack – A typical HPC benchmark with strong floating point focus. Used to derive the famous “Top 500” list.

. TPC-C – The famous OLTP veteran. Very well understood. Scales with disk IO performance. Can be made a CPU benchmark with today’s SSD and In- memory technologies. . TPC-H – Current datawarehouse benchmark. Stresses all aspects of a system. Available in different storage sizes, results only comparable within one class. . TPC-E – Intended OLTP successor to TPC-C, not used very much.

. SPECjbb2005, SPECjvm2008 – Deprecated, focus on server side JVM performance . SPECjEnterprise2010 – Complex, multi tier benchmark. Results difficult (but not impossible) to compare. . SPECjbb2013 – Newest middleware benchmark, focus on CPU & Memory – Reports both throughput and response times, which makes it very challenging and thus interesting.

. SAP SD . Siebel . JD-Edwards . Times-Ten . OLAP

. Not very useful to compare different platforms (except SAP SD) . Interesting to see application performance (not system performance)

. Fluent, Linpack – Typical HPC benchmarks. Not used in commercial context . SPC-1/2 – Storage benchmark . Swingbench – Based on Oracle DB – Useful for local testing (also functional testing and verification) – No comparable results available

. Stream – Memory Bandwidth . Iozone – Filesystem/IO testing . lmbench, unixbench – Old microbenchmarks . iPerf/uPerf/ttcp – Network testing

. Benchmarking is a leapfrogging game . Conflicting Objectives: – Benchmarks want to provide a foundation for comparison – Hardware Vendors want to be „fastest“ . Solution: Publish only where you are fastest (today) – Comparable results only if B beats A after A has published – Useful only if A and B publish within a short time . Publishing Performance/ increases the chance to „be fastest“.

CINT2006 Cores/ Date Vendor System CPU CPUs Rates CPU 08-2010 6150 IBM Power 795 Power7 32 4 09-2010 1720 Oracle M9000 SPARC64 VII+ 32 4 09-2012 6130 IBM Power 780 Power7 16 8 04-2013 3750 Oracle SPARC T5-8 SPARC T5 8 16

All results from www.spec.org as of Nov. 8, 2013

Fastest Best for the job at hand

CINT2006 Cores/ Date Vendor System 8 CPUs Cores Rates CPU 01-2010 2140 IBM Power 770 Power7 64 8 09-2010 2180 HP DL980 G7 Xeon E7 80 10 12-2011 2770 IBM Power 780 Power7 64 8 04-2013 3750 Oracle SPARC T5-8 SPARC T5 128 16

All results from www.spec.org as of Nov. 8, 2013

Price/ Cores/ Date QphH Vendor System CPUs Cores QphH CPU 03-2011 386,478 $18.19 Oracle M9000 64x SPARC64 256 4 VII+ 10-2011 192,001 $6.37 IBM Power 780 8x Power7 32 4 11-2011 205,792 $4.10 Oracle T4-4 4x T4 32 8 06-2013 409,721 $3.94 Oracle T5-4 4x T5 64 16 10-2013 230,119 $1.29 Cisco UCS C420 4x E5-4650 32 8

All results from www.tpc.org as of Nov. 8, 2013

Cores/ SAPS/ Date SAPS Vendor System CPUs Cores CPU Core 2-2013 139220 IBM Power 760 8x Power7+ 48 6 2900 3-2013 220950 Oracle T5-8 8x T5 128 16 1726 5-2013 127030 Fujitsu RX 900 S2 8x E7-8870 80 10 1587

All results from www.sap.de as of Nov. 8, 2013

. SAPS / Core => IBM . SAPS / Socket => Oracle . SAPS on x86 => Fujitsu . SAPS / Oracle DB License => Oracle . SAPS / 3 year TCO => oh no, don‘t make it realistic!!

Winner (64 Benchmark SPARC T5 Power 7 cores calculated) SPECint_rate2006 3750 (T5-8, 128 3730 (Power 780, 64 Power7 (2x) cores, 3.6 GHz) cores, 4.4 GHz) TPC-H @ 3000GB 409721 (T5-4, 64 192001 (Power 780, T5 (1.1x) cores, 3.6 GHz) 32 cores, 4.1 GHz) SAP SD2 220950 SAPS 139220 SAPS (Power Power 7 (1.7x) (T5-8, 128 cores, 760 P7+, 48 cores, 3.4 3.6 GHz) GHz)

System tpmC CPUs Cores tpmC/Core Power 780 1,200,011 2 8 150,000 Power 780 10,366,254 24 192 54,000

SPECint_rate SPECINT/ System CPUs Cores 2006 Core Power 780 652 2 16 40.75 Power 780 2530 8 64 39.53

Results from the IBM Performance Report as of Feb. 2, 2013 and from www.tpc.org as of Nov. 11, 2013

. SPECcpu2006 is “Embarrassingly parallel” – Each instance of the benchmark is independent – Supports “hiding” of scalability limitations in systems . Others are not as easy to scale – TPC-H – TPC-C – SPECjbb2013 . Watch for the difference between scalability issues in Software and Hardware!

Do you have a benchmark?

We have one with 2 CPUs for Oh? performance 4. With our scalability, this will give you OK! performance 32 with just 16 CPUs!

. Two Benchmark Metrics: – “max-jOPS”: Absolute throughput, similar to SPECjbb2005 or TPC-C – “critical-jOPS”: critical throughput under service level agreements (SLAs) specifying response times ranging from 10ms to 500ms . Essentially conflicting objectives – Hard for benchmarking teams to optimize . Throughput: JVM GC in throughput mode . Latency: JVM GC optimized for low latency

Latency Optimization

. HP BL460c G8 – 24 cores, 2 CPUs – Xeon E5-2697 3.5 GHz . 56367 max-jOPS . 30078 critical jOPS

Throughput Optimization

. HP DL560 G8 – 32 cores, 4 CPUs – Xeon E5-4650 3.3 GHz . 66007 max-jOPS . 18049 critical jOPS

max-jOPS/ Relative System Cores max-jOPS Core Scalability ML310e G8 4 12453 3113.25 100% ML350p G8 16 41954 2622.13 84% DL560 G8 32 67850 2120.31 68% DL980 G7 80 106141 1326.76 43% SPARC T4-2 16 34804 2175.25 100% SPARC T5-2 32 75658 2364.31 108%

max-jOPS/ Relative System Cores max-jOPS Core Scalability SPARC T4-2 16 34804 2175.25 100% SPARC T5-2 32 75658 2364.31 108% SPARC T5-2 32 81084 2533.86 116%

Java update from JDK 7 update 17 to JDK 7 update 40 Plus additional Memory/GC tuning

Scale down, not up!

. Right: Scale Large to Small – Big system performance => little system performance . Wrong: Scale Small to Large – Little system performance => big system performance

. Know your benchmarks! . Test what you will use in production, or as close as possible! . If you scale results, scale them down! . Don’t stop at performance! – Consider other factors like . TCA & TCO . Licensing . Administration, Migration, Integration

45 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 46 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 47 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. Benchmark Disclosures

SAP: Two-tier SAP Sales and Distribution (SD) Standard Application benchmarks SAP Enhancement package 5 for SAP ERP 6.0 as of 3/26/13: SPARC T5-8 (8 processors, 128 cores, 1024 threads) 40,000 SAP SD users, 8 x 3.6 GHz SPARC T5, 2 TB memory, Oracle Database 11g, Oracle Solaris 11, Cert# 2013008. IBM Power 760 (8 processors, 48 cores, 192 threads) 25,488 SAP SD users, 8 x 3.41 GHz IBM POWER7+, 1024 GB memory, DB2 10, AIX 7.1, Cert#2013004. Fujitsu PRIMERGY RX900 S1 (8 processors, 64 cores, 128 threads), 18,700 SAP SD users, 8x Intel Xeon Processor X7560, 1048576 MB memory, SQL Server 2008, Windows Server 2008 R2 Datacenter Edition, Cert# 2010050.

SPECint: SPEC and the benchmark names SPECfp and SPECint are registered trademarks of the Standard Performance Evaluation Corporation. Results as of March 26, 2013 from www.spec.org. SPARC T5-8: 3750 SPECint_rate2006, 3490 SPECint_rate_base2006, 3020 SPECfp_rate2006, 2770 SPECfp_rate_base2006; SPARC T5-1B: 467 SPECint_rate2006, 436 SPECint_rate_base2006, 369 SPECfp_rate2006, 350 SPECfp_rate_base2006. Results as of September 13, 2013 from www.spec.org. IBM Power 780: 6130 SPECint_rate2006, 4460 SPECint_rate_base2006. SPARC Enterprise M9000: 1720 SPECint_rate2006, 1570 SPECint_rate_base2006. IBM power 795: 6150 SPECint_rate2006, 5330SPECint_rate_base2006. IBM Power 780: 2770 SPECint_rate2006, 2420 SPECint_rate_base2006. IBM Power 770: 2140 SPECint_rate2006, 1930 SPECint_rate_base2006. HP Proliant DL980 G7: 2180 SPECint_rate2006, 2070 SPECint_rate_base2006. IBM Power 780: 3730 SPECint_rate2006, 2830 SPECint_rate_base2006; IBM Power 780: 2530 SPECint_rate2006, 2300 SPECint_rate_base2006; IBM Power 780: 652 SPECint_rate2006, 586 SPECint_rate_base2006

SPECjbb2013: SPEC and the benchmark name SPECjbb are registered trademarks of Standard Performance Evaluation Corporation (SPEC). Results as of 11/20/2013, see http://www.spec.org for more information. SPARC T5-2 75,658 SPECjbb2013-MultiJVM max-jOPS, 23,334 SPECjbb2013-MultiJVM critical-jOPS. SPARC T4-2 34,804 SPECjbb2013-MultiJVM max-jOPS, 10,101 SPECjbb2013-MultiJVM critical-jOPS. HP ProLiant DL560 Gen8 67,850 SPECjbb2013-MultiJVM max-jOPS, 16,199 SPECjbb2013- MultiJVM critical-jOPS. HP ProLiant ML350p Gen8 41,954 SPECjbb2013-MultiJVM max-jOPS, 15,540 SPECjbb2013-MultiJVM critical-jOPS. HP ProLiant ML310e Gen8 12,453 SPECjbb2013-MultiJVM max-jOPS, 3,925 SPECjbb2013-MultiJVM critical-jOPS. HP ProLiant DL980 G7 106,141 SPECjbb2013-MultiJVM max-jOPS, 23,268 SPECjbb2013-MultiJVM critical-jOPS. HP ProLiant BL 460c G8 56,367 SPECjbb2013-MultiJVM max-jOPS, 30,078 SPECjbb2013-MultiJVM critical-jOPS. HP ProLiant DL560 Gen8 66,007 SPECjbb2013-MultiJVM max-jOPS, 18049 SPECjbb2013-MultiJVM critical-jOPS. SPARC T5-2 81,084 SPECjbb2013-MultiJVM max-jOPS, 39,129 SPECjbb2013-MultiJVM critical-jOPS.

TPC-H: TPC Benchmark, TPC-H, QphH, QthH, QppH are trademarks of the Transaction Processing Performance Council (TPC). Results as of 6/7/13, prices are in USD. SPARC T5-4 www.tpc.org/3288; SPARC T4-4 www.tpc.org/3278; IBM Power 780 www.tpc.org/3277; Cisco UCS C420: http://www.tpc.org/3291 ; SPARC Enterprise M9000: http://www.tpc.org/3262

TPC-C: TPC Benchmark C, tpmC, and TPC-C are trademarks of the Transaction Processing Performance Council (TPC). SPARC T5-8 (8/128/1024) with Oracle Database 11g Release 2 Enterprise Edition with Partitioning, 8,552,523 tpmC, $0.55 USD/tpmC, available 9/25/2013. IBM Power 780 (2/8/32) with IBM DB2 9.5, 1,200,011 tpmC, $0.69 USD/tpmC, available 10/13/2010. IBM Power 780 (24/192/768) with IBM DB2 9.7, 10,366,254 tpmC, $1.38 USD/tpmC, available 10/13/2010. Source: http://www.tpc.org/tpcc, results as of 9/13/2013.

IBM rPerf and Performance Reports: http://www-03.ibm.com/systems/power/hardware/reports/system_perf.html