Java Concurrent Program Performance • Case Studies

1 Copyright © 2012, Oracle and/or its affiliates. All rights Insert Information Protection Policy Classification from Slide 8 reserved. Triage and Diagnostics for E-Business Suite Systems Dave Suri, Project Lead, Applications Performance

• Architecture • Patches/Certified Browsers/JDK versions • JVM Tuning and Tools • Forms Tools (strace/FRD) • Java Concurrent Program Performance • Case studies

• The applications tier connects to the database via JDBC connections • The application connections are pooled, so a database session is used to service requests from multiple users throughout the lifetime of the connection • Java concurrent programs run in their own JVM - each has a special configuration. • Forms runs in Socket mode (Forms Listener) or Servlet mode (HTTP/s to the JVM) • Apache HTTP forwards requests to OC4J through mod_oc4j. OC4J routes them to the right oacore JVM while OPMN helps keep the uptime high with death detection.

Mod_oc4j Lis ten 5 OC4J:oacore OC4J:oafm er 1 4 ` httpd 3 OC4JGroup 2 Apache 7,8 7,8

UIX/ HTML 7,8 Page

Browser

Mod_oc4j Lis ten 5 OC4J:oacore OC4J:oafm er 1 4 ` httpd 3 OC4JGroup 2 Apache 7,8 7,8

UIX/ HTML 7,8 Page

AOL/J Browser 6 RF.jsp 7 9

AOL/J 8 FND/SSO 8 database

OC4J:oacore 8 OC4JGroup

SSO

` 14,15

23 UIX 16-18 BC4J MDS 12

OA.jsp UIX/ database HTML 19-22 OAF Page OC4J:oacore 13

Browser OC4JGroup

• The Forms Listener Servlet is a Java servlet runs inside the OC4J container – The listener servlet delivers the ability to run Forms applications over HTTP – The listener servlet manages the creation of Forms Runtime process (frmweb) for each client and manages any further communications between the client and its designated runtime process. – The main difference is the traffic goes through HTTP/HTTPS and leverages load balancing/security capabilities of the MT. – Does not require a special port to be open

8 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Architecture Overview – Forms Socket Mode • Forms Direct Socket Connection – Client requests forms application – OHS receives requests & sends the request to the forms listener – Listener contacts a forms runtime engine (frmsrv/f60srvm) – frmsrv/f60srvm spawns a runtime process, creates connection to the database & passes the forms params (menu, sid, form name) – forms runtime communicates with the database & gets the forms logic (pl/sql,etc). – forms service sends applet & the response back to client – forms is running, page is rendered and direct connection to runtime process

• Architecture • Patches/Certified Browsers/JDK versions • JVM Tuning and Tools • Forms Tools (strace/FRD) • Java Concurrent Program Performance • Case studies

• Document ID 244040.1 “Recommended Performance Patches for the Oracle E-Business Suite” – Recommended performance patches for all the modules and technology stack components are consolidated in this note • Apply latest Release Update Packs – Release 12.1.3 - Patch 9239090 • Document ID 1080973.1 “Oracle E-Business Suite Release 12.1.3 Readme” – Release 12.1 - Patch 7303030 • Document ID 752619.1 “Oracle E-Business Suite Release 12.1.1 Maintenance Pack Installation Instructions” – Release 12.0 - RUP6/7 Patch 6728000 / 7577660 • Document ID 743368.1 “Oracle E-Business Suite Release 12.1.1 Maintenance Pack Installation Instructions”

• Upgrade to the latest certified technology stack – EBS 12.1 is certified with 11gR2, Oracle Apps Server 10gR3, JRE 6.0. – Document ID 561580.1 • Document ID 380482.1 “Oracle E-Business Suite Release 12 Technology Stack Documentation Roadmap” – Central document for installation, apps, database, security and other resources. • OA Framework Applications – Recommended Patches for Applications: note 275880.1 (Framework Roadmap) • Oracle E-Business Suite Recommended Performance Patches – Document id: 244040.1 “Oracle E-Business Suite Recommended Performance Patches” • For a list of certifications, use Document ID 561580.1 “Oracle E-Business Suite Release 12.1 Release Content Documents”

• Browsers, OAF Pages in R12.x – Firefox 3.6 is supported in 11i, R12 – Windows 7 / Internet Explorer 8 / Firefox 3.6 supported in 11i/R12 – Oracle E-Business Suite 12.1.1 or higher – R12.0 => use R12.ATG_PF.A.DELTA.6 & Forms 10.1.2.3 or higher – 11.5.10 => CU2, 11i.ATG_PF.H Rollup 7 with Document ID 783600.1 – Document ID 1103276.1, 389422.1 (R12), 285218.1 (11i) • R12.1.2 have ajax customizable homepage for increased performance • R12.1.3 available as patch #9239090

• Architecture • Patches/Certified Browsers/JDK versions • JVM Tuning and Tools • Forms Tools (strace/FRD) • Java Concurrent Program Performance • Case studies

• Response Time/JVM Usage • OutOfMemoryErrors • Deadlocks • Some great tools – Jstat : monitor JVM – Eclipse MAT : analyze heap dumps – IBM Thread Dump Analyzer : analyze thread dumps – Visual VM : profile, monitor, dump the JVM – Jconsole : similar to Visual VM except for profiling – Jrockit Mission Control : flight recorder, profiler, dump – Thread Dump Analyzer

15 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. JVM Tuning and Tools Response Time/JVM Usage • First, rule out any SQL issues. – Document ID 357597.1 “How To Generate A SQL Trace In OA Framework For Oracle Applications” • Monitor Request Response Time – Modify $ORA_CONFIG_HOME/10.1.3/Apache/Apache/conf/httpd.conf • LogFormat "%h %T • Logs: $LOG_HOME/ora/10.1.3/Apache/access_log* • If there are no database-related issues, take a look into the JVM – Techniques you can use • Thread dumps – Send to Oracle Support for analysis • GC logs for sizing, Profilers - Send to Oracle Support for analysis

16 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. JVM Tuning and Tools Response Time/JVM Usage • GC tuning – Review the frequency of collections, especially major collections (i.e. Full GC) in the logs for healthy and unhealthy activity • Use jstat tool with the –gc option to review FCGT count • Important columns to pay attention to are survivor spaces, eden and old gen – Jstat column details (observe the grey highlighted columns) • Timestamp, S0C, S1C, S0U, S1U, EC, EU, OC, OU, PC, PU, YGC, YGCT, FGC, FGCT, GCT • 933.9,59904.0,57728.0,0.0,40504.5,230784.0,0.0,699072.0, 666307.1,131072.0,119088.8,31,2.054,1,0.000,2.054

17 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. JVM Tuning and Tools Response Time/JVM Usage • GC tuning – Bigger heaps => Full GC will take longer – Longer GCs => users may experience pauses – Increase the number of JVMs used may provide the ability to handle more users with faster GC times. • Each JVM has a smaller Xmx,Xms memory footprint so GCs will be faster! – Also try to change the JVM collectors used and track the performance with Jconsole – Enable the ParallelGC collector with ParallelGCThreads = 2

• GC tuning – Unhealthy JVM with FullGC activity • 898753.288: [Full GC 825983K->743065K(913024K), 15.8442715 secs] 898772.243: [Full GC 825983K->750548K(913024K), 13.3512933 secs] 898798.124: [Full GC 825983K->731902K(913024K), 13.3466676 secs] 898832.082: [Full GC 825983K->738500K(913024K), 13.1591655 secs] 898857.239: [Full GC 825973K->728250K(913024K), 16.4536985 secs] – Healthy JVM with FullGC activity • 902130.499: [GC 829181K->716211K(935296K), 0.2895464 secs] 902220.498: [GC 833715K->721542K(935296K), 0.3210096 secs] 902400.049: [GC 839046K->725265K(936768K), 0.3448279 secs] 902400.394: [Full GC 725265K->652064K(936768K), 8.5214844 secs] 902455.778: [GC 771040K->661918K(935296K), 0.1618026 secs]

Response Time/JVM Usage Full GC too • GC tuning frequent!!! – Unhealthy JVM with FullGC activity • 898753.288: [Full GC 825983K->743065K(913024K), 15.8442715 secs] 898772.243: [Full GC 825983K->750548K(913024K), 13.3512933 secs] 898798.124: [Full GC 825983K->731902K(913024K), 13.3466676 secs] 898832.082: [Full GC 825983K->738500K(913024K), 13.1591655 secs] 898857.239: [Full GC 825973K->728250K(913024K), 16.4536985 secs] – Healthy JVM with FullGC activity • 902130.499: [GC 829181K->716211K(935296K), 0.2895464 secs] Infrequent 902220.498: [GC 833715K->721542K(935296K), 0.3210096 secs] and shorter 902400.049: [GC 839046K->725265K(936768K), 0.3448279 secs] times! 902400.394: [Full GC 725265K->652064K(936768K), 8.5214844 secs] 902455.778: [GC 771040K->661918K(935296K), 0.1618026 secs]

• Common causes of OutOfMemoryError – Sizing Issues – Heap is undersized for expected load – Memory Leak – Some objects not cleaned up • Strong reached references, connection objects, etc. – Memory Hemorrhage • Reading large data sets consuming vast amounts of memory in any period of time • Connection leaks leading to vast memory/stack allocations • SQL Statements/caching may lead to excess memory used in JDBC libraries

• Out of Memory Heap Dump Example – This will be present in the $LOG_HOME/ora/10.1.3/opmn/default_group~oacore~default_group~1.log • 10067.492: [Full GC 118783K->118783K(124928K), 1.2127460 secs] • 10068.706: [Full GC 118783K->118783K(124928K), 1.2166770 secs] • 10069.924: [Full GC 118783K->117977K(124928K), 1.2416330 secs] • java.lang.OutOfMemoryError: GC overhead limit exceeded • Dumping heap to java_pid8515.hprof ... • Heap dump file created [149076109 bytes in 10.240 secs] • Heap Location cd $INST_TOP/../../../apps/tech_st/10.1.3/j2ee/home ls -l *hprof – -rw------1 oraperf dba 149076109 Jan 26 11:11 java_pid8515.hprof

23 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. JVM Tuning and Tools JVM Tools • JDK 6 – Jconsole – Enable heap dumps to be collected from the attached JVM – Goto java.sun.management | HotSpotDiagnostic | Operations | dumpHeap • P0 : absolute file name (directory + filename) • P1: leave to true so that only reachable objects are dumped • Use any binary hprof heap analyzer to analyze the heap dump and see which objects take up the most space • JDK 6u7 and above has VisualVM – Does thread/heap dumps, profiling, graphs

24 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. JVM Tuning and Tools JVM Tools • Jstat – Shows the JVM heap internals – Shows Timestamps and Capacity/Utilization for • Survivor Spaces – where live objects become old enough to copy • Eden – new objects are created here • Old Generation – long lived objects are kept here • Young Generation – composed of eden + 2 survivor spaces • Perm Generation – native heap and other non-heap objects – Shows the Full GC and Young GC collection count – To get the data from PID nnnnn every 60s for 2 hours • jstat -gc -t nnnnn 60000 120 > jstat_gcutil_pid_31041 &

• Architecture • Patches/Certified Browsers/JDK versions • JVM Tuning and Tools • Forms Tools (strace/FRD) • Java Concurrent Program Performance • Case studies

• Forms – Next, use strace (linux), trace (solaris) to attach to the pid: • strace -o formsR12_frmweb.log -p 22874 • strace -o formsR12_frmsrv.log -p 17347 – The logs will show the activity of the forms server as well as the listener while the user is performing their actions. – FRD, Tracing, Logging for Forms • Document ID 438652.1 • FRD gives clues into forms code execution, parameters, variables, item values – Along with strace, FRD can provide valuable clues into researching issues.

• Architecture • Patches/Certified Browsers/JDK versions • JVM Tuning and Tools • Forms Tools (strace/FRD) • Java Concurrent Program Performance • Case studies

• Observe the concurrent program log file after adding –verbose:gc to see if the JVM is healthy or is it suffering from low memory • Increase the Memory when you encounter OOME in the logs – location: Concurrent -> Program -> Define • Remember it runs in a separate JVM from oacore • All the regular java support utilities will work (jstat, eclipse mat, tda, etc) • Can specify regular JVM commands in the define field to observe the JCP behaviour during the run

• Architecture • Patches/Certified Browsers/JDK versions • JVM Tuning and Tools • Forms Tools (strace/FRD) • Java Concurrent Program Performance • Case studies

• Issue – Slow Performance with Rules Enabled vs Rules Disabled • Tool Used – IBM Thread Dump Analyzer • Analyzes thread dumps • Can compare multiple thread dumps • Shows different thread states with different colors • Primary area of concentration is : AJPRequestHandler- HTTPThreadGroup-* • These thread groups are the Apache/AJP Requests coming to oacore • Solution – Thread dumps showed JVM waiting for DB to send results. This eventually lead to the creation of a cache to improve Rules Enabled performance

• Issue – Error saving a service request • Tools Used –Forms FRD • FRD showed a NULL being passed to a variable named ‘control.mfg_org_id’ – Executing USER_EXIT Built-in: – In Argument 0 - Type: String Value: FND FFLEX PRE-UPDATE CSXSRISR, 17, Built-In, Entry, -1424992664, USER_EXIT – Analyzed the CSXSRISR_LIBRARY.Init() method to find the NULL – Analyzed the SR_CONTACTS.PRE_UPDATE to find how the NULL was passed to the control.mfg_org_id • Solution – Product team notified of the issue and a test case was given to isolate the affected code

• Issue Program – Out of Memory error experienced when running a Java Concurrent Program • Tool Used – Eclipse MAT • Analyzed heap dump • Showed which package consumes significant percentage of the heap • Helps derive clues and questions as to what is happening in the tech stack layers • Can see what objects are occupying space and their values • Solution – The Java Concurrent Program’s max JVM memory was increased to 1GB to allow the program to finish successfully

Patches 244040.1 Oracle E-Business Suite Recommended Performance Patches 561580.1 Oracle E-Business Suite Release 12.1 Release Content Documents Roadmap 380482.1 Oracle E-Business Suite Release 12 Technology Stack Documentation Roadmap 275880.1 Oracle Application Framework Release 11i Documentation Road Map

Browsers 1103276.1 Is Windows 7 Certified With Oracle E-Business Suite? 389422.1 Recommended Browsers for Oracle E-Business Suite Release 12 285218.1 Recommended Browsers for Oracle E-Business Suite 11i Setup 362851.1 Guidelines to setup the JVM in Apps E-Business Suite 11i and R12 Diagnosis 357597.1 How To Generate A SQL Trace In OA Framework For Oracle Applications 1160285.1 R12: Forms Runtime Diagnostics (FRD), Tracing And Logging For Forms In Oracle Applications ATG 438652.1 Application Technology Group (ATG) Product Information Center

40 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 41 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 42 Copyright © 2012, Oracle and/or its affiliates. All rights Insert Information Protection Policy Classification from Slide 8 reserved.