ArcGIS Enterprise: Performance and Scalability Testing Methodologies Andrew Sakowicz, [email protected], Professional Services Motivation Growing complexity of ArcGIS Enterprise Requires dependable infrastructure

Certificates Load balancer Firewall ArcGIS Web Adaptor Portal for ArcGIS ArcGIS Storage with immediate consistency ArcGIS Data Store Identify typical cases

Overload System: -users -services Identify typical cases

Unstable Infrastructure: -Network -NAS -VMWare Identify typical cases

Bottlenecks: -configuration -maintenance -workflows Administration Challenges

• Multiple administrators • Multiple disparate monitoring/diagnostic tools • Data collected in a reactive fashion: on demand and for limited time • Correlation of data with different timestamp is difficult • ArcGIS administrators do not have access to all tools, data and reports • Challenging to quickly identify the root cause and take appropriate measures When problems arise, what is the root cause? Definitions Performance

• Speed, e.g. response time (seconds) Scalability

• The ability to increase output and maintain acceptable performance Capacity

• The maximum level of output the system can produce, e.g. • X cars/sec • X maps/sec

At capacity Over capacity Bottleneck

• Resource(s) limiting the performance or capacity

Not bottleneck bottleneck Think of : Lanes -as CPU processor Toll -as ArcGIS Server instances Cars -as map requests Step Load and Response Time

Step Load (users)

Response Time (sec)

time Throughput (request/hr)

Step Load (users)

Throughput(req/hr)

Response Time (sec)

time Resource utilization: CPU, Memory, Network

Step Load (users)

Throughput(req/hr)

CPU Utilization (%)

Network used (Mbps)

Response Time (sec)

Memory used (Mb)

time Capacity

User load

Throughput(req/hr)

CPU Utilization (%)

Network used (Mbps)

Response Time (sec)

Capacity (~ 85% utilization) Memory used (Mb)

Content length (bytes)

Time Process and tools Process and tools Esri tools Process and tools Esri tools Tools download location

• ArcGIS Monitor - https://my.esri.com/

• Others - http://www.arcgis.com - owner:EnterpriseImp - Show ArcGIS Desktop Content Performance Testing Tuning methodology Profile each tier starting from the top

Browser Total Response t1 t2 Time (t1-t2)

Web Server

Wait Time ArcGIS Server

Usage Time ArcSOC

Search & ArcSDE/DBMS Retrieval Time Fiddler Fiddler measurement approximately 5.2 seconds Mxdperfstat http://www.arcgis.com/home/item.html?id=a269d03aa1c840638680e2902dadecac Oracle Trace Compare elapsed time

Elapsed time slightly changed due to different test runs Oracle Execution plan

Inefficient spatial index Testing process

Application

GIS Services

Infrastructure: Hardware and Software Test tools feature comparison

Tool Cost Learning OS Metrics GIS Data GIS Test Curve Generation Automation

Load Runner High High Windows/ No No

Visual Studio Medium High Windows No No

JMeter Free High Requires additional No No plugin

System Test Free Low Windows/Linux Yes Yes

Tech Support by Esri PS as part of consulting support Performance testing

• Identify bottlenecks • Determine system capacity • Demonstrate performance SLA

Value System Test Tool features ArcGIS Monitor End to end Actionable information Alerts: actionable information Starting point for troubleshooting

Analyze Available in 10.6.1 Charts and Stats Details Source alert groups Charts and stats Details Availability Aggregated view based on Critical alerts

• Downtime defined only by Critical alerts

36 Categories End to end Web

• Response Time • HTTP code ArcGIS

• Server • Portal • GeoEvent • GeoAnalytics • ArcSOC Optimizer

Additional portal stats coming soon, e.g. dependent items, views per item, users ArcGIS: WebGIS Health

• Severity • Healthy ArcSOC Optimizer Setting min / max instances across 100 to 1000s of services in dynamic environments is challenging Database

• ArcGIS Datastore • Oracle • SQL Server

AWS RDS and SQL Azure coming soon Infrastructure CPU, Memory, Disk, Network, Load Balancer stats Monitor network between ArcGIS components Critical to have communication to config store and nodes

Web Server

Web Adapter

ArcGIS Server Usage User stats

• Transactions • Response Time • HTTP Codes GeoInfo Requests per user IP Get Started https://enterprise.arcgis.com/en/monitor/latest/get-started/arcgis-monitor-system-requirements.htm Demo site – the latest features https://arcgismonitor.esri.com/ Gallery: https://arcgismonitor.maps.arcgis.com

• Tutorials • Video • Extensions / Tasks

https://www.arcgis.com/home/group.html?id=58d996e9b40d45439d298d14fa309534&start=1&view=list&sortOrder=asc&sortField=owner#content Use Case: NOAA National Weather Service IDP GIS Program NOAA National Weather Service IDP GIS Program

• 24 x 7 High Availability (HA) • Service Level Agreement (SLA) • High volume of requests High volume of requests

Service Update Frequency Performance Nominal Map/Image requests (Est) Peak Workflow Map/Image requests (Est)

Watches\Warnings and 1 minute \ Light Vector 10K Hr 100K Hr Advisories 10 minutes Daily Global Precipitation Daily Light Raster 1K Hr 10K Hr

Radar (1x1 km base reflectivity) 5 minutes Light Raster 20K Hr 200K Hr

NDFD Temperature 15 minutes Heavy Raster 1K Hr 20K Hr 1 hour Hurricane Tracks/Wind/Surge 10 minutes or less Light Vector 1K Hr 100K Hr

AHPS gauges 15 minutes Light Vector 1K Hr 10K Hr

Flood Outlook Product Daily Light Vector 1K Hr 10K Hr CPC Weather Hazards Daily Light Vector 10K Hr 100K Hr Quantitative Precipitation 15 minutes Light Vector 1K Hr 10K Hr Forecast (QPF)

Weather Features Daily Light Raster 1K Hr 10K Hr Sea Surface Temp Daily Light Raster 1K Hr 10K Hr NDFD Wind Velocity Forecasts 1 hour Light Raster 1K Hr 10K Hr Tools

• Big Brother (NOAA internal system of record) • Check MK (SA’s system of ‘preference’) • System Monitor • System Test • JMeter • mxdperfstat Nagios - Check MK System Monitor KPI currently released as ArcGIS Monitor

GIS Server/Service Key Performance Indicators (KPI) System Monitor (Reports) currently released as ArcGIS Monitor

• Tr/Sec

• Free System Monitor (Reports) currently released as ArcGIS Monitor

• CPU

• Mem System Test

• Individually test services that we expect to receive high traffic to define a service throughput baseline given a certain min/max instance configuration

• Investigate/analyze performance for services that have been alerting as slow or non- responsive in Big Brother

• Develop test extents/parameter files to be exported using custom python scripts so that they can be configured in long running system wide load tests performed by the SA’s using JMeter System Test System Test System Test System Test JMeter mxdperfstat

• Been around for a while and still very useful • Easy to batch script • All service sponsor provided map documents were tested using mxdperfstat to identify cases where poor configuration could impact server performance mxdperfstat

• 1.59 seconds may not seem horrible but this service is expected to see very heavy traffic and accounts for 6 instances on 8 servers. • Improvements to the cartography could make a noticeable difference in overall server performance and/or free resources for other services Summary Process and tools Esri tools ArcGIS Monitor: Optimize Your Enterprise GIS Deployments

• Monitor the health, usage, SLA • Improve performance • Reduce administration costs • Quickly diagnose: - Unstable infrastructure - Overloaded system - Bottlenecks

Scalable and non-intrusive: Used by Esri Managed Cloud Services to Monitor 500+ GIS Servers Print Your Certificate of Attendance Print stations located in the 140 Concourse

Tuesday Wednesday 12:30 pm – 6:30 pm 10:30 am – 5:15 pm GIS Solutions Expo GIS Solutions Expo Hall B Hall B

5:00 pm – 6:30 pm 6:30 pm – 9:00 pm GIS Solutions Expo Social Networking Reception Hall B Smithsonian National Portrait Gallery Please Take Our Survey in the Esri Events App

Download the Esri Events Select the session Scroll down to find the Complete answers app and find your event you attended feedback section and select “Submit”