ArcGIS Enterprise: Performance and Scalability Testing Methodologies Andrew Sakowicz, [email protected], Professional Services Motivation Growing complexity of ArcGIS Enterprise Requires dependable infrastructure
Certificates Load balancer Firewall ArcGIS Web Adaptor Portal for ArcGIS ArcGIS Server Storage with immediate consistency ArcGIS Data Store Database Identify typical cases
Overload System: -users -services Identify typical cases
Unstable Infrastructure: -Network -NAS -VMWare Identify typical cases
Bottlenecks: -configuration -maintenance -workflows Administration Challenges
• Multiple administrators • Multiple disparate monitoring/diagnostic tools • Data collected in a reactive fashion: on demand and for limited time • Correlation of data with different timestamp is difficult • ArcGIS administrators do not have access to all tools, data and reports • Challenging to quickly identify the root cause and take appropriate measures When problems arise, what is the root cause? Definitions Performance
• Speed, e.g. response time (seconds) Scalability
• The ability to increase output and maintain acceptable performance Capacity
• The maximum level of output the system can produce, e.g. • X cars/sec • X maps/sec
At capacity Over capacity Bottleneck
• Resource(s) limiting the performance or capacity
Not bottleneck bottleneck Think of : Lanes -as CPU processor Toll -as ArcGIS Server instances Cars -as map requests Step Load and Response Time
Step Load (users)
Response Time (sec)
time Throughput (request/hr)
Step Load (users)
Throughput(req/hr)
Response Time (sec)
time Resource utilization: CPU, Memory, Network
Step Load (users)
Throughput(req/hr)
CPU Utilization (%)
Network used (Mbps)
Response Time (sec)
Memory used (Mb)
time Capacity
User load
Throughput(req/hr)
CPU Utilization (%)
Network used (Mbps)
Response Time (sec)
Capacity (~ 85% utilization) Memory used (Mb)
Content length (bytes)
Time Process and tools Process and tools Esri tools Process and tools Esri tools Tools download location
• ArcGIS Monitor - https://my.esri.com/
• Others - http://www.arcgis.com - owner:EnterpriseImp - Show ArcGIS Desktop Content Performance Testing Tuning methodology Profile each tier starting from the top
Browser Total Response t1 t2 Time (t1-t2)
Web Server
Wait Time ArcGIS Server
Usage Time ArcSOC
Search & ArcSDE/DBMS Retrieval Time Fiddler Fiddler measurement approximately 5.2 seconds Mxdperfstat http://www.arcgis.com/home/item.html?id=a269d03aa1c840638680e2902dadecac Oracle Trace Compare elapsed time
Elapsed time slightly changed due to different test runs Oracle Execution plan
Inefficient spatial index Testing process
Application
GIS Services
Infrastructure: Hardware and Software Test tools feature comparison
Tool Cost Learning OS Metrics GIS Data GIS Test Curve Generation Automation
Load Runner High High Windows/Linux No No
Visual Studio Medium High Windows No No
JMeter Free High Requires additional No No plugin
System Test Free Low Windows/Linux Yes Yes
Tech Support by Esri PS as part of consulting support Performance testing
• Identify bottlenecks • Determine system capacity • Demonstrate performance SLA
Value System Test Tool features ArcGIS Monitor End to end Actionable information Alerts: actionable information Starting point for troubleshooting
Analyze Available in 10.6.1 Charts and Stats Details Source alert groups Charts and stats Details Availability Aggregated view based on Critical alerts
• Downtime defined only by Critical alerts
36 Categories End to end Web
• Response Time • HTTP code ArcGIS
• Server • Portal • GeoEvent • GeoAnalytics • ArcSOC Optimizer
Additional portal stats coming soon, e.g. dependent items, views per item, users ArcGIS: WebGIS Health
• Severity • Healthy ArcSOC Optimizer Setting min / max instances across 100 to 1000s of services in dynamic environments is challenging Database
• ArcGIS Datastore • Oracle • SQL Server
AWS RDS and SQL Azure coming soon Infrastructure CPU, Memory, Disk, Network, Load Balancer stats Monitor network between ArcGIS components Critical to have communication to config store and nodes
Web Server
Web Adapter
ArcGIS Server Usage User stats
• Transactions • Response Time • HTTP Codes GeoInfo Requests per user IP Get Started https://enterprise.arcgis.com/en/monitor/latest/get-started/arcgis-monitor-system-requirements.htm Demo site – the latest features https://arcgismonitor.esri.com/ Gallery: https://arcgismonitor.maps.arcgis.com
• Tutorials • Video • Extensions / Tasks
https://www.arcgis.com/home/group.html?id=58d996e9b40d45439d298d14fa309534&start=1&view=list&sortOrder=asc&sortField=owner#content Use Case: NOAA National Weather Service IDP GIS Program NOAA National Weather Service IDP GIS Program
• 24 x 7 High Availability (HA) • Service Level Agreement (SLA) • High volume of requests High volume of requests
Service Update Frequency Performance Nominal Map/Image requests (Est) Peak Workflow Map/Image requests (Est)
Watches\Warnings and 1 minute \ Light Vector 10K Hr 100K Hr Advisories 10 minutes Daily Global Precipitation Daily Light Raster 1K Hr 10K Hr
Radar (1x1 km base reflectivity) 5 minutes Light Raster 20K Hr 200K Hr
NDFD Temperature 15 minutes Heavy Raster 1K Hr 20K Hr 1 hour Hurricane Tracks/Wind/Surge 10 minutes or less Light Vector 1K Hr 100K Hr
AHPS gauges 15 minutes Light Vector 1K Hr 10K Hr
Flood Outlook Product Daily Light Vector 1K Hr 10K Hr CPC Weather Hazards Daily Light Vector 10K Hr 100K Hr Quantitative Precipitation 15 minutes Light Vector 1K Hr 10K Hr Forecast (QPF)
Weather Features Daily Light Raster 1K Hr 10K Hr Sea Surface Temp Daily Light Raster 1K Hr 10K Hr NDFD Wind Velocity Forecasts 1 hour Light Raster 1K Hr 10K Hr Tools
• Big Brother (NOAA internal system of record) • Nagios Check MK (SA’s system of ‘preference’) • System Monitor • System Test • JMeter • mxdperfstat Nagios - Check MK System Monitor KPI currently released as ArcGIS Monitor
GIS Server/Service Key Performance Indicators (KPI) System Monitor (Reports) currently released as ArcGIS Monitor
• Tr/Sec
• Free System Monitor (Reports) currently released as ArcGIS Monitor
• CPU
• Mem System Test
• Individually test services that we expect to receive high traffic to define a service throughput baseline given a certain min/max instance configuration
• Investigate/analyze performance for services that have been alerting as slow or non- responsive in Big Brother
• Develop test extents/parameter files to be exported using custom python scripts so that they can be configured in long running system wide load tests performed by the SA’s using JMeter System Test System Test System Test System Test JMeter mxdperfstat
• Been around for a while and still very useful • Easy to batch script • All service sponsor provided map documents were tested using mxdperfstat to identify cases where poor configuration could impact server performance mxdperfstat
• 1.59 seconds may not seem horrible but this service is expected to see very heavy traffic and accounts for 6 instances on 8 servers. • Improvements to the cartography could make a noticeable difference in overall server performance and/or free resources for other services Summary Process and tools Esri tools ArcGIS Monitor: Optimize Your Enterprise GIS Deployments
• Monitor the health, usage, SLA • Improve performance • Reduce administration costs • Quickly diagnose: - Unstable infrastructure - Overloaded system - Bottlenecks
Scalable and non-intrusive: Used by Esri Managed Cloud Services to Monitor 500+ GIS Servers Print Your Certificate of Attendance Print stations located in the 140 Concourse
Tuesday Wednesday 12:30 pm – 6:30 pm 10:30 am – 5:15 pm GIS Solutions Expo GIS Solutions Expo Hall B Hall B
5:00 pm – 6:30 pm 6:30 pm – 9:00 pm GIS Solutions Expo Social Networking Reception Hall B Smithsonian National Portrait Gallery Please Take Our Survey in the Esri Events App
Download the Esri Events Select the session Scroll down to find the Complete answers app and find your event you attended feedback section and select “Submit”