Automating ArcGIS Deployments Using Chef Cherry Lin ([email protected]) Bill Major ([email protected]) David Cordes ([email protected]) Slides: http://bit.ly/2uiEG0p Overview • What, Why, When? • Esri’s Chef Cookbooks • Easy Cooking Demo • Best Practices • Case Study: Esri What, Why, When What is Automation and Why • The linking of disparate systems and software in such a way that they become self-acting or self-regulating. • Reliable, repeatable and testable deployments and upgrades - DEV → TEST → PROD - Moving between Data Centers - Faster Disaster Recovery • Quick configuration with disposable machines and rapid auto-scaling • Automate and become more efficient at administration - Focus more on GIS tasks - Prepare for larger scale and clouds - Ready-to-use tools with minimum requirements to start. Operate Software Org Operation (HA, DR) Administration Content Software Configuration Management Software Analytics Installation GIS Infrastructure Automation Software Operation (HA, DR) Software Configuration Software Installation Software Setups, Enterprise Builder, Chef, Cloud formation, Cloud Builder GIS Workflow Automation Org Administration Content Management Analytics Python API for ArcGIS, WebGISDR, Admin API, ArcPy, R-bridge for ArcGIS GIS Infrastructure Automation Software Operation (HA, DR) Software Configuration Software Installation Software Setups, Enterprise Builder, Chef, Cloud formation, Cloud Builder ArcGIS Enterprise Software Setups = + + + ArcGIS Portal ArcGIS ArcGIS ArcGIS Web Adaptor for ArcGIS Server Data Store Enterprise ArcGIS Enterprise Software Setups • No need to learn 3-party technology. • Deploy exact the way you want • Other Esri software • Time consuming • More manual steps • Manual configurations following different docs • One time task • For customized deployments • Not recommended for highly available and scalable deployments ArcGIS Enterprise Builder ArcGIS Portal Web Adaptor for ArcGIS ArcGIS Hosting Web Adaptor Server ArcGIS Data Store (relational + tile cache) ArcGIS Enterprise Builder • All-in-one deployment • One GUI setup, 5 or 6 parameters • Easy to use • A lot less configurations • One deployment pattern for one machine quick start • Not for a system that needs quickly redeployed • Not a DR deployment? ArcGISIn the Cloud Enterprise Software Setups • AWS - Esri AMIs - Esri CloudFormation Templates - Python and Powershell scripts •Microsoft Azure - Esri Images - Cloud Builder •Highly recommend using our Images and tools - Customization based on our Images and tools - Details: AWS and Azure sessions ArcGISEsri ArcGIS Enterprise Chef Cookbooks Software Setups • IT standard automation framework • Automate all that you can manually do • Deploy exact the way you want. You have full control. •Other Esri software •Require small up-front investments •Reliable, repeatable and testable - Documented deployments - High Availability and Disaster Recovery deployments - Move between Data Centers - Potential scale-ups - Automate upgrades GIS Infrastructure Automation ArcGIS Enterprise Software Setups Enterprise Software Chef Builder Setups ArcGIS Cookbooks Quick start Reliable Easy to use Repeatable Typical deployment pattern Testable Simple deployment Documented deployment Limited control Full control Not for repeatable deployments Esri’s Chef Cookbooks What is Chef • Chef - One of the most popular IT automation platforms - Supported on Windows and Linux - Provide tools to take on configuration files • Chef Components - Cookbook - Attribute - Chef-Server vs. Client • How it works - Install ready-to-use cookbooks and define the configuration - Run chef with the configuration file ArcGIS Cookbook Recipe • Recipe - Attributes - Action • Property Settings: Attributes • Run-list Define You Own Attributes Define You Own Run List Steps to get started with a single machine deployment with Chef • Check out Esri Github Chef Cookbooks website – v.3.1 just released - https://github.com/Esri/arcgis-cookbook/wiki/Get-Started-With-ArcGIS-Cookbook • Install the Chef Client (v. 12.5.1 or later) • Copy ArcGIS Cookbooks to server and unzip to c:\chef - https://github.com/Esri/arcgis-cookbook/releases/tag/v3.1.0 • Stage Setups and License Files • Edit JSON configuration file to meet your own needs - c:\chef\roles\webgis-windows.json • Run chef-solo from command prompt ‘As Administrator’ - chef-solo -j C:\chef\roles\webgis-windows.json Easy Cooking Demo Supported ArcGIS Software • ArcGIS Enterprise -Portal for ArcGIS -ArcGIS Server - Enabled Roles: Hosting, Image Server, GeoAnalytics, GeoEvent Server -ArcGIS Data Store - General, spatialtemporal -ArcGIS Web Adaptor • And other Esri Software: - ArcGIS Pro - ArcGIS Insight - ArcGIS Desktop - ArcGIS License Manager Support ArcGIS Enterprise Deployment Patterns Best Practices ArcGISDistributed Enterprise Deployment Software Setups • Running Chef on different machines in order from a workstation machine - Different property files - Runs in order as required by ArcGIS Enterprise •Chef Server - Central repository - Assign roles to every machine - Run the recipes/updates accordingly - Setting up -By yourself or -Hosted Solution ArcGISUpgrade toEnterprise 10.5 or 10.5.1 Software Setups • Required properties -New Version # -New Setups -New License File •Recommend to use Chef deploy before upgrade -Easier to get the properties right ArcGISWork with Enterprise Other Automation Software Tools Setups • Esri ArcGIS Chef Cookbooks is ArcGIS deployment tool • ArcGIS Infrastructure part of your automation • Work with GIS workflow automation tools -Python API for ArcGIS -WebGISDR tool • Other software deployment automation tools • Whole system infrastructure automation -High Availability and Disaster Recovery deployment -Moving to different data centers ArcGISChef Deployment Enterprise Options Software Setups ArcGISDisconnected Enterprise Environment Software Considerations Setups • Make sure you have proper Esri license files staged for use • Software setups in a common location for access • Setting up your own Chef Server • Having Chef Client installation staged for installation ArcGISTricks and Enterprise Tips Software Setups • Consider creating a base image for some environment - Have some processes done, e.g, setups - Have some components baked in, e.g. base map data, security updates, etc. •Chef Client version 12 •Set password in an environment variable •“-l debug” for debug •Attributes value of True or False, no “”. e.g. “configure_autostart”: true •ArcGIS Cookbooks support enabling hosting server role and GeoAnalytics Server role. •Support security configuration for ArcGIS Server. ArcGISKey Points Enterprise about ArcGIS Software Chef Automation Setups • Automate Your Manual Work • Doesn’t require programming skills • Need plan ahead -Design -Configuration -Testing •Repeatable -Success in production environment -Less downtime in production -Faster recovery Case Study : Esri ArcGIS Enterprise : New Software Daily Many software changes Daily testing prevents each day. accumulation complexity. Test Automation : Not just automated functional testing Configuration automation • Software installed and configured many times per build • Many platforms • Many software components • Many architectures • Limited resources Hundreds of Scripts • Every tester had their own configuration script - Python - .Net - Java - Perl - Batch files - Etc • Wasted effort • Hard to maintain ArcGIS Cookbooks to the Rescue Solution: Use our cookbooks • Tested • Reliable • Maintained Challenge #1: But I already have a script…. “Not invented here” • Resistance to moving away from existing scripts. • Time eventually removed this barrier from most of our test teams. Challenge #2: But I don’t have time to learn a new technology... Training • In groups • One-on-one • Troubleshooting alias Outcome: Successful adoption, focus on the functionality Has taken time but we spend • Less time maintaining and troubleshooting scripts. • More time focused on functional testing. Multi-Machine Configuration Step 1: Prepare the image Multi-Machine Architecture Command Machine Tomcat + Web Adaptor test-datastore.dev.geocloud.com Hosting Portal Server Registered Data Store test-portal.dev.geocloud.com test-server.dev.geocloud.com test-datastore.dev.geocloud.com Baked an Image in Three Easy Steps Install Chef Copy Cookbook System Requirements Multiple Machine Automation Step 2: Set the Parameters One Deployment - Many Machines - One Set of Parameters Portal Server Data Store Web Server Parameters Multiple Machine Automation Step 3: Create Role-Specific Runlists Run Lists - What is Run On Each Machine Portal Server Data Store Web Server Parameters Run Lists - What is Run On Each Machine Portal Server Data Store Web Server Parameters Run Lists - What is Run On Each Machine Portal Server Data Store Web Server Parameters Run Lists - What is Run On Each Machine Portal Server Data Store Web Server Parameters Run Lists - Federation - Back to Portal Portal Server Data Store Web Server Parameters Multiple Machine Automation Step 4: Create Shell Script (Batch File) Script Writing for Multiple Machines Command Machine ● Scripts stored on command machine ● Copied to test machines at deploy time ● Run with ssh Multiple Machine Automation Step 5: Run Deployment Script Run: Multi-Machine Configuration in 30 minutes Q&A and Slides Slides: http://bit.ly/2uiEG0p .