Integration Service Platform
Total Page:16
File Type:pdf, Size:1020Kb
Integration Service Platform Version 2.0.0 Table of Contents Introduction to the Integration Service 9 What is the Integration Service? 10 What is a Step? 11 What is an Integration Application? 13 What is a Configuration Object? 14 Creating and Saving Integration Service Components 14 Elements of the Integration Service User Interface 14 Logging In and Out of the Integration Service User Interface 15 Integration Service Pages 16 Additional Navigation 16 Monitoring the Integration Service on the Dashboard Page 18 Installing and Configuring the Integration Service 19 Integration Service Architecture 20 Integration Service Container Architecture 20 Integration Workflow 21 High-Availability, Disaster Recovery, and Proxy Architecture 21 Prerequisites for the Integration Service 22 System Requirements 23 Hardened Operating System 24 Installing the Integration Service 24 Installing the Integration Service via ISO 24 Installing the Integration Service via RPM 27 Upgrading the Integration Service 30 Running the Upgrade Script 31 Upgrading Manually 32 Step 1. Upgrading Host Packages and Python 3.6 32 Step 2. Upgrading to Oracle 7.3 or Later 33 Step 3. Upgrading to Docker 18.09.2 or later 34 Installing Docker in Clustered Configurations 35 Step 4. Installing the Integration Service 2.0.0 RPM 35 Troubleshooting Upgrade Issues 36 To roll back to a version before Integration Service 2.0.0 36 Cannot access the Integration Service or an Internal Server Error occurs 36 After upgrading, the syncpack_steprunner fails to run 36 Licensing for the Integration Service 37 Configuring Additional Elements of the Integration Service 40 Setting a Hard Memory Limit in Docker 40 Setting a Soft Memory Limit in the Worker Environment 40 Changing the Integration Service Password 41 Configuring a Proxy Server 42 Configuring Security Settings 42 Changing the HTTPS Certificate 42 Using Password and Encryption Key Security 43 Integration Service Management Endpoints 44 Flower API 44 Couchbase API 45 RabbitMQ 46 Docker Swarm Visualizer 46 Docker Statistics 47 Configuring the Integration Service for High Availability 49 Types of High Availability Deployments for the Integration Service 50 Standard Single-node Deployment (1 Node) 50 Requirements 51 Risks 51 Configuration 51 Standard Three-node Cluster (3 Nodes) 51 Requirements 52 Risks 52 Mitigating Risks 52 Configuration 52 3+ Node Cluster with Separate Workers (4 or More Nodes) 55 Requirements 56 Worker Node Sizing 56 Risks 56 Mitigating Risks 57 Configuration 57 3+ Node Cluster with Separate Workers and Drained Manager Nodes (6 or More Nodes) 57 Requirements 58 Risks 58 Configuration 58 Additional Deployment Options 59 Cross-Data Center Swarm Configuration 59 Additional Notes 60 Requirements Overview 60 Docker Swarm Requirements for High Availability 61 Couchbase Database Requirements for High Availability 62 RabbitMQ Clustering and Persistence for High Availability 62 RabbitMQ Option 1: Persisting Queue to Disk on a Single Node (Default Configuration) 62 RabbitMQ Option 2: Clustering Nodes with Persistent Queues on Each Node 63 Checking the Status of a RabbitMQ Cluster 64 Preparing the Integration Services for High Availability 65 Configuring Clustering and High Availability 65 Automating the Configuration of a Three-Node Cluster 66 Configuring Docker Swarm 66 Configuring the Couchbase Database 67 Code Example: docker-compose-override.yml 70 Scaling iservices-contentapi 73 Manual Failover 73 Additional Configuration Information 75 Exposing Additional Couchbase Cluster Node Management Interfaces over TLS 75 HAProxy Configuration (Optional) 76 Known Issues 77 Docker Network Alias is incorrect 77 Docker container on last swarm node cannot communicate with other swarm nodes 77 Couchbase service does not start, remains at nc -z localhost 77 Couchbase-worker fails to connect to master 78 Couchbase rebalance fails with "Rebalance exited" error 78 When setting up a three-node High Availability Couchbase cluster, the second node does not appear 78 The Integration Service user interface fails to start after a manual failover of the swarm node 78 The Integration Service user interface returns 504 errors 78 NTP should be used, and all node times should be in sync 78 Example Logs from Flower 79 Managing Users in the Integration Service 80 Configuring Authentication with the Integration Service 81 User Interface Login Administrator User (Default) 81 Basic Authentication Using a REST Administrator User (Default) 82 User Interface Login Using a Third-party Authentication Provider 82 OAuth Client Authentication Using a Third-party Provider 84 Basic Authentication Lockout Removal 84 Role-based Access Control (RBAC) Configuration 85 Assigning a Role to a Specific User 85 Assigning Roles to a Specific User Group 85 Viewing User and Group Information 85 Changing Roles and Permissions 85 Configuring Authentication Settings in the Integration Service 86 User Groups, Roles, and Permissions 86 Creating a User Group in the Integration Service 87 Managing SyncPacks 90 What is a SyncPack? 91 Viewing the List of SyncPacks 92 Importing and Installing a SyncPack 93 Importing a SyncPack 94 Installing a SyncPack 94 Default SyncPacks 95 Base Steps SyncPack 95 System Utils SyncPack 96 Managing Integration Applications 97 Viewing the List of Integration Applications 98 The Integration Application Window 100 Editing an Integration Application 102 Creating Integration Applications and Steps 103 Creating an Integration Application 103 Creating a Step 105 Aligning a Configuration Object with an Integration Application 106 Running an Integration Application 108 Viewing Previous Runs of an Integration Application 110 Scheduling an Integration Application 113 Viewing Integration Service System Diagnostics 115 Backing up Data 118 Creating a Backup 119 Restoring a Backup 122 Viewing a Report for an Integration Application 125 Managing Configuration Objects 127 What is a Configuration Object? 128 Creating a Configuration Object 130 Editing a Configuration Object 133 Viewing Logs in the Integration Service 135 Logging Data in the Integration Service 136 Local Logging 136 Remote Logging 136 Viewing Logs in Docker 136 Logging Configuration 137 Integration Service Log Files 137 Accessing Docker Log Files 137 Accessing Local File System Logs 138 Understanding the Contents of Log Files 138 Viewing the Step Logs for an Integration Application 138 Removing Logs on a Regular Schedule 139 Using SL1 to Monitor the Integration Service 141 Monitoring the Integration Service 142 Configuring the Docker PowerPack 143 Configuring the Integration Service PowerPack 145 Configuring the PowerPack 146 Events Generated by the PowerPack 147 Configuring the Couchbase PowerPack 148 Configuring the RabbitMQ PowerPack 150 Stability of the Integration Service Platform 153 What makes up a healthy SL1 system? 153 What makes up a healthy Integration Service system? 153 Troubleshooting the Integration Service 155 Initial Troubleshooting Steps 156 Integration Service 156 ServiceNow 156 Resources for Troubleshooting 156 Useful Integration Service Ports 156 Helpful Docker Commands 157 Viewing Container Versions and Status 157 Restarting a Service 157 Stopping all Integration Service Services 157 Restarting Docker 157 Viewing Logs for a Specific Service 158 Clearing RabbitMQ Volume 158 Viewing the Process Status of All Services 158 Deploying Services from a Defined Docker Compose File 159 Dynamically Scaling for More Workers 159 Completely Removing Services from Running 159 Helpful Couchbase Commands 159 Checking the Couchbase Cache to Ensure an SL1 Device ID is Linked to a ServiceNow Sys ID 159 Clearing the Internal Integration Service Cache 160 Accessing Couchbase with the Command-line Interface 160 Useful API Commands 161 Getting Integrations from the Integration Service API 161 Creating and Retrieving Schedules with the Integration Service API 161 Diagnosis Tools 162 Identifying Why a Service or Container Failed 162 Step 1: Obtain the ID of the failed container for the service 162 Step 2: Check for any error messages or logs indicating an error 163 Step 3: Check for out of memory events 163 Troubleshooting a Cloud Deployment of the Integration Service 164 Identifying Why an Integration Application Failed 165 Determining Where an Integration Application Failed 165 Retrieving Additional Debug Information (Debug Mode) 165 Frequently Asked Questions 166 What is the first thing I should do when I have an issue with my Integration Service? 166 Why do I get a "Connection refused" error when trying to communicate with Couchbase? 167 How do I remove a schedule that does not have a name? 167 How do I identify and fix a deadlocked state? 168 How can I optimize workers, queues, and tasks? 170 Why do I have client-side timeouts when communicating with Couchbase? 172 What causes a Task Soft Timeout? 173 Why are incident numbers not populated in SL1 on Incident creation in ServiceNow? 173 Why am I not getting any Incidents after disabling the firewall? 173 Why are Incidents not getting created in ServiceNow? 173 What if my Incident does not have a CI? 173 How can I point the "latest" container to my latest available images for the Integration Service? 174 How do I manually create a backup of my Integration Service system, and how do I manually restore? 174 What do I do if I get a Code 500 error when I try to access the Integration Service user interface? 175 What are some common examples of using the iscli tool? 176 How do I view a specific run of an integration application on the Integration Service?