DIMS Administrator Guide Release 0.1.18 David Dittrich Dec 05, 2017 Contents 1 Introduction 3 1.1 Overview.................................................3 2 Referenced documents 5 3 Onboarding Developers 7 3.1 Initial Account Setup...........................................7 3.2 GPG Encryption Keys for Email, etc...................................8 3.3 Creating accounts............................................8 3.4 Installing initial SSH key(s).......................................8 3.5 Remote Account Setup..........................................9 3.6 JIRA Onboarding............................................. 13 4 Installation of DIMS Components on “Bare-metal” 19 4.1 Control and Target Prerequisites..................................... 19 4.2 Setting up a DIMS Developer Laptop.................................. 20 5 Installation of DIMS Components Using Virtual Machines 31 5.1 DIMS on Virtual Machines....................................... 31 5.2 Prerequisites for Instantiating Virtual Machines............................. 32 5.3 VM Build Workflow........................................... 33 5.4 Run Directory Helper Makefile Targets................................. 34 6 Installation of a Complete DIMS Instance 39 6.1 Cluster Foundation Setup........................................ 39 6.2 Bootstrapping User Base......................................... 40 7 Trident 41 7.1 Installing Trident manually....................................... 41 7.2 Installing Trident with Ansible...................................... 46 7.3 Trident Prerequisites........................................... 48 7.4 Install Trident.............................................. 58 7.5 Running Trident............................................. 58 7.6 Using tcli on the command line..................................... 59 7.7 Configuring Trident via web app..................................... 63 7.8 Upgrading configuration across Trident versions............................ 105 7.9 Emails and other non-official documentation.............................. 110 i 8 AMQP and RabbitMQ 113 8.1 RabbitMQ use in DIMS......................................... 113 8.2 Basic Service Administration...................................... 113 8.3 Managing RabbitMQ........................................... 114 8.4 Management with Ansible playbooks.................................. 124 9 RaspberryPi and Docker 125 9.1 Installing HypriotOS w/Docker..................................... 125 9.2 Installing a Persistent Docker Container................................. 128 10 Docker Datacenter 135 10.1 Initial Inquiry............................................... 135 10.2 Docker Trusted Repository Issues.................................... 135 10.3 Further Information........................................... 135 11 Managing Long-running Services 141 11.1 Services using supervisord ..................................... 142 11.2 Services using Upstart.......................................... 142 12 Diagnosing System Problems and Outages 157 12.1 Using dimscli ............................................. 157 12.2 Debugging Vagrant............................................ 164 12.3 Other Tools for Diagnosing System Problems.............................. 167 13 Managing CoreOS with Systemd and Other Tools 171 13.1 State of systemd............................................. 171 13.2 State of systemd units.......................................... 177 13.3 Managing systemd units......................................... 189 14 Managing Virtualbox VMs 191 14.1 Remotely Managing Virtualbox..................................... 191 15 Appendices 195 15.1 Add New Connection to Apache Directory Studio........................... 195 16 Contact 199 17 License 201 ii DIMS Administrator Guide, Release 0.1.18 This document (version 0.1.18) covers issues related to system administration of DIMS components from an adminis- trator’s perspective. Contents 1 DIMS Administrator Guide, Release 0.1.18 2 Contents CHAPTER 1 Introduction This chapter introduces the system administration policies, methodology for configuration file management, automated installation and configuration of DIMS components using Ansible, and use of continuous integration mechanisms used for deployment and testing of DIMS components. This document is closely related to the DIMS Developer Guide v 1.0.0, which covers a number of related tasks and steps that will not be repeated here (rather, will be cross-referenced using intersphinx links.) • All documentation for the DIMS project is written using restructured text (reST) and Sphinx. Section Docu- menting DIMS Components of the DIMS Developer Guide v 1.0.0 covers how to use these tools for producing professional looking and cross-referenced on-line (HTML) and off-line (PDF) documentation. • DIMS software – including Ansible playbooks for installation and configuration of DIMS system components, Packer, Vagrant, and Docker subsystem creation scripts, are all maintained under version control using Git and the HubFlow methodology and tool set. Section Source Code Management with Git of the DIMS Developer Guide v 1.0.0 covers how these tools are used for source code, documentation, and system configuration files. • Changes to source code that are pushed to Git repositories trigger build processes using the Jenkins continuous integration environment. These triggers build and/or deploy software to specified locations, run tests, and/or configure service components. In most cases, Ansible is used as part of the process driven by Jenkins. Section Continuous Integration of the DIMS Developer Guide v 1.0.0 provides an overview of how this works and how to use it in development and testing DIMS components. • System software installation and configuration of DIMS components are managed using Ansible playbooks that are in turn maintained in Git repositories. Only a bare minimum of manual steps are required to bootstrap a DIMS deployment. After that, configuration changes are made to Git repositories and those changes trigger continuous integration processes to get these changes into the running system. Section Deployment and Con- figuration of the DIMS Developer Guide v 1.0.0 covers how to use this framework for adding or managing the open source components that are used in a DIMS deployment. 1.1 Overview This document is focused on the system administrative tasks that are involved in adding open source software com- ponents to the DIMS framework, how to convert installation instructions into Ansible playbooks or Dockerfile 3 DIMS Administrator Guide, Release 0.1.18 instructions that can be used to instantiate a service or microservice, how a complete DIMS instance (i.e., a comple- mentary set of service and microservice components that function together as a coherent system) is installed, config- ured, debugged and/or tuned, and kept in running order over time. 4 Chapter 1. Introduction CHAPTER 2 Referenced documents 1. DIMS Developer Guide v 1.0.0 2. ansibleinventory:ansibleinventory 3. ansibleplaybooks:ansibleplaybooks 4. dimsdockerfiles:usingdockerindims 5. dimsdockerfiles:dockerincoreos 6. dimspacker:dimspacker 7. dimsciutils:dimsciutilities 8. dimssr:dimssystemrequirements 9. DIMS Architecture Design v 2.10.0 10. dittrich:homepage home page. 5 DIMS Administrator Guide, Release 0.1.18 6 Chapter 2. Referenced documents CHAPTER 3 Onboarding Developers This chapter covers the process for onboarding new developers to provide them access to DevOps components nec- essary to work on elements of a DIMS deployment. In short, developers (and system administrators) will need the following: • An account in the Trident portal system for access to email lists, etc. • A GPG/PGP key pair. The public key will be loaded into the Trident portal so others can access the key and so it can be used for encrypted email. • A Google account for OpenID Connect authentication used for single-signon access to internal resources, along with an LDAP database entry that links to this Google account. • SSH public/private key pairs allowing access to Git repositories, Ansible control host, DIMS system compo- nents, etc. • Initial copies of Git repositories used to develop and build a DIMS deployment instance. Once all of these resources have been procured, developers or system administrators are ready to work on a DIMS instance. 3.1 Initial Account Setup The first step in adding a new DIMS developer is getting them set up with an account on our internal ops-trust portal instance. Note: We will transition to using Trident, rather than the old Ops-Trust portal code base initially set up for this project, as soon as we are able. Trident has an internal wiki, so the FosWiki server mentioned here will also be retired. Our FosWiki server has a page that was dedicated to the steps necessary for Provisioning New DIMS Users. 7 DIMS Administrator Guide, Release 0.1.18 Caution: The FosWiki page Provisioning New DIMS Users looks like it may be out of date, or include steps that may not be necessary for just adding a new user. It has a huge number of steps that should be made more streamlined or added to the DIMS web app to simplify the process of adding and removing DIMS users in concert with the ops-trust portal at the center of DIMS. Once the user has been given their password to the ops-trust portal, they need to change their MemberID to match the account name that should be used within DIMS. (E.g., Dave Dittrich may be given the MemberID of davedittrich2475 by the portal, but the desired account name within DIMS subsystems