Clouder Documentation Release 1.0

Yannick Buron

May 15, 2017

Contents

1 Getting Started 3 1.1 Odoo installation...... 3 1.2 Clouder conﬁguration...... 4 1.3 Services deployed by the oneclick...... 6

2 Connect to a new node 9

3 Images 13

4 Applications 15 4.1 Application Types...... 15 4.2 Application...... 16

5 Services 21

6 Domains and Bases 25 6.1 Domains...... 25 6.2 Bases...... 27

7 Backups and Conﬁguration 31 7.1 Backups...... 31 7.2 Conﬁguration...... 33

i ii Clouder Documentation, Release 1.0

Contents:

Contents 1 Clouder Documentation, Release 1.0

2 Contents CHAPTER 1

Getting Started

In this chapter, we’ll see a step by step guide to install a ready-to-use infrastructure. For the example, the base we will create will be another Clouder.

Odoo installation

This guide will not cover the Odoo installation in itself, we suggest you read the installation documentation on the ofﬁcial website. You can also, and it’s probably the easier way, use an Odoo Docker image like https://hub.docker.com/ _/odoo/ or https://hub.docker.com/r/tecnativa/odoo-base/ Due to the extensive use of ssh, Clouder is only compatible with Linux. Once your Odoo installation is ready, install the paramiko, erppeek and apache-libcloud python libraries (pip install paramiko erppeek apache-libcloud), download the OCA/Connector module on Github and the Clouder modules on Github and add them in your addons directory, then install the clouder module and clouder_template_odoo (this module will install a lot of template dependencies, like postgres, postﬁx etc...).

3 Clouder Documentation, Release 1.0

Clouder conﬁguration

The ﬁrst thing to do is to set the sysadmin email in the Clouder conﬁguration.

You can also configure here how Clouder will deploy the container. You can either use Docker Engine (for single node) or Docker Swarm mode (for multiple node), see Configuration for more information about the settings and actions available in configuration page. Then you have to create a new Environment. An environment shall be seen as a project, this is the object where access right are based upon. All services/bases are linked to an environment, you’ll usually create one for each customers. So first we’ll create the main environment, which will deploy all the core services.

4 Chapter 1. Getting Started Clouder Documentation, Release 1.0

You’ll also need to create a Domain. The ﬁrst base you’ll deploy will be accessible under this domain.

Finally you have to conﬁgure the node you want to connect. See how to connect a node for more explanation.

1.2. Clouder conﬁguration 5 Clouder Documentation, Release 1.0

In the node form, click on Test Connection to check there is no problem. Also make sure your node is conﬁgured as the DNS nameserver of your domain. If everything is ok, select the Odoo oneclick, assign the critical port and click on deploy to launch the oneclick deploy. This will deploy all on your node all the services you need for your infrastructure, with an example of Odoo service with his database. Keep an eye on your Odoo logs because you’ll see all executed command there, and see if something wrong happen. You can also afterward check the log in the node record to see them.

Theses logs can be seen on all objects in Clouder, and are generated each time an action is perform. They are the best way for you to understand how each action work, and see if something wrong happened.

Services deployed by the oneclick

The oneclick will deploy 5 root services.

6 Chapter 1. Getting Started Clouder Documentation, Release 1.0

• The backup service, which will store your backup. Of course, we suggest you move this service on another node once in production. • The DNS service, which will resolve the DNS record of your infrastructure. You can either use a bind container on your node for this (deployed by the oneclick), or a cloud provider _. • The mail service, which will redirect outgoing and incoming emails from your containers. • The proxy service, which will redirect incoming http request to the correct service. It also take care of https encryption and LetsEncrypt cert generation. • The Odoo/Clouder service, which will deploy the Postgres database and the Odoo service. The example base deploy by the oneclick will be linked to this service. Services has a notion of parent/child, when a root service is deploy it will also deploy all child services. You can see them by removing the root ﬁlter.

Only children on the lowest level are containers actually deployed on your node.

1.3. Services deployed by the oneclick 7 Clouder Documentation, Release 1.0

Usually a service can contain theses containers/childs : • A Data container, which only represent the volume created for this service. Removing this container means destroying the data. • A Files container, which represent the version of the application you want to deploy. For example for Odoo, this container will contain the files downloaded from Github, of Odoo but also community module you specified. Replace this container with another one when you want to update you service. • An Exec container, which contain the binary executed by this service. The proxy services usually redirect to the http port of this container. This container shall contain no data, it is linked to Data and Files containers, hence you can destroy and rebuilt it at will. • A SSH container, linked to others containers it allow you to give an SSH access to files to your customers, without having to give them access to the node itself. Note that you can have several layer of services. For example, the root Odoo services contain PostgreSQL and Odoo children, which themselves have Data/Files/Exec/SSH children. Finally, once all services are deployed, the oneclick will also deploy an example base.

This will create a new database on your Odoo service, and will configure it with the credentials and with the modules specified in application (here the clouder module). It will also configure the links to DNS/Proxy/Mail and generate LetsEncrypt certificate. After the deployment, you shall be able to access your example Odoo instance on the url clouder9.mydomain.com. Congratulations! You can now easily create another base or deploy any other application you can find in the clouder_template_* modules, or even create your own images and applications.

8 Chapter 1. Getting Started CHAPTER 2

Connect to a new node

We will see in this chapter how we can conﬁgure the nodes and connect them to your Clouder. Clouder is only compatible with Linux, and the Debian distribution is recommended.

First, if it’s not already done, you shall install openssh on your server and Docker. In Debian, you just have to execute these commands :

For ssh :

apt-get install openssh-client

For Docker :

sudo curl -sSL https://get.docker.com/ | sudo sh

You can also follow the ofﬁcial instructions . Then, you need to add the public key of your clouder in the /root/.ssh/authorized_keys ﬁle, so Clouder can connect to the server. Root rights are needed for the docker commands. If you don’t have a public key or don’t know how to create one, a public key will be automatically generated when you register the server in your Clouder. And that’s it. Now we can go to the Server menu in Clouder and create a new record.

9 Clouder Documentation, Release 1.0

Here the fields needed : • The prefix domain name used to contact the server. You can only use here lowercase, digit, hyphen and dot. • The domain name used to contact the server. • The server public IP. You can only use here digit and dot. This IP will be used to publish the services to Internet. • The server private IP. You can only use here digit and dot. This IP will be used internally for communication between services deployed by Clouder. • The ssh login user to use. Note you shall make sure this user can use the docker daemon. Otherwise root will be used. • The ssh port to connect to the server. • The environment of the server, define which user can use this server. • The provider field, if you want to use clouder provider. More detail below. • The SSH private and public key to connect to the server. A couple of keys are automatically generated when you create a new record, but you can use your owns. • The ports range for the containers which will be created in this server. When you create a container without specifying an hostport, a new port from this range will be attributed to it. • Assign port with public IP if you want to have several services with the same port on this node, with IP failover. This is interesting with Docker Engine to have several Clouder infrastructure on the same node. Otherwise, the 25/53/80/443 ports of mail/DNS/proxy services will conflicts. This doesn’t work with Docker Swarm mode. • Check the public checkbox if you want all users of the Clouder to be able to use this server. Otherwise, a user can only access a server if he is the manager of this server (or an administrator). When you save the new record, the ssh key will be saved in the system hosting the Clouder so it can connect to the new server. You can check the result of the command in the logs.

10 Chapter 2. Connect to a new node Clouder Documentation, Release 1.0

You have several actions available : • The reinstall action, if you think the ssh key wasn’t correctly installed. • The purge action, to remove all configuration from the server. This is executed when you unlink/remove the node. • The configure action will connect to the node and make sure everything is correctly installed and configured (SSH, Docker, etc...), and will correct it otherwise. • The test connection action will check the node is ready to use. • The restart services will stop and restart all containers in the node. • The stop services will stop all containers in the node. • The Reinstall/Stop DNS will refresh the DNS record of the node in the DNS. • The Clean disk space action will perform some action on the node to clean Docker on the node (removing unused containers/volumes/images). At the bottom of the page, you can specify a oneclick and deploy it. This will deploy all core services needed for your infrastructure on this node. Assigning critical port means some services defined with specific port (for example 25/53/80/443 for mail/DNS/proxy) will use them instead of the usual auto-assigned port. Finally, you can specify a provider. In this case, a Libcloud tab will be displayed.

11 Clouder Documentation, Release 1.0

This tab is used to deploy a new node at your provider. When you use this method, you don’t need to specify IP/port/keys, everything will be configured automatically. The provider credentials is configured in your provider record in configuration. Here you’ll have to specify which image/size/location you want to use, and the name of your node. Then you can use the create button to create your node at your provider. The IP address will be recover and everything will be automatically configured so the node is ready to use. You can use the other button destroy/reboot/stop to change the state of your node directly from Clouder interface.

12 Chapter 2. Connect to a new node CHAPTER 3

Images

The Image menu is the place where you’ll manage the images of your containers.

The ﬁelds included : • The name of the image. You can only use here lowercase, digit and underscore. • The templates it inherit (deﬁne volumes and ports). Templates are very useful to deduplicate some very redundant code in modules. • The parent image to use. We recommend building a base image containing the packages you need to launch your process (like supervisor, cron, syslog etc. . . ) which you can then use as a parent for the other images. • Check the public checkbox if you want all users of the Clouder to be able to use this image. Otherwise, a user can only access an image if he is the manager of this image (or an administrator).

13 Clouder Documentation, Release 1.0

• The volume_from this image shall inherit. When the service is deployed, it will search for all other children with the tag speciﬁed here and attached all their volumes to this service.

Then you have the Dockerfile field, used to know which commands shall be executed during the image building. Follow the Docker documentation https://docs.docker.com/reference/builder/ if you don’t know how to write it. Note that the commands FROM (use parent field), MAINTENER (use the sysadmin email), VOLUME (use the volumes configuration) and EXPOSE (use the ports configuration) are automatically inserted in the Dockerfile during the building process, so you don’t have to take care of them. Then you have the volumes configuration. This is used both to indicate the volume of the container but also the directory to backup in the container. When you restore a container backup, all volumes will be erased and replaced by the backup, the rest of the container will be untouched. Also after the restore and if filled, the volume will be chown with the systemuser value to ensure rights are correctly attributed. You can also map a directory to a directory in the host system with the host path option. Next you have the ports configuration, used to know the ports which need to be exposed to the outside of the container, to the local network or to the whole Internet. In the container, one field will be added : the hostport which is the port to access the localport in the container from outside of the host system. It is automatically attributed within the port range of the server but you can also force a specific port (Especially needed for the Bind, Proxy and Postfix applications which have to listen on ports 25, 53, 80 and 443 of the host system) Then, if you have a parent image, you have to select the parent version you want to use. Else you have to specify the FROM statement like you would use in a Dockerfile (example : debian:latest). You may also specify the registry container where the image will be stored. Finally, you can press the build button. This will create a new image version, named from the current date, and execute the commands to build it and store it in the registry. You don’t have to use it thought, as when you deploy a service without specifying a version, it will auto-build the image before running it. Remember you can manage your own images, but you also can use the ready-to-use images from the clouder_template_* modules until you feel confident enough to create yours.

14 Chapter 3. Images CHAPTER 4

Applications

The Application represent the software you can host in your Clouder. In this chapter, we’ll see how we can conﬁgure them.

Application Types

This menu contain the list of application type deﬁned in your Clouder. They are mainly just a tag so the python code can know which code shall be executed, depending of the type of the application.

Since it’s so deeply connected with the python code, you shall never create an application type through the graphical interface, it shall always be deﬁned in a template module.

15 Clouder Documentation, Release 1.0

It contains : • The name of the type, can only contains lowercase, digit or hyphen. • The system user which will run the applications. • Finally, if the application needs several databases (like seafile), you can indicate in the multiple databases fields the name of the databases which need to be created. In the application type you can also define the options. Options allow to manage configuration settings specifics to an application type, so you can manage some values needed for your python code without having to add new fields in the Clouder interface. An option can be assigned to applications, containers, services or bases. In the application type you’ll define the possible options for each objects, and the values will be defined in the objects. Each option has : • The name of the option, can only contains lowercase, digit or underscore. • The type of object where the value of the option will be defined. • If the option shall be suggested when you create a new record in the designed object. • If this option is required, if set to True you’ll not be able to create a new record if the option is not set. • Finally, the default value for this option if needed.

Application

The application represent the software which will be hosted in your Clouder. You can have several applications for each application types.

16 Chapter 4. Applications Clouder Documentation, Release 1.0

For example, for the application type odoo, you can have three applications: • An Odoo application, which will just install a regular Odoo with only the base module. • A Clouder application, to host Clouder for other peoples (Yeah!). It will add the clouder modules in the application archive and auto install the clouder module thanks to the install_modules option. A Clouder application is nothing more than a verticalisation.

Since the modules can be installed automatically when the base is deployed, this mean you can easily create your own verticalisation with modules your developped or you like, and add a form on your website so your visitors will be able to automatically get their own instances of your verticalisation. We took the example of Odoo, but it works with any software which have a module system, like Drupal, Wordpress etc. . . If you know a lot of very good modules and want to share this knowledge you can easily use Clouder to build a package which your visitors can install in just a few click.

Let’s get back to the application form. For each application you can define : • The application name, no restrictions here. • The templates it inherit (define options and links). Templates are very useful to deduplicate some very redundant code in modules. • The application type of the application. • The application code, can only contains lowercase, digit or hyphen. Note that since this code will be used is many location in your infrastructure, you can’t change his value once defined. • The current version of the application. This is used for the version name which will use the format ${cur- rent_version}.${current_datetime}. • The tags of the application, often used to trigger some action in the code. • If the application can deploy a base, hence if the application is a web application. For example an Odoo application can deploy a base, but not a postfix application. • The default image/version which shall be used for this application when we deploy a container. • The next node/service to use for this application, if you create a new base without providing a service it will automatically deploy a new service in this node.

4.2. Application 17 Clouder Documentation, Release 1.0

• The default admin name and email, theses options can be overridden in the base. • Check the public checkbox if you want all users of the Clouder to be able to use this application. Otherwise, a user can only access an application if he is the manager of this application (or an administrator). • The sequence is used if this application has a parent application, to know the order the child shall be deployed. Obviously very important, you don’t want to deploy Exec container before Data container. • The required field, again used if this application is a child the application will be auto-deployed when parent is deployed. • The update strategy, if this application shall be updated each day with the new image from Docker Hub (Exec), if it shall be manually updated with a button when needed (Files) or never (Data). Updating Exec each day is important for security reasons, this is the exposed container. • Update base, if an update if this service shall update bases linked to it. For example, when we update Odoo files service. • Dummy, if this service shall not deploy any container, even if it does not have children. This is used for LibCloud services, which will connect to cloud API and not deploy container. • Provider, if we want to display the provider field in service form, again for LibCloud services.

Next you have the options configuration, where you can define the values for the options specifics to this application. Following, you have the links configuration. Theses are an extremely important concept, you can link an application to the others in order to build a coherent infrastructure. For example, an Odoo container by himself can’t work, you need at least to link it to a Postgres container so it can have their databases. Same, you can link the Odoo container to a Postfix container for sending and receiving emails, a Proxy container for redirecting the defaults ports to Odoo and adding the https, a Shinken container to make sure it’s always up and get stats, a Bind container to assign a domain name to the Odoo base, a Piwik to get analytic stats etc. . . When a link is deployed, some specific python code will be called and will make sure the other container know the existence of this container and adapt his configuration. Very similar to the options interface, links are configured in the application form and are defined in the containers/services/bases. So for each link you can define : • The application to link. • If this link is required, if set to True you’ll not be able to create a new record if the link is not set. • If the link shall be suggested when you create a new record in the designed object.

18 Chapter 4. Applications Clouder Documentation, Release 1.0

• If a direct docker link shall be made between the two containers. This improve security for the communications between the containers but require the containers to be in the same server. • The type of object where this link shall be available (service/base) • The next container which shall be used for this link. If ﬁlled, the link will be autocompleted in the container/service/base. Then after the links, you can specify if this application have children, in a many2many ﬁeld. When deploying this application, all children will be created as sub-services and deploy in the order. Finally, on the backup tab you can manage the save options for this application. You can specify : • The backup services for containers and bases. • The time between each automatic save. Since some backup methods use global deduplication, you can make frequent saves without having to worry too much about disk space in the backup container. • The time between each save repository change the time before a repository is deleted. Check the Backup chapter for more informations about repositories and their utilities. • The time before a backup expiration. All theses settings are inherited but can be changed in containers and bases forms.

4.2. Application 19 Clouder Documentation, Release 1.0

20 Chapter 4. Applications CHAPTER 5

Services

To create a new service on one of your node, you just have to add it in the Services menu and it will be automatically deployed.

The important fields are : • The environment, which define which users can access to this services. It also define the prefix used by the container which will be deployed. • The service suffix. You can only use here lowercase, digit and hyphen. • The application which will be deployed in this service, the node where it will be deployed and the image which will be used. When you change the application, it will update most of the fields with the infos from applications and images. • The volume_from this service shall inherit. When the service is deployed, it will search for all other children with the tag specified here and attached all their volumes to this service.

21 Clouder Documentation, Release 1.0

• The number of containers which shall be deployed by this service. Use the scale command for Docker Swarm mode. • Check the public checkbox if you want all users of the Clouder to be able to use this container. Otherwise, a user can only access a container if he is the manager of this container (or an administrator). • Check the dummy checkbox is this service shall not deploy any container, even without children. • Specify the provider is this services shall be deployed in the cloud with libcloud.

See the Images and Applications chapters for more informations about options, ports, volumes, links and childs. When you save the new services : • If any childs, the childs will be deployed and create subservices. No container will be created for the parent service. • The command to create the new container will be launched on the target server • The ports and volumes will be configured, • The commands post creation will finish to configure the container • The links will be deployed. You can check all the executed command in the log.

22 Chapter 5. Services Clouder Documentation, Release 1.0

If something went wrong, you can use the reinstall button to purge and reinstall the container. You can also also use the restart/stop buttons to restart the container. You can also easily create a subservice by indicating the name of the subservice and clicking on Install subservice. This will create the subservice with the same ﬁles, link it to the current service, and duplicate all the base linked to the current service.

Finally, you can configure in the save tab the backup container where the backup will be stored. The repository will be automatically created if needed, see the Applications chapter for more informations about the others fields You can use the Backup checkbox if you want to auto backup this service each day. You can use the Backup button to force a manual backup, use the comment field if you want to easily find this backup later. Note that all saves update the next save date field.

23 Clouder Documentation, Release 1.0

24 Chapter 5. Services CHAPTER 6

Domains and Bases

Domains

In the Domain menu you can manage the domain managed by your Clouder.

25 Clouder Documentation, Release 1.0

In the domain form, you have to specify : • The domain, only lowercase, digit, hyphen and dot. are accepted. • The organization managing the domain. • The DNS service which will manage the domain. • If you have one, a wildcard ssl certificate which you’ll be able to use for all subdomains. • Check the public checkbox if you want all users of the Clouder to be able to use this domain. Otherwise, a user can only access a domain if he is the manager of this domain (or an administrator). When you save the new record, a configuration file will be installed in the DNS server to manage the domain. You can check the result of the command in the logs

You can use the reinstall button if you think the domain has not been correctly installed.

26 Chapter 6. Domains and Bases Clouder Documentation, Release 1.0

Bases

A base correspond to an url which will be available on the Internet, and often correspond to a database.

The important fields are : • The base name. You can only use here lowercase, digit and hyphen. • The domain to use. The application will be available at the url BASENAME.DOMAIN • The environment, which define the access right. • The application of the base. When you select it, some fields like the admin name/email, links and options will be inherited. • The service which will manage the base. If not filled, a new service will automatically be created. • You can specify the login/password/email for the admin account. The password will be automatically generated. • Same for the poweruser login/password/email. A poweruser account is used if we don’t want to give admin access to the user who requested the creation of the base. • The SSL key/cert for https connection. You can generate automatically with the button Generate Certs, which will call the letsencrypt binary in proxy service. • If this base can be reset with another base. If yes, you can click on a button to drop the current database et restore it from a dump of the specified base. This is especially useful for staging environment, linked to a production base to have real data. You can also ask that this base is reset each day automatically, useful for demo instance. • The default language which shall be configured in the application. • The type of installation which shall be made. Build will make a full installation of the application, while restore (not yet working) will restore and configure a dump file. • You can use the ssl only checkbox if you want the url to be only accessible through ssl. All users will be redirected on the SSL port of the proxy. • You can check the test checkbox to install the application with the demo data. • Check the public checkbox if you want all users of the Clouder to be able to use this base. Otherwise, a user can only access a base if he is the manager of this base (or an administrator). • You can use the Backup checkbox to auto-backup this base.

6.2. Bases 27 Clouder Documentation, Release 1.0

See the Applications chapter for more informations about options and links. When you save the new base : • The new database will be created. • The commands post creation will conﬁgure the base. • The admin and poweruser will be created. • If test is checked, the demo data will be installed. • The links will be deployed. You can check all the executed commands in the log.

If something went wrong, you can use the reinstall button to purge and reinstall the base. Finally, you have to configure in the save tab the backup container where the backup will be stored. See the Applica- tions chapter for more information about the others fields. You can use the Backup button to force a manual backup, use the comment field if you want to easily find this backup later. Note that all saves update the next save date field.

28 Chapter 6. Domains and Bases Clouder Documentation, Release 1.0

6.2. Bases 29 Clouder Documentation, Release 1.0

30 Chapter 6. Domains and Bases CHAPTER 7

Backups and Conﬁguration

Backups

Clouder take care of the backup management and will periodically launch a backup on containers and bases. Two methods are today available for the backups : Simple, will simply copy the ﬁles in a directory of the backup container. Bup, will store the backup into a bup repository. Bup is a backup solution based on git which will do global deduplication on the backup to massively save disk space. With bup, you can make hourly backup without having to worry about disk space. Among the downside it’s difﬁcult to remove a backup once on bup, that’s why we sometimes change the repository where backup are stored, and we need to manage the concept of repository on Clouder. You can see the list of all backup on the Backup menu.

31 Clouder Documentation, Release 1.0

Each backup contains : • His name • The Backup service where it is stored. • The reason for the backup. • The date. • The date when the backup will be deleted. All backups are linked to the container/service/base from which it was generated, but also most of the informations from container/service/base are stored into the save form because we need them if we need to regenerate them because the object was deleted.

To restore a backup, you simply can press the button restore, after having change the Restore to fields if you want to restore in another container of base. If not, it will delete the current container/base (after another backup). The restore will : • If container, it will delete the content of the volumes and restore them from the backup. • If base, if will drop the database and restore it • Some specific files may be restored if specified in the python code of the template modules. You can check the save and restore commands execution on the log window.

32 Chapter 7. Backups and Conﬁguration Clouder Documentation, Release 1.0

Conﬁguration

On the Configure Clouder menu you’ll find some fields used in the whole Clouder instance. • The sysadmin email, used in services to configure them. • The runner, which specify how container are deployed. Docker Engine will use Docker in Engine mode, Clouder itself connect to each node and execute command to deploy container. In Docker Swarm mode, Clouder will connect to the master node and create service here, then Docker Swarm itself will create the pods. You can also specify a custom runner, like Kubernetes/OpenShift/Rancher, in this case you have to specify the link to the service running them. • Compose mode. In this mode, when you deploy a root service a compose file is generated for all the children and deployed. Otherwise, Clouder itself create the container one by one. Experimental until Docker Swarm itself is compatible with Docker Compose, not the case yet. • The executor, which specify how Clouder execute command on node/containers. By default it use ssh and docker exec, but you can also use other orchestrateur like Salt or Ansible recipes.

7.2. Conﬁguration 33 Clouder Documentation, Release 1.0

You’ll find also here some buttons to execute maintenance tasks : • Backup All will perform some maintenance task on the backup containers (if they use Bup), will launch a save on all containers and bases, and upload a copy of the backup containers to their backup-upload links. • Purge expired saves will drop all saves which pass their expiration date. • Update services will destroy and reinstall all services with an autoupdate strategy. This mainly concern Exec containers, which need to be always uptodate for security and has no risk of losing data. • Reset bases will launch the reset action on the bases which have the Reset each day checkbox set to True. • Cert renewal will check for any LetsEncrypt cert about to expire and renew them. • Launch daily cron will execute all the above task, and is linked to a daily cron. • Launch next save will execute a backup on all containers/bases which pass the next save date. It’s linked to a cron which is executed every thirty minutes. Backup All, Update services, Reset bases and Cert renewal, the longest tasks, will update the configuration dates when they end their actions, so you can easily know at which time the actions finished. Finally, you can also specify here the providers used by LibCloud to deploy new nodes, DNS records, SMTP etc... used in node and services form.

34 Chapter 7. Backups and Conﬁguration Clouder Documentation, Release 1.0

7.2. Conﬁguration 35