Table of Contents

Introduction 1.1 System Users 1.2 CouchDB 1.3 1.4 Passenger 1.5 Solr 1.6 Beanstalkd 1.7 Backburner 1.8 CouchWatcher 1.9 Primero Scheduler 1.10 primeroctl Script 1.11

1 Introduction

Primero Application

Primero is a application. It is served by which is proxied to by an Nginx . Static assets such as JS, CSS, and application icons and images are served by Nginx.

Nginx is configured to serve Primero via HTTPS. It runs on default port HTTPS 443 which accepts traffic redirected from port 80.

The backing database is CouchDB. When CouchDB is configured to replicate, it will be running on port 6984, via HTTPS.

The core run-time system dependencies of Primero are:

Nginx, which accepts external web traffic and routes it to Phusion Passenger or serves up static assets

Phusion Passenger which launches and services 1-N Ruby processes

CouchDB, the database backing Primero (see below)

Apache Solr, the Lucene search index service (see below)

The following services ensure proper Primero behavior when applying business rules, but will not cause fatal application errors if they are down:

Couch Change Watcher notifier

Primero Scheduler

The system is designed to be deployed to 16.04, the long term support edition which is guaranteed to be supported through the end of 2020. All deployment is automated using (https://www.chef.io/).

2 System Users

System users

In order to perform system maintenance, system support staff will need to log on to the server.

The system allows remote logon via SSH for the user ubuntu. This user has passwordless sudo (superuser) privileges and can be used to run system security upgrades and administer services. Designated support staff will be granted access to this user via public SSH keys.

In order to grant access to the ubuntu user, append the designated support person’s key to the file /home/ubuntu/.ssh/authorized_keys. For more on SSH key access to linux refer to the following guide: https://www.linode.com/docs/security/use-public-key-authentication- with-ssh.

Some administrative tasks such as data cleanup tasks must be done with the primero system user. This user is distinct from the application primero user. To switch from the ubuntu user to the primero user:

$ sudo -Hu primero bash

$ source ~/.rvm/scripts/rvm

3 CouchDB

CouchDB

CouchDB (http://couchdb.apache.org) is the backing database of Primero. It is accessible via HTTP on 127.0.0.1:5984 and is optionally reverse proxied by Nginx for HTTPS on 0.0.0.0:6984. This exposes CouchDB to the outside world for replication.

The admin is exposed at https://cpims-myimplementation.primero.org:6984/_utils.

If CouchDB is not exposed to the external world, one will need to set up a SSH tunnel and forward the port to the localhost in order to access it for maintenance:

$ ssh -f -N -L 45984:localhost:5984 admin_user@your_primero_server

The admin will now be available at http://localhost:45984/_utils

CouchDB is started as a service on system startup via the Upstart Daemon (http://upstart.ubuntu.com/)

Start $ sudo systemctl start couchdb

Stop $ sudo systemctl stop couchdb

$ sudo systemctl status couchdb couchdb.service - System-wide CouchDB instance Loaded: loaded (/lib/systemd/system/couchdb.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2017-09-19 05:01:44 UTC; 10h ago Main PID: 4515 (beam.smp) Tasks: 14 Memory: 77.9M CPU: 3min 44.982s CGroup: /system.slice/couchdb.service Status ├─ 4515 /usr/lib/erlang/erts-7.3/bin/beam.smp -Bd -K true -A 4 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/couchdb -- -noshell -noinput -os_mon start_memsup false start_cpu_sup fal ├─ 4553 sh -s disksup └─26184 /usr/bin/couchjs -S 134217728 /usr/share/couchdb/server/main.js Sep 19 15:09:31 primero-int-1-4 couchdb[4515]: [info] [<0.5509.1>] 127.0.0.1 - - PUT /primero_system_settings_production/52ffbe061b4b930ce941ff9a8f9b526c 201 Sep 19 15:09:31 primero-int-1-4 couchdb[4515]: [info] [<0.5510.1>] 127.0.0.1 - - GET /primero_sessions_production/e3255dc9a0964d1292f75e51113e0f87 200 Sep 19 15:09:31 primero-int-1-4 couchdb[4515]: [info] [<0.32009.0>] Starting index update for db: primero_system_settings_production idx: _design/SystemSettings

Log /srv/primero/logs/couchdb/couch.log File

Data /var/lib/couchdb Dir

Run root, couchdb Users

4 Nginx

Nginx

The Primero application is served by the Nginx server. Shutting the server down halts the application.

Note that Nginx is an optional runtime dependency of CouchDB for situations where external syncing is employed (as it is in Sierra Leone). Where CouchDB and the Primero Ruby on Rails application are deployed on the same box, they will share an instance of Nginx.

Start $ sudo systemctl start nginx

Stop $ sudo systemctl stop nginx

$ sudo systemctl status nginx nginx.service - A high performance web server and a reverse proxy server Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2017-09-19 05:02:53 UTC; 10h ago Process: 12520 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS) Process: 12517 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS) grep nginx` Main PID: 12523 (nginx) root 12523 1 0 05:02 ? 00:00:00 nginx: master Tasks: 3 process /usr/sbin/nginx -g daemon on; master_process on; Status Memory: 8.0M CPU: 2.987s www-data 12524 12523 0 05:02 ? 00:00:02 nginx: CGroup: /system.slice/nginx.service worker process www-data 12525 12523 0 05:02 ? 00:00:00 nginx: ├─12523 nginx: master process /usr/sbin/nginx -g daemon on; master_process on worker process ├─12524 nginx: worker process └─12525 nginx: worker process Sep 19 05:02:53 primero-int-1-4 systemd[1]: Starting A high performance web server and a reverse proxy server... Sep 19 05:02:53 primero-int-1-4 nginx[12517]: nginx: [warn] "ssl_stapling" ignored, issuer certificate not found Sep 19 05:02:53 primero-int-1-4 nginx[12520]: nginx: [warn] "ssl_stapling" ignored, issuer certificate not found

`$ ps -fA \

/srv/primero/logs/couchdb/nginx_server.log Log /srv/primero/logs/couchdb/nginx_error.log Files /var/log/nginx/access.log

Run root, www-data User

5 Passenger

Passenger

Passenger is an open source, scalable server. If there is instability or unpredictable behavior experienced after applying a configuration bundle, stopping then starting Passenger is a good approach to resolving them.

Start $ sudo systemctl start passenger

Stop $ sudo systemctl stop passenger

$ sudo systemctl status passenger passenger.service - Passenger Standalone Loaded: loaded (/etc/systemd/system/passenger.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2017-09-19 05:02:49 UTC; 10h ago Main PID: 12316 (PassengerWebHel) Tasks: 26 Memory: 277.7M CPU: 1min 31.858s CGroup: /system.slice/passenger.service ├─12298 PassengerWatchdog ├─12301 PassengerHelperAgent ├─12307 PassengerLoggingAgent ├─12316 PassengerWebHelper: master process /srv/primero/.passenger/standalone/4.0.59/webhelper-1.6.2-x86_64- linux/PassengerWebHelper - /tmp/passenger-standalone.rjitj/config -p /tmp/passenger- ├─12317 PassengerWebHelper: worker process ├─12322 /srv/primero/.passenger/standalone/4.0.59/support-x86_64-linux/agents/TempDirToucher /tmp/passenger- Status standalone.rjitj --cleanup --daemonize --pid-file /tmp/passenger-standalone.rjitj/tem └─12552 Passenger RackApp: /srv/primero/application Sep 19 05:02:48 primero-int-1-4 systemd[1]: Starting Passenger Standalone Application Server... Sep 19 05:02:49 primero-int-1-4 passenger-worker.sh[12161]: ======Phusion Passenger Standalone web server started ======Sep 19 05:02:49 primero-int-1-4 passenger-worker.sh[12161]: PID file: /srv/primero/application/tmp/passenger.pid Sep 19 05:02:49 primero-int-1-4 passenger-worker.sh[12161]: Log file: /srv/primero/logs/rails//passenger.log Sep 19 05:02:49 primero-int-1-4 passenger-worker.sh[12161]: Environment: production Sep 19 05:02:49 primero-int-1-4 passenger-worker.sh[12161]: Accessible via: http://127.0.0.1:4000/ Sep 19 05:02:49 primero-int-1-4 passenger-worker.sh[12161]: Serving in the background as a daemon. Sep 19 05:02:49 primero-int-1-4 passenger-worker.sh[12161]: Problems? Check https://www.phusionpassenger.com/documentation/Users%20guide%20Standalone.html#troubleshooting Sep 19 05:02:49 primero-int-1-4 passenger-worker.sh[12161]: ======Sep 19 05:02:49 primero-int-1-4 systemd[1]: Started Passenger Standalone Application Server.

`$ ps -fA \

Log /srv/primero/logs/rails/passenger.log File

Run root, primero User

6 Solr

Solr

Apache Solr (http://lucene.apache.org/solr/) is used as the search and query index for Primero. All records saved in Primero are indexed in Solr. Solr is a Java process that runs on a application server.

Solr is launched by Supervisor (http://supervisord.org/). A nightly cron job restarts Solr to prevent the memory cache from over- expanding.

$ sudo supervisorctl start Start solr

$ sudo supervisorctl stop Stop solr

$ sudo supervisorctl status solr grep solr` solr RUNNING pid 14899, solr 14899 1423 0 06:25 ? 00:00:32 java -Djetty.port=8983 - Status uptime 9:52:10 Dsolr.data.dir=/srv/primero/application/solr/data/production - Dsolr.solr.home=/srv/primero/application/solr -Djava.awt.headless=true -jar `$ ps -fA \ start.jar

$ RAILS_ENV=production Reindex bundle exec sunspot:reindex

Log /srv/primero/logs/solr/output.log Files

Run root, solr User

7 Beanstalkd

Beanstalkd

Beanstalk (https://github.com/kr/beanstalkd/wiki/faq) is a queue for processes. Currently, beanstalk is by Primero for bulk exports. Beanstalk prevents the system from getting overwhelmed by thousands of tasks and just queues up the tasks until the system has enough resources.

Start $ sudo systemctl start beanstalkd

Stop $ sudo systemctl stop beanstalkd

$ sudo systemctl status beanstalkd beanstalkd.service - Simple, fast work queue Loaded: loaded (/lib/systemd/system/beanstalkd.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2017-09-19 04:17:53 UTC; 11h ago Docs: man:beanstalkd(1) Main PID: 1086 (beanstalkd) grep beanstalkd` Status Tasks: 1 beansta+ 1086 1 0 04:17 ? 00:00:00 /usr/bin/beanstalkd -l Memory: 1.0M localhost -p 11300 -b /srv/primero/beanstalkd CPU: 5ms CGroup: /system.slice/beanstalkd.service └─1086 /usr/bin/beanstalkd -l localhost -p 11300 -b /srv/primero/beanstalkd Sep 19 04:17:53 primero-int-1-4 systemd[1]: Started Simple, fast work queue.

`$ ps -fA \

Log /srv/primero/logs/solr/output.log Files

Run root, primero User

8 Backburner

Backburner

Backburner (https://github.com/nesquena/backburner/wiki) is ‘queue consumer’ which takes items from the beanstalk queue and executes them as resources become available.

$ sudo supervisorctl start Start backburner

$ sudo supervisorctl stop Stop backburner

grep backburner` $ sudo supervisorctl status backburner primero 7915 1423 0 05:02 ? 00:00:00 /bin/bash backburner RUNNING pid /srv/primero/application/daemons/backburner-worker.sh Status 7915, uptime 11:04:12 primero 8005 7915 0 05:02 ? 00:00:02 ruby /srv/primero/.rvm/gems/ruby-2.1.5- railsexpress/bin/rake backburner:work `$ ps -fA \ primero 8566 8005 0 05:02 ? 00:00:00 ruby /srv/primero/.rvm/gems/ruby-2.1.5- railsexpress/bin/rake backburner:work

Log /srv/primero/logs/solr/output.log Files

Run root, primero User

9 CouchWatcher

CouchDB CouchWatcher

The CouchDB change watcher monitors CouchDB server and reacts to certain data events:

New records being created trigger Solr re-indexing

Changes to the forms metadata trigger rebuilding of the Rails mapping objects that interface with CouchDB and Solr

Detects and resolves record conflicts generated by bulk updates or CouchDB-to-CouchDB syncs and stored in CouchDB.

The Change Watcher is Ruby on Rails process running on an EventMachine (https://github.com/eventmachine/eventmachine). It is launched by Supervisor. When a new configuration bundle is applied, the Couch Watcher is restarted.

Start $ sudo supervisorctl start couch-watcher

Stop $ sudo supervisorctl stop couch-watcher

$ sudo supervisorctl status couch-watcher grep couch_changes` couch-watcher RUNNING pid 11497, uptime 11:07:36 primero 11587 11497 0 Status 05:02 ? 00:00:06 `$ ps -fA \ lib/couch_changes/base.rb

History /srv/primero/application/tmp/couch_watcher_history.json file

Log /srv/primero/logs/couch_watcher/output.log/srv/primero/logs/couch_watcher/production.log Files

Run root, primero User

10 Primero Scheduler

Primero Scheduler

The Primero scheduler runs periodic data maintenance tasks. It corrects or updates data:

Updates age based on birthday

Auto generates flags to prompt for follow ups (functionality not used in SL)

In the future other maintenance tasks will be added

The Primero scheduler is based on the Rufus Scheduler gem (https://github.com/jmettraux/rufus-scheduler). It is launched as a process using Rake tasks as an application process.

Note the RAILS_SCHEDULER_LOGDIR is /srv/primero/logs/scheduler on a default system.

For Sierra Leone AWS deploy, RAILS_SCHEDULER_LOGDIR is /data/logs/scheduler

$ sudo supervisorctl start Start primero-scheduler

$ sudo supervisorctl start Stop primero-scheduler

$ sudo supervisorctl status primero-scheduler primero-scheduler RUNNING pid grep primero-scheduler` Status 10823, uptime 11:13:15 primero 10823 1423 0 05:02 ? 00:00:00 /bin/bash /srv/primero/application/primero-scheduler-worker.sh `$ ps -fA \

Log /srv/primero/logs/scheduler/primero- Files scheduler.output

Run root, primero User

11 primeroctl Script

primeroctl Script

The primeroctl script should be used to act simulaneously on all of the primero applications. so you can simultaneously start, stop or check the status of: CouchDB, Nginx, Passenger, Solr, Beanstalkd, Backburner, CouchWatcher, and Primero Scheduler.

Start $ sudo /srv/primero/bin/primeroctl start

Stop $ sudo /srv/primero/bin/primeroctl stop

$ sudo /srv/primero/bin/primeroctl status Primero status: nginx: active (running) since Tue 2017-09-19 05:02:53 UTC; 12h ago passenger: active (running) since Tue 2017-09-19 05:02:49 UTC; 12h ago couch-watcher RUNNING pid 11497, uptime 12:38:48 Status who-watches-the-couch-watcher RUNNING pid 11604, uptime 12:38:47 primero-scheduler RUNNING pid 10823, uptime 12:38:55 backburner RUNNING pid 7915, uptime 12:39:24 beanstalkd: active (running) since Tue 2017-09-19 04:17:53 UTC; 13h ago solr RUNNING pid 14899, uptime 11:16:06 couchdb: active (running) since Tue 2017-09-19 05:01:44 UTC; 12h ago

Run User root

12