Table of Contents
Introduction 1.1 System Users 1.2 CouchDB 1.3 Nginx 1.4 Passenger 1.5 Solr 1.6 Beanstalkd 1.7 Backburner 1.8 CouchWatcher 1.9 Primero Scheduler 1.10 primeroctl Script 1.11
1 Introduction
Primero Application
Primero is a Ruby on Rails application. It is served by Phusion Passenger which is proxied to by an Nginx web server. Static assets such as JS, CSS, and application icons and images are served by Nginx.
Nginx is configured to serve Primero via HTTPS. It runs on default port HTTPS 443 which accepts traffic redirected from port 80.
The backing database is CouchDB. When CouchDB is configured to replicate, it will be running on port 6984, via HTTPS.
The core run-time system dependencies of Primero are:
Nginx, which accepts external web traffic and routes it to Phusion Passenger or serves up static assets
Phusion Passenger which launches and services 1-N Ruby processes
CouchDB, the database backing Primero (see below)
Apache Solr, the Lucene search index service (see below)
The following services ensure proper Primero behavior when applying business rules, but will not cause fatal application errors if they are down:
Couch Change Watcher notifier
Primero Scheduler
The system is designed to be deployed to Ubuntu 16.04, the long term support edition which is guaranteed to be supported through the end of 2020. All deployment is automated using Chef (https://www.chef.io/).
2 System Users
System users
In order to perform system maintenance, system support staff will need to log on to the Linux server.
The system allows remote logon via SSH for the user ubuntu. This user has passwordless sudo (superuser) privileges and can be used to run system security upgrades and administer services. Designated support staff will be granted access to this user via public SSH keys.
In order to grant access to the ubuntu user, append the designated support person’s key to the file /home/ubuntu/.ssh/authorized_keys. For more on SSH key access to linux refer to the following guide: https://www.linode.com/docs/security/use-public-key-authentication- with-ssh.
Some administrative tasks such as data cleanup tasks must be done with the primero system user. This user is distinct from the application primero user. To switch from the ubuntu user to the primero user:
$ sudo -Hu primero bash
$ source ~/.rvm/scripts/rvm
3 CouchDB
CouchDB
CouchDB (http://couchdb.apache.org) is the backing database of Primero. It is accessible via HTTP on 127.0.0.1:5984 and is optionally reverse proxied by Nginx for HTTPS on 0.0.0.0:6984. This exposes CouchDB to the outside world for replication.
The admin is exposed at https://cpims-myimplementation.primero.org:6984/_utils.
If CouchDB is not exposed to the external world, one will need to set up a SSH tunnel and forward the port to the localhost in order to access it for maintenance:
$ ssh -f -N -L 45984:localhost:5984 admin_user@your_primero_server
The admin will now be available at http://localhost:45984/_utils
CouchDB is started as a service on system startup via the Upstart Daemon (http://upstart.ubuntu.com/)
Start $ sudo systemctl start couchdb
Stop $ sudo systemctl stop couchdb
$ sudo systemctl status couchdb couchdb.service - System-wide CouchDB instance Loaded: loaded (/lib/systemd/system/couchdb.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2017-09-19 05:01:44 UTC; 10h ago Main PID: 4515 (beam.smp) Tasks: 14 Memory: 77.9M CPU: 3min 44.982s CGroup: /system.slice/couchdb.service Status ├─ 4515 /usr/lib/erlang/erts-7.3/bin/beam.smp -Bd -K true -A 4 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/couchdb -- -noshell -noinput -os_mon start_memsup false start_cpu_sup fal ├─ 4553 sh -s disksup └─26184 /usr/bin/couchjs -S 134217728 /usr/share/couchdb/server/main.js Sep 19 15:09:31 primero-int-1-4 couchdb[4515]: [info] [<0.5509.1>] 127.0.0.1 - - PUT /primero_system_settings_production/52ffbe061b4b930ce941ff9a8f9b526c 201 Sep 19 15:09:31 primero-int-1-4 couchdb[4515]: [info] [<0.5510.1>] 127.0.0.1 - - GET /primero_sessions_production/e3255dc9a0964d1292f75e51113e0f87 200 Sep 19 15:09:31 primero-int-1-4 couchdb[4515]: [info] [<0.32009.0>] Starting index update for db: primero_system_settings_production idx: _design/SystemSettings
Log /srv/primero/logs/couchdb/couch.log File
Data /var/lib/couchdb Dir
Run root, couchdb Users
4 Nginx
Nginx
The Primero application is served by the Nginx server. Shutting the server down halts the application.
Note that Nginx is an optional runtime dependency of CouchDB for situations where external syncing is employed (as it is in Sierra Leone). Where CouchDB and the Primero Ruby on Rails application are deployed on the same box, they will share an instance of Nginx.
Start $ sudo systemctl start nginx
Stop $ sudo systemctl stop nginx
$ sudo systemctl status nginx nginx.service - A high performance web server and a reverse proxy server Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2017-09-19 05:02:53 UTC; 10h ago Process: 12520 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS) Process: 12517 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS) grep nginx` Main PID: 12523 (nginx) root 12523 1 0 05:02 ? 00:00:00 nginx: master Tasks: 3 process /usr/sbin/nginx -g daemon on; master_process on; Status Memory: 8.0M CPU: 2.987s www-data 12524 12523 0 05:02 ? 00:00:02 nginx: CGroup: /system.slice/nginx.service worker process www-data 12525 12523 0 05:02 ? 00:00:00 nginx: ├─12523 nginx: master process /usr/sbin/nginx -g daemon on; master_process on worker process ├─12524 nginx: worker process └─12525 nginx: worker process Sep 19 05:02:53 primero-int-1-4 systemd[1]: Starting A high performance web server and a reverse proxy server... Sep 19 05:02:53 primero-int-1-4 nginx[12517]: nginx: [warn] "ssl_stapling" ignored, issuer certificate not found Sep 19 05:02:53 primero-int-1-4 nginx[12520]: nginx: [warn] "ssl_stapling" ignored, issuer certificate not found
`$ ps -fA \
/srv/primero/logs/couchdb/nginx_server.log Log /srv/primero/logs/couchdb/nginx_error.log Files /var/log/nginx/access.log
Run root, www-data User
5 Passenger
Passenger
Passenger is an open source, scalable web application server. If there is instability or unpredictable behavior experienced after applying a configuration bundle, stopping then starting Passenger is a good approach to resolving them.
Start $ sudo systemctl start passenger
Stop $ sudo systemctl stop passenger
$ sudo systemctl status passenger passenger.service - Passenger Standalone Application Server Loaded: loaded (/etc/systemd/system/passenger.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2017-09-19 05:02:49 UTC; 10h ago Main PID: 12316 (PassengerWebHel) Tasks: 26 Memory: 277.7M CPU: 1min 31.858s CGroup: /system.slice/passenger.service ├─12298 PassengerWatchdog ├─12301 PassengerHelperAgent ├─12307 PassengerLoggingAgent ├─12316 PassengerWebHelper: master process /srv/primero/.passenger/standalone/4.0.59/webhelper-1.6.2-x86_64- linux/PassengerWebHelper -c /tmp/passenger-standalone.rjitj/config -p /tmp/passenger- ├─12317 PassengerWebHelper: worker process ├─12322 /srv/primero/.passenger/standalone/4.0.59/support-x86_64-linux/agents/TempDirToucher /tmp/passenger- Status standalone.rjitj --cleanup --daemonize --pid-file /tmp/passenger-standalone.rjitj/tem └─12552 Passenger RackApp: /srv/primero/application Sep 19 05:02:48 primero-int-1-4 systemd[1]: Starting Passenger Standalone Application Server... Sep 19 05:02:49 primero-int-1-4 passenger-worker.sh[12161]: ======Phusion Passenger Standalone web server started ======Sep 19 05:02:49 primero-int-1-4 passenger-worker.sh[12161]: PID file: /srv/primero/application/tmp/passenger.pid Sep 19 05:02:49 primero-int-1-4 passenger-worker.sh[12161]: Log file: /srv/primero/logs/rails//passenger.log Sep 19 05:02:49 primero-int-1-4 passenger-worker.sh[12161]: Environment: production Sep 19 05:02:49 primero-int-1-4 passenger-worker.sh[12161]: Accessible via: http://127.0.0.1:4000/ Sep 19 05:02:49 primero-int-1-4 passenger-worker.sh[12161]: Serving in the background as a daemon. Sep 19 05:02:49 primero-int-1-4 passenger-worker.sh[12161]: Problems? Check https://www.phusionpassenger.com/documentation/Users%20guide%20Standalone.html#troubleshooting Sep 19 05:02:49 primero-int-1-4 passenger-worker.sh[12161]: ======Sep 19 05:02:49 primero-int-1-4 systemd[1]: Started Passenger Standalone Application Server.
`$ ps -fA \
Log /srv/primero/logs/rails/passenger.log File
Run root, primero User
6 Solr
Solr
Apache Solr (http://lucene.apache.org/solr/) is used as the search and query index for Primero. All records saved in Primero are indexed in Solr. Solr is a Java process that runs on a Jetty application server.
Solr is launched by Supervisor (http://supervisord.org/). A nightly cron job restarts Solr to prevent the memory cache from over- expanding.
$ sudo supervisorctl start Start solr
$ sudo supervisorctl stop Stop solr
$ sudo supervisorctl status solr grep solr` solr RUNNING pid 14899, solr 14899 1423 0 06:25 ? 00:00:32 java -Djetty.port=8983 - Status uptime 9:52:10 Dsolr.data.dir=/srv/primero/application/solr/data/production - Dsolr.solr.home=/srv/primero/application/solr -Djava.awt.headless=true -jar `$ ps -fA \ start.jar
$ RAILS_ENV=production Reindex bundle exec rake sunspot:reindex
Log /srv/primero/logs/solr/output.log Files
Run root, solr User
7 Beanstalkd
Beanstalkd
Beanstalk (https://github.com/kr/beanstalkd/wiki/faq) is a queue for processes. Currently, beanstalk is by Primero for bulk exports. Beanstalk prevents the system from getting overwhelmed by thousands of tasks and just queues up the tasks until the system has enough resources.
Start $ sudo systemctl start beanstalkd
Stop $ sudo systemctl stop beanstalkd
$ sudo systemctl status beanstalkd beanstalkd.service - Simple, fast work queue Loaded: loaded (/lib/systemd/system/beanstalkd.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2017-09-19 04:17:53 UTC; 11h ago Docs: man:beanstalkd(1) Main PID: 1086 (beanstalkd) grep beanstalkd` Status Tasks: 1 beansta+ 1086 1 0 04:17 ? 00:00:00 /usr/bin/beanstalkd -l Memory: 1.0M localhost -p 11300 -b /srv/primero/beanstalkd CPU: 5ms CGroup: /system.slice/beanstalkd.service └─1086 /usr/bin/beanstalkd -l localhost -p 11300 -b /srv/primero/beanstalkd Sep 19 04:17:53 primero-int-1-4 systemd[1]: Started Simple, fast work queue.
`$ ps -fA \
Log /srv/primero/logs/solr/output.log Files
Run root, primero User
8 Backburner
Backburner
Backburner (https://github.com/nesquena/backburner/wiki) is ‘queue consumer’ which takes items from the beanstalk queue and executes them as resources become available.
$ sudo supervisorctl start Start backburner
$ sudo supervisorctl stop Stop backburner
grep backburner` $ sudo supervisorctl status backburner primero 7915 1423 0 05:02 ? 00:00:00 /bin/bash backburner RUNNING pid /srv/primero/application/daemons/backburner-worker.sh Status 7915, uptime 11:04:12 primero 8005 7915 0 05:02 ? 00:00:02 ruby /srv/primero/.rvm/gems/ruby-2.1.5- railsexpress/bin/rake backburner:work `$ ps -fA \ primero 8566 8005 0 05:02 ? 00:00:00 ruby /srv/primero/.rvm/gems/ruby-2.1.5- railsexpress/bin/rake backburner:work
Log /srv/primero/logs/solr/output.log Files
Run root, primero User
9 CouchWatcher
CouchDB CouchWatcher
The CouchDB change watcher monitors CouchDB server and reacts to certain data events:
New records being created trigger Solr re-indexing
Changes to the forms metadata trigger rebuilding of the Rails mapping objects that interface with CouchDB and Solr
Detects and resolves record conflicts generated by bulk updates or CouchDB-to-CouchDB syncs and stored in CouchDB.
The Change Watcher is Ruby on Rails process running on an EventMachine (https://github.com/eventmachine/eventmachine). It is launched by Supervisor. When a new configuration bundle is applied, the Couch Watcher is restarted.
Start $ sudo supervisorctl start couch-watcher
Stop $ sudo supervisorctl stop couch-watcher
$ sudo supervisorctl status couch-watcher grep couch_changes` couch-watcher RUNNING pid 11497, uptime 11:07:36 primero 11587 11497 0 Status 05:02 ? 00:00:06 `$ ps -fA \ lib/couch_changes/base.rb
History /srv/primero/application/tmp/couch_watcher_history.json file
Log /srv/primero/logs/couch_watcher/output.log/srv/primero/logs/couch_watcher/production.log Files
Run root, primero User
10 Primero Scheduler
Primero Scheduler
The Primero scheduler runs periodic data maintenance tasks. It corrects or updates data:
Updates age based on birthday
Auto generates flags to prompt for follow ups (functionality not used in SL)
In the future other maintenance tasks will be added
The Primero scheduler is based on the Rufus Scheduler gem (https://github.com/jmettraux/rufus-scheduler). It is launched as a process using Rake tasks as an application process.
Note the RAILS_SCHEDULER_LOGDIR is /srv/primero/logs/scheduler on a default system.
For Sierra Leone AWS deploy, RAILS_SCHEDULER_LOGDIR is /data/logs/scheduler
$ sudo supervisorctl start Start primero-scheduler
$ sudo supervisorctl start Stop primero-scheduler
$ sudo supervisorctl status primero-scheduler primero-scheduler RUNNING pid grep primero-scheduler` Status 10823, uptime 11:13:15 primero 10823 1423 0 05:02 ? 00:00:00 /bin/bash /srv/primero/application/primero-scheduler-worker.sh `$ ps -fA \
Log /srv/primero/logs/scheduler/primero- Files scheduler.output
Run root, primero User
11 primeroctl Script
primeroctl Script
The primeroctl script should be used to act simulaneously on all of the primero applications. so you can simultaneously start, stop or check the status of: CouchDB, Nginx, Passenger, Solr, Beanstalkd, Backburner, CouchWatcher, and Primero Scheduler.
Start $ sudo /srv/primero/bin/primeroctl start
Stop $ sudo /srv/primero/bin/primeroctl stop
$ sudo /srv/primero/bin/primeroctl status Primero status: nginx: active (running) since Tue 2017-09-19 05:02:53 UTC; 12h ago passenger: active (running) since Tue 2017-09-19 05:02:49 UTC; 12h ago couch-watcher RUNNING pid 11497, uptime 12:38:48 Status who-watches-the-couch-watcher RUNNING pid 11604, uptime 12:38:47 primero-scheduler RUNNING pid 10823, uptime 12:38:55 backburner RUNNING pid 7915, uptime 12:39:24 beanstalkd: active (running) since Tue 2017-09-19 04:17:53 UTC; 13h ago solr RUNNING pid 14899, uptime 11:16:06 couchdb: active (running) since Tue 2017-09-19 05:01:44 UTC; 12h ago
Run User root
12