Table of Contents Introduction 1.1 System Users 1.2 CouchDB 1.3 Nginx 1.4 Passenger 1.5 Solr 1.6 Beanstalkd 1.7 Backburner 1.8 CouchWatcher 1.9 Primero Scheduler 1.10 primeroctl Script 1.11 1 Introduction Primero Application Primero is a Ruby on Rails application. It is served by Phusion Passenger which is proxied to by an Nginx web server. Static assets such as JS, CSS, and application icons and images are served by Nginx. Nginx is configured to serve Primero via HTTPS. It runs on default port HTTPS 443 which accepts traffic redirected from port 80. The backing database is CouchDB. When CouchDB is configured to replicate, it will be running on port 6984, via HTTPS. The core run-time system dependencies of Primero are: Nginx, which accepts external web traffic and routes it to Phusion Passenger or serves up static assets Phusion Passenger which launches and services 1-N Ruby processes CouchDB, the database backing Primero (see below) Apache Solr, the Lucene search index service (see below) The following services ensure proper Primero behavior when applying business rules, but will not cause fatal application errors if they are down: Couch Change Watcher notifier Primero Scheduler The system is designed to be deployed to Ubuntu 16.04, the long term support edition which is guaranteed to be supported through the end of 2020. All deployment is automated using Chef (https://www.chef.io/). 2 System Users System users In order to perform system maintenance, system support staff will need to log on to the Linux server. The system allows remote logon via SSH for the user ubuntu. This user has passwordless sudo (superuser) privileges and can be used to run system security upgrades and administer services. Designated support staff will be granted access to this user via public SSH keys. In order to grant access to the ubuntu user, append the designated support person’s key to the file /home/ubuntu/.ssh/authorized_keys. For more on SSH key access to linux refer to the following guide: https://www.linode.com/docs/security/use-public-key-authentication- with-ssh. Some administrative tasks such as data cleanup tasks must be done with the primero system user. This user is distinct from the application primero user. To switch from the ubuntu user to the primero user: $ sudo -Hu primero bash $ source ~/.rvm/scripts/rvm 3 CouchDB CouchDB CouchDB (http://couchdb.apache.org) is the backing database of Primero. It is accessible via HTTP on 127.0.0.1:5984 and is optionally reverse proxied by Nginx for HTTPS on 0.0.0.0:6984. This exposes CouchDB to the outside world for replication. The admin is exposed at https://cpims-myimplementation.primero.org:6984/_utils. If CouchDB is not exposed to the external world, one will need to set up a SSH tunnel and forward the port to the localhost in order to access it for maintenance: $ ssh -f -N -L 45984:localhost:5984 admin_user@your_primero_server The admin will now be available at http://localhost:45984/_utils CouchDB is started as a service on system startup via the Upstart Daemon (http://upstart.ubuntu.com/) Start $ sudo systemctl start couchdb Stop $ sudo systemctl stop couchdb $ sudo systemctl status couchdb couchdb.service - System-wide CouchDB instance Loaded: loaded (/lib/systemd/system/couchdb.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2017-09-19 05:01:44 UTC; 10h ago Main PID: 4515 (beam.smp) Tasks: 14 Memory: 77.9M CPU: 3min 44.982s CGroup: /system.slice/couchdb.service Status ├─ 4515 /usr/lib/erlang/erts-7.3/bin/beam.smp -Bd -K true -A 4 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/couchdb -- -noshell -noinput -os_mon start_memsup false start_cpu_sup fal ├─ 4553 sh -s disksup └─26184 /usr/bin/couchjs -S 134217728 /usr/share/couchdb/server/main.js Sep 19 15:09:31 primero-int-1-4 couchdb[4515]: [info] [<0.5509.1>] 127.0.0.1 - - PUT /primero_system_settings_production/52ffbe061b4b930ce941ff9a8f9b526c 201 Sep 19 15:09:31 primero-int-1-4 couchdb[4515]: [info] [<0.5510.1>] 127.0.0.1 - - GET /primero_sessions_production/e3255dc9a0964d1292f75e51113e0f87 200 Sep 19 15:09:31 primero-int-1-4 couchdb[4515]: [info] [<0.32009.0>] Starting index update for db: primero_system_settings_production idx: _design/SystemSettings Log /srv/primero/logs/couchdb/couch.log File Data /var/lib/couchdb Dir Run root, couchdb Users 4 Nginx Nginx The Primero application is served by the Nginx server. Shutting the server down halts the application. Note that Nginx is an optional runtime dependency of CouchDB for situations where external syncing is employed (as it is in Sierra Leone). Where CouchDB and the Primero Ruby on Rails application are deployed on the same box, they will share an instance of Nginx. Start $ sudo systemctl start nginx Stop $ sudo systemctl stop nginx $ sudo systemctl status nginx nginx.service - A high performance web server and a reverse proxy server Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2017-09-19 05:02:53 UTC; 10h ago Process: 12520 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS) Process: 12517 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS) grep nginx` Main PID: 12523 (nginx) root 12523 1 0 05:02 ? 00:00:00 nginx: master Tasks: 3 process /usr/sbin/nginx -g daemon on; master_process on; Status Memory: 8.0M CPU: 2.987s www-data 12524 12523 0 05:02 ? 00:00:02 nginx: CGroup: /system.slice/nginx.service worker process www-data 12525 12523 0 05:02 ? 00:00:00 nginx: ├─12523 nginx: master process /usr/sbin/nginx -g daemon on; master_process on worker process ├─12524 nginx: worker process └─12525 nginx: worker process Sep 19 05:02:53 primero-int-1-4 systemd[1]: Starting A high performance web server and a reverse proxy server... Sep 19 05:02:53 primero-int-1-4 nginx[12517]: nginx: [warn] "ssl_stapling" ignored, issuer certificate not found Sep 19 05:02:53 primero-int-1-4 nginx[12520]: nginx: [warn] "ssl_stapling" ignored, issuer certificate not found `$ ps -fA \ /srv/primero/logs/couchdb/nginx_server.log Log /srv/primero/logs/couchdb/nginx_error.log Files /var/log/nginx/access.log Run root, www-data User 5 Passenger Passenger Passenger is an open source, scalable web application server. If there is instability or unpredictable behavior experienced after applying a configuration bundle, stopping then starting Passenger is a good approach to resolving them. Start $ sudo systemctl start passenger Stop $ sudo systemctl stop passenger $ sudo systemctl status passenger passenger.service - Passenger Standalone Application Server Loaded: loaded (/etc/systemd/system/passenger.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2017-09-19 05:02:49 UTC; 10h ago Main PID: 12316 (PassengerWebHel) Tasks: 26 Memory: 277.7M CPU: 1min 31.858s CGroup: /system.slice/passenger.service ├─12298 PassengerWatchdog ├─12301 PassengerHelperAgent ├─12307 PassengerLoggingAgent ├─12316 PassengerWebHelper: master process /srv/primero/.passenger/standalone/4.0.59/webhelper-1.6.2-x86_64- linux/PassengerWebHelper -c /tmp/passenger-standalone.rjitj/config -p /tmp/passenger- ├─12317 PassengerWebHelper: worker process ├─12322 /srv/primero/.passenger/standalone/4.0.59/support-x86_64-linux/agents/TempDirToucher /tmp/passenger- Status standalone.rjitj --cleanup --daemonize --pid-file /tmp/passenger-standalone.rjitj/tem └─12552 Passenger RackApp: /srv/primero/application Sep 19 05:02:48 primero-int-1-4 systemd[1]: Starting Passenger Standalone Application Server... Sep 19 05:02:49 primero-int-1-4 passenger-worker.sh[12161]: =============== Phusion Passenger Standalone web server started =============== Sep 19 05:02:49 primero-int-1-4 passenger-worker.sh[12161]: PID file: /srv/primero/application/tmp/passenger.pid Sep 19 05:02:49 primero-int-1-4 passenger-worker.sh[12161]: Log file: /srv/primero/logs/rails//passenger.log Sep 19 05:02:49 primero-int-1-4 passenger-worker.sh[12161]: Environment: production Sep 19 05:02:49 primero-int-1-4 passenger-worker.sh[12161]: Accessible via: http://127.0.0.1:4000/ Sep 19 05:02:49 primero-int-1-4 passenger-worker.sh[12161]: Serving in the background as a daemon. Sep 19 05:02:49 primero-int-1-4 passenger-worker.sh[12161]: Problems? Check https://www.phusionpassenger.com/documentation/Users%20guide%20Standalone.html#troubleshooting Sep 19 05:02:49 primero-int-1-4 passenger-worker.sh[12161]: =============================================================================== Sep 19 05:02:49 primero-int-1-4 systemd[1]: Started Passenger Standalone Application Server. `$ ps -fA \ Log /srv/primero/logs/rails/passenger.log File Run root, primero User 6 Solr Solr Apache Solr (http://lucene.apache.org/solr/) is used as the search and query index for Primero. All records saved in Primero are indexed in Solr. Solr is a Java process that runs on a Jetty application server. Solr is launched by Supervisor (http://supervisord.org/). A nightly cron job restarts Solr to prevent the memory cache from over- expanding. $ sudo supervisorctl start Start solr $ sudo supervisorctl stop Stop solr $ sudo supervisorctl status solr grep solr` solr RUNNING pid 14899, solr 14899 1423 0 06:25 ? 00:00:32 java -Djetty.port=8983 - Status uptime 9:52:10 Dsolr.data.dir=/srv/primero/application/solr/data/production - Dsolr.solr.home=/srv/primero/application/solr -Djava.awt.headless=true -jar `$ ps -fA \ start.jar $ RAILS_ENV=production Reindex bundle exec rake sunspot:reindex Log /srv/primero/logs/solr/output.log Files Run root, solr User 7 Beanstalkd Beanstalkd Beanstalk (https://github.com/kr/beanstalkd/wiki/faq) is a queue for processes. Currently, beanstalk is by Primero for bulk exports. Beanstalk prevents the system from getting overwhelmed by thousands of tasks and just queues up the tasks until the system has enough resources.