Mataara Documentation Release 0.1
Catalyst IT
Sep 08, 2020
Architecture
1 Name 3 1.1 Features...... 3 1.2 Client-Server Model Design...... 4 1.3 Front-End Architecture...... 5 1.4 Getting Started...... 8 1.5 Installing the Client...... 8 1.6 Configuration...... 10 1.7 Drush Commands...... 12 1.8 Getting Started...... 13 1.9 Installing the Server...... 14 1.10 Generating an OpenSSL key-pair...... 18 1.11 Configuration...... 19 1.12 Configuring SAML authentication...... 22 1.13 Getting Started...... 25 1.14 Quick Start guide...... 26 1.15 Environment setup...... 26 1.16 Fabric tasks...... 29 1.17 Useful Commands...... 31 1.18 Database Backup and Restore...... 33 1.19 Setting up your development environment for ReactJS...... 34 1.20 Report Processing...... 34 1.21 Mataara report format (JSON)...... 34 1.22 Archimedes report format (legacy XML)...... 39 1.23 Key Development Technologies...... 42 1.24 Source code repository structure...... 42
i ii Mataara Documentation, Release 0.1
Mataara is a security reporting tool that collects information on websites and their dependencies, uploading reports to a server that monitors module vulnerabilities and site configuration changes from a central location. The primary intent of the Mataara project is to provide a system that helps site hosting and development organisations to manage and secure their software configurations across large numbers of sites. The documentation for Mataara is organised into a number of sections: • System Architecture • Drupal Client • Server • Development • Development Quickstart Guide The Mataara project was created at Catalyst IT.
Architecture 1 Mataara Documentation, Release 0.1
2 Architecture CHAPTER 1
Name
An earlier version of this project was known as Archimedes, and a significant proportion of the code still refers to itself as such. Because naming things is hard, and there were already several other projects called Archimedes out there, it was decided to change the name to something more distinctive. Mataara (pronounced by syllable: ma-ta-a-ra) is a word from the Maori¯ language that means vigilance, alertness, watchfulness. (It is unrelated to the Kenyan tea factory of the same name.)
1.1 Features
The following is a description of the Mataara system and some of its features. For a better and more complete understanding of the software, it might be best to download the source code of the client and server modules, and experiment with the system in a test environment. Mataara currently consists of a server built in Django and ReactJS, communicating via a REST API, and a Drupal CMS client module. It is intended to be an extensible system, and development of a client for the SilverStripe CMS is underway.
1.1.1 Drupal Client Reporting
Mataara’s Drupal client is an installable module that periodically sends reports to the Mataara server (daily by default, but configurable within Drupal’s cron system). The primary payload of the report is a list of the modules and themes installed on the host website, that can be checked against drupal.org’s APIs to determine whether they have known security vulnerabilities. Reports also contain details pertinent to the security posture of the site, including: • A generated site key • Hostname • Core platform version • Databases connected to the site
3 Mataara Documentation, Release 0.1
• The number of users, administrator roles and content nodes Report data is encoded in JSON and public-key encrypted during transport. Reports can be sent to the server via either HTTP/S POST requests or by email attachment. The report format is designed to be customisable, and new items can be added to the report by extending a base ReportItem PHP class in the client module. For use in the system console, Drush commands are available to send reports or to generate a report in various for- mats. Settings can also be configured through the standard Drupal administration interface, including the transport mechanism (HTTP or email) and the server’s public key for encryption. The Drupal client module has legacy support for Drupal 6, is compatible with Drupal 7 and 8, and will support Drupal 9. In order to ensure it can be used in as many hosting environments as possible, has no dependencies on any other Drupal modules.
1.1.2 Django Server Structure
The Django server receives reports via either an HTTP/S endpoint or via email, where they are decrypted and placed in a task processing queue. Queue workers ingest the reports and match them to Site nodes within the system, updating them with any new information contained in the reports. Reports are also archived in their raw JSON format for reference, and potentially for further processing in other tools. Reports are matched against sites using their generated site key, and can also be grouped into a “site group”. This is intended for multiple versions of the same site across different environments (e.g. development, staging, production), to make them easier to keep track of and to store common information. Note: At the time of writing this site group subsystem is quite new, and does not have front-end functionality yet, but it is envisioned that this will allow for integrations with other systems for issue management, bug tracking, customer relationship management, etc. The ReactJS frontend is primarily for viewing and querying site data, including modules and security advisories. At present, server management commands are primarily handled through the Django admin web interface, or by console management commands.
1.1.3 Drupal Security Advisories and Project Release History
Earlier versions of Mataara used Drupal.org advisories as the primary source of truth for matching against insecure module versions. The data within the advisories is sometimes hard to automatically parse. As a result, the system has moved to using the release history API as its primary means of making Drupal modules as insecure. The Mataara server periodically checks the Drupal.org Release History API for every project it has encountered, checking their release types for insecure versions. Drupal security advisories are still checked via the Drupal.org D7 APIs, and saved for display to Mataara users. The legacy advisory parser will also automatically correct errors in some older advisories. Advisory errors and warnings will be logged to a Sentry server if you have one available. In the frontend, sites can be checked for vulnerable module releases (including Drupal core releases), and modules with vulnerable releases can list the sites potentially affected by those vulnerabilities.
1.2 Client-Server Model Design
Mataara uses a client-server model that practises inversion of control; the server passively receives encrypted reports from client modules installed in the other systems.
4 Chapter 1. Name Mataara Documentation, Release 0.1
This allows Mataara to be a write-only repository of information about sites over time, decoupled from any “dial in” command-and-control systems, so that it can capture an understanding of the security posture of websites without needing to have ongoing privileged access to each host. This reduces the inbound attack surface on individual sites, and provides a means for managing the release update cycle on sites hosted in managed secure environments such as intranets, where it may not be desirable for them to call out directly to public update servers for information. The Mataara server will independently check for version updates and security advisories, allowing them to be matched against incoming reports. A planned future enhancement is to provide a subscription-notification system for users, so that users who are subscribed to a site or group of sites can be notified when a new insecure version is discovered.
1.3 Front-End Architecture
1.3.1 Introduction
Mataara uses the ReactJS framework to provide a front-end user interface, built on top of Django REST Framework APIs. The React frontend also uses Redux to store the state of the app in a single state tree.
What does Redux do, and how does it work?
Redux is a predictable state container for JavaScript apps. It helps you write applications that behave consistently, run in different environments (client, server, and native), and are easy to test. Redux only has a single store that holds no logic by itself. Actions are dispatched and handled directly by the store, eliminating the need for a standalone dispatcher. In turn, the store passes the actions to state-changing functions called reducers. API or client-side middleware may also be included to add authentication, or to stop, modify or add more actions.
Redux Components
Redux features four major system components: Store: Controls the state. Actions: Describe the changes we want to make. Reducers: Services that state change based on the requested action. Middleware: Handles housekeeping tasks of the app. N.B.: Reducers never change their state; they are immutable. An entire application state is kept in a single location, the store- This provides enormous benefits during debugging, serialization and development.
Redux Principles
Redux has three major principles to be aware of: State Principle: The state of an application is stored in an object tree within a single store, meaning everything that changes in an app is contained in a single object called state or state tree. Action Principle: A state tree cannot be modified or written to. To change a state, an action has to be dispatched. An action ensures that views nor the network callbacks will ever write directly to the state. Actions must be plain JS objects, that can be logged, serialized, stored and later replayed for debugging or testing purposes.
1.3. Front-End Architecture 5 Mataara Documentation, Release 0.1
Reducer Principle: Reducers exists to specify how the state tree is transformed by actions. Reducers are pure func- tions that takes the previous state and an action then return the next state. Using the default switch, it must return current state for undefined actions. Further description can be found here: https://redux.js.org/introduction/three-principles
1.3.2 Basic Requirements
• NodeJS 5.0.0 or higher, plus npm • React + Redux • Babel 6 • Webpack • ESLint • Redux DevTools + Logger middleware (easily removable/replaceable if you need something else) • Jest to run unit tests
1.3.3 Directory / File Structure
devServer.js README.md webpack-stats.json src index.js actions store components constants containers api.js reducers setupJest.js webpack.config.prod.js index_frontend.html package.json webpack.config.dev.js
Actions Directory
Actions are payloads of information that send data from the application to the store. They are the only source of information for the store. We put all our action files in this directory.
Constants Directory
All types of action that are performed within the app are defined in the ActionsTypes.js file. The date.js file contains a standard date formatting string.
6 Chapter 1. Name Mataara Documentation, Release 0.1
Reducers Directory
Actions describe events that have happened, but don’t specify how the application’s state changes in response. This is the job of reducers. Because we are using Redux, we combine reducers using the Redux tool combineReducers.
Containers Directory
Container components are a React pattern used to separate data fetching from rendering concerns. The idea is simple: A container does data fetching, and then renders that data using its corresponding sub-component.
Store Directory
The Store is the object that brings actions and reducers together. We only have a single store in our Redux application. The store has the following responsibilities: • Holds the application state • Allows access to that state via getState() • Allows it to be updated via dispatch(action) • Registers listeners via subscribe(listener) • Handles unregistering of listeners via the function returned by subscribe(listener) api.js File
This file is responsible for building the query strings, fetching data from Django REST API and parsing the JSON responses. Every action uses API file functions to fetch data.
Components Directory
Components in ReactJS are like LEGO pieces. Every part of the application’s visuals should be wrapped inside a self- contained component module. Our components use Material UI, a set of React components that implement Google’s Material design.
Unit Test Directory
We use Jest to create unit tests. Jest test files are placed into a subdirectory called __tests__. It will look automat- ically for test files inside that subdirectory.
1.3.4 ESLint
We use the ESLint JavaScript linting utility to find problematic patterns or code that don’t adhere to certain style guidelines. Linting rules are defined in a .eslintrc file. You can find more ESLint rules here.
1.3. Front-End Architecture 7 Mataara Documentation, Release 0.1
1.4 Getting Started
This section covers the Mataara client module for Drupal, including installation, configuring settings and issuing console commands via the Drupal shell utility, drush.
1.4.1 Requirements
The Drupal Mataara client is supported for Drupal 6, 7 and 8 sites, with support planned for Drupal 9. You will need sufficient administrator access on the site to install and configure modules. Console access to run drush commands is not strictly necessary, but may be useful while setting up and testing the configuration. Sending reports to a Mataara server via email requires that your Drupal site and server have been configured to work with a mail transfer agent (e.g. Postfix or sendmail).
1.4.2 Choosing a Method
Mataara supports sending encrypted messages via two methods: email and HTTP/S. Each of them has advantages. • Using HTTP or preferably HTTPS sends data to the server directly for immediate processing. However, it will need to be able to make a direct network connection which may require some extra firewall configuration on some networks. If the Mataara client cannot make a connection for some reason, the report will not be sent and it will have to be either re-sent manually or wait until the system’s next scheduled reporting time. • Using email is less direct, but in situations where the network is less reliable the mail transfer agent can queue messages and attempt to send them multiple times. Depending on the volume of messages being handled, it may also mean that reports come in a long time after they are generated (usually minutes, but possibly longer). Email messages can also be archived external to the Mataara server, for backup or auditing purposes.
1.5 Installing the Client
Note: This guide assumes you are installing the Mataara Client on a Drupal 7 site. If you are using Drupal 6 or 8, replace any references to the ‘7.x’ branch with ‘6.x’ or ‘8.x’ accordingly.
1.5.1 Cloning from Git
To clone the Archimedes Client project from git, run the following: cd {drupal web root directory}/sites/all/modules git clone--branch 7.x https://gitlab.com/mataara/drupal-module.git mataara_client
1.5.2 Composer installation in Drupal 8
Add this to your composer.json repositories section:
{ "type":"vcs", "url":"https://gitlab.com/mataara/drupal-module.git" }
And run composer require drupal/archimedes_client.
8 Chapter 1. Name Mataara Documentation, Release 0.1
1.5.3 Enabling the module
You will then need to enable the Mataara Client module on your Drupal site.
Using Drush cd {drupal web root directory} drush en archimedes_client-y
Using the web interface
• Navigate to http://your-client-site.com/admin/modules • Find Archimedes Client in the Archimedes section and check the ‘Enabled’ checkbox • Click Save configuration at the bottom of the page
1.5.4 Initial configuration
Once installed, the client needs to be configured before it is able to send reports to a Mataara server. The following must be setup before the client will run correctly: • The server location must be defined; an email address or URL, whichever is relevant to the reporting method • A reporting method must be chosen (email or http) • A public key must be set These can be configured using the instructions in Client Configuration. Alternatively, a variation on the following could be used 1. Set the server location drush vset archimedes_client_server_email [email protected] drush vset archimedes_client_server_url http://your-server-site.com/reports/endpoint
2. Choose a reporting method (choose one) drush vset archimedes_client_server_method email drush vset archimedes_client_server_method http
3. Set the public key drush ev"variable_set('archimedes_client_pubkey','`cat /path/to/publickey.pem`');"
The drush ev command is used rather than drush vset here so that Drush doesn’t mistake the starting ----BEGIN PUBLIC... string for a CLI option.
1.5.5 Sending a test report
To verify that the client is setup correctly, you can trigger an immediate report to the server:
1.5. Installing the Client 9 Mataara Documentation, Release 0.1
Further uses for Drush
Refer to the documentation for the arch-send Drush command.
1.6 Configuration
The Mataara client has some configuration options that allow you to change how often and it reports to the server, and what transport method it uses.
1.6.1 Changing client settings
There are a number of ways to configure Mataara - using Drush or the web interface as described, or more advanced users may use Strongarm to preset the configuration as part of a site profile.
Using Drush
When changing settings using Drush, you must know the exact value that the Drupal variables need to be set to otherwise the client may not operate correctly.
cd {drupal web root directory} drush vset archimedes_client_SETTING_NAME SETTING_VALUE
Using the web interface
• Log in as an administrator • Navigate to http://your-client-site.com/admin/config/archimedes_client/settings • Change the settings to the desired values (see below) • Click Save configuration at the bottom of the page
Using settings.php
When creating a base install for Drupal, it may be desirable to preset the configuration of the client - such as the server address and public key. This can be achieved by using Strongarm in a profile or Features module, or by setting variables in the site’s settings.php file as follows:
// Archimedes server public key $conf['archimedes_client_pubkey'] = "-----BEGIN PUBLIC KEY----- 1234567890ABCDEFG1234567890ABCDEFG ABCDEFG1234567890ABCDEFGH123456790 1234567890ABCDEFG1234567890ABCDEFG ABCDEFG1234567890ABCDEFGH123456790 -----END PUBLIC KEY-----";
// Archimedes server URL $conf['archimedes_client_server_url'] = "https://your-server-site.com/archimedes/
˓→endpoint"; (continues on next page)
10 Chapter 1. Name Mataara Documentation, Release 0.1
(continued from previous page)
// Archimedes server email address $conf['archimedes_client_server_email'] = '[email protected]';
// Archimedes report sending method - "email" or "http" $conf['archimedes_client_server_method'] = 'http';
?>
1.6.2 Client settings reference
The following settings are available in the Mataara Client:
Reporting Method
Mataara can be configured to send reports to the server over different channels: • email: Reports are emailed as attachments to a mailbox monitored by the server. • http: Reports are posted directly to the server over HTTP(s) using POST. All reporting methods are encrypted.
Variable archimedes_client_server_method Default email
Reporting URL
URL of the Mataara server’s HTTP(s) POST endpoint.
Variable archimedes_client_server_url Default (not set)
Reporting Email
Email address of a mailbox monitored by the Mataara server.
Variable archimedes_client_server_email Default (not set)
Reporting Interval
Minimum time in seconds between Mataara reports being sent. This time may be longer depending on the frequency with which Drupal’s cron is run.
Variable archimedes_client_cron_interval Default 86400
1.6. Configuration 11 Mataara Documentation, Release 0.1
Public Key
Mataara uses a public/private key pair to encrypt data during transmission (to prevent eavesdropping). Encrypted reports include an expiry time in order to minimise the effect of a reply attack. This variable holds the public key used to encrypt the reports sent to the server.
Variable archimedes_client_pubkey Default (not set)
Report Expiry Time
Number of seconds after sending that a report becomes invalid. The server must have enough time to receive the report before it expires, however the expiry should be set to a minimal value to minimise the effect of a replay attack.
Variable archimedes_client_report_expiry Default 1800
1.7 Drush Commands
The client module implements a number of Drush commands that can be used to send or display reports at leisure for debugging purposes.
1.7.1 Generate a report (arch-report)
Generate and display a Mataara client report.
Options format: The format in which the report is displayed. • rendered: Rendered as a table (default) • json: JSON • json-pretty: JSON (pretty-printed) • encrypted: Encrypted JSON
Aliases ar Examples • drush arch-report: Display a report • drush arch-report json: Display a re- port in JSON format • drush arch-report encrypted > report.enc: Save an encrypted report to file
1.7.2 Send a report (arch-send)
Send a Mataara Client report to the default, or a desired server.
12 Chapter 1. Name Mataara Documentation, Release 0.1
Options method: How the report is sent. Defaults is the site’s configured method. • email: via email attachment • http: via HTTP(s) POST location: Email address or URL to send to (depend- ing on the chosen method) Aliases as Examples • drush arch-send: Send a report using the configured method & location • drush arch-send http: Send a report us- ing HTTP (using the configured URL) • drush arch-send http http:// server.com/endpoint: Send an HTTP report to this URL
The arch-send command may fail with the message “Cannot send report: could not find a valid hostname for this site.” This means that drush cannot determine the base URL to use, so Mataara cannot identify the site that it is sending from. There are multiple ways to address this, including: • Fixing the underlying hostname issue, if you have the system permissions to do so • Using an alias in a drushrc.php file • Specifying the base URL in the command line, e.g. drush -l https://example.com arch-send
1.8 Getting Started
Here’s how to get a Mataara Server configured and some notes on basic usage - such as generating OpenSSL keys and setting up products from sets of websites.
1.8.1 Requirements
The Mataara server uses Django and thus requires Python to be installed. Python 3 is strongly preferred. Other requirements: • web server • SSL certificate (if you want to access the site over HTTPS) • PostgreSQL Database • a user authentication source (such as LDAP) The instructions here describe installing Mataara in a single-server, non-high availability manner. The components are all standard and the stack can easily be expanded. Instructions have been tested on Ubuntu Trusty (14.04 LTS) and Xenial (16.04 LTS). The server can optionally integrate with a Sentry server for error reporting.
1.8. Getting Started 13 Mataara Documentation, Release 0.1
Network Requirements
The following firewall rules will be required for the application in addition to any used for standard system manage- ment and monitoring. Inbound: • HTTP(S) Access Outbound: • HTTP(S) to advisories websites • IMAP(S) or POP3(S) for polling a mail server for reports • LDAP(S) if LDAP will be used for authentication • SMTP for sending error emails out A proxy server may optionally be configured for outbound HTTP(S).
1.8.2 Key Directories and Files
The following default directories and files are used in these installation instructions. If you choose other options, please update accordingly as you install Mataara. /etc/archimedes/archimedes.ini Configuration file for Mataara itself. /etc/nginx/sites-available/default Configuration file for nginx that will define the external Mataara site and internal gunicorn site. /etc/supervisor/conf.d/archimedes.conf Mataara specific configuration settings for supervisor. /usr/share/archimedes/static Directory that will contain the static files to be served up directly by the web server. /var/www/archimedes Directory to contain the Mataara virtual environment and the actual program
1.9 Installing the Server
Currently the Mataara Server is only available for download from its git repository.
1.9.1 Download wheel
Note: TODO - Package up and publish wheels somewhere
1.9.2 Cloning from Git
Instead of downloading and installing the wheel directly, we can grab the source from git using: git clone [email protected]:mataara/Mataara-Server.git mataara
14 Chapter 1. Name Mataara Documentation, Release 0.1
1.9.3 Add Node.js repository
Node.js is used in the build process. It is not required on a production server if Mataara is installed by wheel. Mataara has been tested using the NodeJS v6 LTS. It may work without any problems using later versions. LTS versions are recommended. If your operating system provides a suitable LTS it is recommended to install that version so that subsequent updates are automatically detected and can be applied. An individual version can be downloaded from the Node.js download page. If not available directly from the operating system provider, it is recommended that an LTS version is installed through a suitable third-party package manager. Instructions are available on the Installing Node.js via package manager page.
1.9.4 Ubuntu installation
Install JavaScript build dependencies
We need to install npm so that we can compile the JavaScript front-end bundle. Install nodejs: sudo apt-get install-y nodejs
Create JavaScript bundle
The JavaScript bundle can be created live on a production server, or on a build server prior to wheels being built: cd archimedes/frontend npm run build
Install redis
Redis is used internally by Mataara. There should be specific configuration required if it can run on its default TCP port of 10448. Install it using: sudo apt-get install-y redis-server
Database setup
Mataara has been developed with PostgreSQL
1.9. Installing the Server 15 Mataara Documentation, Release 0.1
Install web server
Mataara will work with any web server that supports WSGI. These instructions are for nginx. Install using: sudo apt-get install-y nginx
Assuming this site will be the default one for nginx, configure /etc/nginx-sites-available/default to contain: upstream gunicorn { server localhost:4005; } server { listen 80; client_max_body_size 20M; location / { proxy_pass http://gunicorn; proxy_set_header X-Request-Id $pid-$msec-$remote_addr-$request_length; }
location /static/ { alias /usr/share/archimedes/static/; } }
Warning: This won’t work until we’ve installed gunicorn and archimedes
Note: TODO - add the extra commands required for this to run on HTTPS
Install other packages
To install the core packages required to run Mataara run: sudo apt-get install-y python3-all libxml2 libxslt1.1
To install the additional packages required to build Mataara (required if installing from source) run: sudo apt-get install-y python3-dev, python3-setuptools, libpq-dev, libldap2-dev,
˓→libsasl2-dev, libssl-dev, libffi-dev
Install Python packages
Pip is used to install all the Python packages. Unfortunately, the version of pip distributed by many systems is old. Hence, the script bin/get-pip.py which will bootstrap us a sufficiently up-to-date version. First we’ll create the venv, install our pip and check if there are any updates, then use it to install the required packages:
VENV="/var/www/archimedes" /usr/bin/python3 -m venv --system-site-packages --without-pip ${VENV} ${VENV}/bin/python3 archimedes/bin/get-pip.py --ignore-installed (continues on next page)
16 Chapter 1. Name Mataara Documentation, Release 0.1
(continued from previous page) ${VENV}/bin/pip install -U pip ${VENV}/bin/pip install -r /vagrant/requirements/dev.txt
Note: TODO - when we provide a pre-built wheel, it will be possible to switch the above requirements to ‘common.txt‘
Install supervisor
To install supervisor, run: sudo apt-get install-y supervisor
Create the supervisor configuration file /etc/supervisor/supervisord.conf :
[unix_http_server] file=/var/run/supervisor.sock ; (the path to the socket file) chmod=0770 ; socket file mode (default 0700) chown=root:supervisor
[supervisord] logfile=/var/log/supervisor/supervisord.log ; (main log file;default $CWD/supervisord.
˓→log) pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid) childlogdir=/var/log/supervisor ; (AUTO child log dir, default $TEMP)
; the below section must remain in the config file for RPC ; (supervisorctl/web interface) to work, additional interfaces may be ; added by defining them in separate rpcinterface: sections [rpcinterface:supervisor] supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl] serverurl=unix:///var/run/supervisor.sock ; use a unix:// URL for a unix socket
; The [include] section can just contain the "files" setting. This ; setting can list multiple files (separated by whitespace or ; newlines). It can also contain wildcards. The filenames are ; interpreted as relative to this file. Included files *cannot* ; include files themselves.
[include] files = /etc/supervisor/conf.d/*.conf
Now create the Mataara specific configuration file /etc/supervisor/conf.d/archimedes.conf :
[program:gunicorn] command=${venv}/bin/gunicorn --log-level=DEBUG archimedes.wsgi --bind=0.0.0.0:4005 --
˓→reload --workers 4 user=vagrant numprocs=1
[program:celery] command=${venv}/bin/celery --app=archimedes.celeryapp --no-color worker --
˓→loglevel=DEBUG --beat --autoscale=6,2 (continues on next page)
1.9. Installing the Server 17 Mataara Documentation, Release 0.1
(continued from previous page) user=vagrant stopsignal=INT numprocs=1 environment=AUTOSCALE_KEEPALIVE="120"
Note: TODO - work out replacement for ${venv} above
Restart the supervisor service using: supervisorctl restart all
Install Mataara itself
Steps involved in installing Mataara itself (when not using wheels):
${VENV}/bin/pip install archimedes cd ${VENV}/bin archimedes migrate --noinput cd archimedes/frontend npm run build archimedes collectstatic --noinput --link --clear
If you will be receiving reports by email, and want to use encryption, you will need a public-private set of encryption keys. For more information, see Generating an OpenSSL key-pair Now customise your Configuration in /etc/archimedes/archimedes.ini.
1.10 Generating an OpenSSL key-pair
Mataara uses an OpenSSL RSA key pair to encrypt reports, and supports encrypting the private key with a passphrase. If you don’t already have a key pair, you can generate one using the OpenSSL tools. The key pair can also be used for SAML communications during login. You can optionally create a second key pair just for SAML, or re-use the first key pair. Note that SAML requires the public key to be in an X509 certificate. A passphrase can be used to protect the private key used for reports, but not for SAML authentication. To generate the private key without protecting it with a passphrase: openssl genrsa-out private.pem 4096
To generate a random passphrase and then create a private key protected by a passphrase: pwgen-1s 20 openssl genrsa-des3-out private.pem 4096
For encryption reports, you need to export the public key: openssl rsa- in private.pem-outform PEM-pubout-out public.pem
For SAML, you need to generate a certificate containing the public key:
18 Chapter 1. Name Mataara Documentation, Release 0.1
openssl req-new-x509-key private.pem-out publickey.cer
1.10.1 Further information
Still confused? Checkout this blog for a more in-depth guide on how OpenSSL keys work: https://rietta.com/blog/ 2012/01/27/openssl-generating-rsa-key-from-command/
1.11 Configuration
The Mataara server has some configuration options that allow you to set up encryption and other features. The settings are all stored in the file /etc/archimedes/archimedes.ini. The settings file is structured into sections introduced by a header in square brackets. For example, the following extract defines the django section and the first configuration option:
[django] debug= false
1.11.1 Django standard configuration options
Further information on all the Django standard options can be found at https://docs.djangoproject.com/en/1.11/ref/ settings/ This section summarises some of the key ones: admin_email Specify email address(es) used to receive error reports. Requires you to have defined an outbound email server. allowed_hosts Define which host(s) this website can appear to be. For development environments where we’re less concerned, permit all hosts using * - we’ll probably be refer- encing it by IP address anyway. For production environments, should configure localhost and the FQDN, e.g. allowed_hosts = local archimedes.example.com configuration_files Can be used to split out configuration to additional files. debug Set to True in development environments, otherwise False. secret_key Must be set. A default in used in vagrant, but a unique value must be set for production servers. To generate a new key of a suitable length, you can try this website: http://www.miniwebtool.com/ django-secret-key-generator/ media_root Directory used to define a specific writeable location for the storage of uploaded files and processed emails static_root Define the directory that will contain your static assets, e.g. /usr/share/archimedes/static
1.11. Configuration 19 Mataara Documentation, Release 0.1
1.11.2 Configuration options for other modules
Celery configuration broker_url URL for celery to use when talking to redis. Unless you are running the redis server on non-standard port, or another machine, this should be safe to leave at the default. Default: broker_url = redis://localhost:6379/0
Database configuration host Hostname of the database. To connect using UNIX sockets to PostgreSQL running on the same server leave the value blank. name Name of the Mataara database. In the vagrant development environment this defaults to vagrant. pass Password for the databae user, if defined. user Name of the database user that should be used to connect to the database. In the vagrant development environment this defaults to vagrant.
LDAP configuration
Only specify settings in this section if you are using LDAP for authentication. Further information on the settings can be found at https://pythonhosted.org/django-auth-ldap/index.html bind_dn LDAP bind DN to use. Leave blank for anonymous bind connections. bind_password Password to use with the LDAP bind DN. Leave blank for anonymous bind connections. ldap_starttls Set to True if the LDAP server URI does not use ldaps but the server support STARTTLS and you want, or need, to use it. ldap_ignore_cert Set to True to ignore any SSL certificate errors from the LDAP server. Useful in development environments. search_dn LDAP search string to be used when searching for users. Typical form is uid=%(user)s,ou=users, dc=example,dc=com server LDAP server URI of the form ldap://hostname/ or ldaps://hostname/. A comma separated list of URIs can be provided if you wish to specify backup LDAP servers.
Redis configuration location URL for the redis service. Unless you are running the redis server on non-standard port, or another machine, this should be safe to leave at the default. Default: location = redis://localhost:6379/0
20 Chapter 1. Name Mataara Documentation, Release 0.1
Mail server configuration
server Hostname of a mail server to be used for outbound email port Port to be used with the mail server (probably 25).
Sentry configuration
Only specify settings in this section if you are using integration with Sentry. dsn Sentry DSN value to be used in reporting errors.
Web proxy configuration
Often, but not always, a web proxy will handle both HTTP and HTTPS on the same port. http URL of a web proxy that will handle HTTP requests. https URL of a web proxy that will handle HTTPS requests.
1.11.3 Mataara configuration options
Encryption for reports
If a public-private keypair has been generated and is being used by clients to encrypt reports, it can be defined here. reports_keyfile Path to the file with the private key. Ensure that suitable file permissions are applied unless a passphrase is specified. reports_keyfile_passphrase Optional passphrase to use if the private key has been protected by a passphrase. If a passphrase is specified, ensure that suitable file permissions are applied to the settings file.
Inbound email reports mailbox_uri URI connection string for the mailbox that will be used to receive email reports. Typical form is imap+ssl//myusername:[email protected]. Note, the password will be stored in plain-text on the server. Ensure that you have suitable file permissions applied to protect the settings file.
Outbound email email_subject_prefix Used with outbound emails to indicate that they are from the Mataara server. Typical value is [Archimedes]. from_email Specify the from email address to be used for outbound emails.
Report processing report_processing_retry_delay Delay in automatic processing of a report in scheduled tasks. reports_processing_max_retries Maximum number of time automatic processing of a report is attempted during a scheduled task.
1.11. Configuration 21 Mataara Documentation, Release 0.1
Updates
drupal_update_server Default value is https://updates.drupal.org/`
1.12 Configuring SAML authentication
1.12.1 Dependencies
The following instructions use the djangosaml2 package. This requires the xmlsec binary to be installed. On Ubuntu systems it can be installed using:
sudo apt-get install xmlsec1
The djangosaml2 package can be manually installed using:
pip install djangosaml2
On a server, it should be added to the requirements file requirements/common.txt.
1.12.2 Enable SAML in config file
Define a unique SAML entity identifier URL for the website in the archimedes.ini configuration file. Defining a SAML entity identifier will cause Mataara to automatically load the SAML modules. Although the entity identifier does not have to map to a real-world resolvable web page, by convention it is often desirable to specify the URL that advertises the server’s SAML metadata.
[saml] entityid= http://mataara.company.com/saml2/metadata
Note: Use an EntityID that describes the service, not the server currently hosting it, so that the same value can persist over time.
An optional root URL can be set for the Mataara server that will be used by the IdP to contact it. The default is to extract the relevant value from the entityid.
[saml] root_url= http://mataara.company.com/
xmlsec1 binary location
Mataara expects the xmlsec1 binary to be installed to /usr/bin/xmlsec1. This value can be overwritten in the archimedes.ini file:
[saml] xmlsec_binary=/path/to/binary
22 Chapter 1. Name Mataara Documentation, Release 0.1
Define IdP user attributes
Mataara will default to only checking for the existing of the attribute uid in the assertion that the IdP provides. Alternative, or additional, required attributes can be defined in the configuration file. These are specified as a command separated set of strings. For example:
[saml] required_attributes= uid,email
Mataara will also default to using the uid for the username value. You can also optionally tell Mataara which SAML IdP user attributes should be used for the user’s name and email address. For example:
[saml] username= uid email= mail first_name= givenName last_name= sn
The above values were taken from an IdP that uses LDAP as its back-end authentication source. Other sources are likely to have different attributes that need to be configured.
Configure Mataara to accept unknown attributes
Mataara will attempt to identify the user’s attributes using standard unique identifiers. This configuration setting allows Mataara to accept attributes that are not recognised. It will rely on being able to identify an attribute whose name matches that defined in the primary_attribute value:
[saml] allow_unknown_attributes= True
This bug is known to affect SimpleSamlPHP version 1.8.2 (see djangosaml2 documentation).
Define metadata expiry
The generated metadata has a default expiry of 24 hours. This is normally sufficient for most Identity Provider’s that dynamically refresh. If you want to set a longer expiry time there is an optional configuration option for the number of hours before expiry:
[saml] expiry=
If you want the metadata to never expire, set a value of 0.
Create SAML crypto keys
You’ll need to create a public/private key pair for SAML to use when protecting communications from the Mataara server to the IdP server. See Generating an OpenSSL key-pair for more information. Specify these in the archimdes.ini configuration file using:
[saml] key_file=/path/to/private.key cert_file=/path/to/public.cer
1.12. Configuring SAML authentication 23 Mataara Documentation, Release 0.1
Note: The certificate file, containing the public key, needs to be a full X509 certificate file and not just a public key. You can verify this using openssl x509 -noout -text -in public.cer
If you need to use separate key pairs for signing and encryption these can be specified using:
[saml] enc_key_file=/path/to/private.key enc_cert_file=/path/to/public.cer
Install IdP server metadata
Obtain the IdP’s server metadata and store it in a file. Mataara uses the default filename remote_metadata.xml. An alternative filename can be defined in archimedes.ini:
[saml] remote_metadata_file=/path/to/metadata.xml
The IdP entity identifier and endpoint information should be automatically determined from the metadata. However, if it is necessary to specify the specific IdP entity identifier to use, for example if the metadata contains more than one, it can be manually specified in archimedes.ini. Additionally, the binding URLs for the IdP’s single sign-on service and single logout service can be manually specified: idp_entityid= [saml] idp_sso= idp_sls=
Contact details
Optional technical and/or administrative contact details can be specified in archimedes.ini which will appear in the SAML metadata. Specify suitable values using:
[saml-technical] company= name= surname= email=
[saml-administrative] company= name= surname= email=
SAML Attribute mapping files
Depending on the IdP software, it may be necessary to install some attribute mapping files to translate between the SAML short attribute names and global oid values. Create a directory and install some attribute maps. You will probably want to download these from the pysaml2 source code at https://github.com/rohe/pysaml2/tree/master/example/attributemaps Specify the directory in the configuration file using:
24 Chapter 1. Name Mataara Documentation, Release 0.1
[saml] attribute_map_dir=
1.12.3 Configuring Mataara in IdP
For Mataara to successfully talk SAML to the IdP, the SAML metadata for Mataara must be configured in the IdP. The local SAML metadata can be obtained by visiting: http://[hostname]/saml2/metadata
Upload this metadata file to your IdP. How to do this is beyond the scope of this document as it depends on the software involved.
1.13 Getting Started
Note: If you are familiar with the project or just want to have a go at standing up the default development environment check the quickstart guide.
The Mataara server is built on Django using Python with a ReactJS frontend.
1.13.1 Coding Style
As the server is built on Django, Mataara has adopted the Django coding style guidelines. To assist in identifying potential style problems, the Flake8 Python package is used. It is invoked automatically in the git pre-commit hook to prevent errors from being committed into the repo.
1.13.2 Repositories
There are different repositories for the server, and client. The server repository contains both the Django server and API, and the ReactJS frontend.
1.13.3 Network Requirements
In addition to the network requirements for production servers, development/build environments require outbound HTTP(S) to the Internet for the following: • https://forge.puppet.com - Puppet Forge • http://archive.ubuntu.com/ - Ubuntu update repositories or a local mirror • https://deb.nodesource.com - Nodejs Ubuntu repository • https://www.npmjs.com/ - NPM repository • https://pypi.python.org - the Python Package Index, as used by pip
1.13. Getting Started 25 Mataara Documentation, Release 0.1
1.14 Quick Start guide
This quick start guide is aimed at people that have stood up the development environment before or people that want to have a go at just diving in. Initial assumption: you’re using a Debian-based *nix. This should be copy/pastable into a terminal. Just start in the directory you want your development environment to start in.:
# Get the recommended tools sudo apt-get install virtualbox vagrant fabric python3-pip python3-setuptools python3-
˓→bandit # Get the repo git clone [email protected]:mataara/Mataara-Server.git mataara cd mataara # Install git pre-commit hook etc fab setup_dev_env # Put the default config in place cp conf/hiera/config.yaml.example conf/hiera/config.yaml # Get vagrant extension for hostnames vagrant plugin install vagrant-hostsupdater # Start the VM vagrant up # Set up the default advisory sources fab setup_feeds # Start the ReactJS server. fab runjs fab runjs will remain running but at this point you should be able to point your browser at http://mataara-server. local:8008 and login with vagrant and the password vagrant. If you edited conf/hiera/config.yaml above port and/or login details may have changed. If you’ve edited that file though, we assume that you know enough already to work out how to access and log in to your development environment. Similarly, the hostname can be edited in Vagrantfile.
1.14.1 References
• Get the recommended tools • Get the repo • Install git pre-commit hook etc • Put the default config in place • Start the VM • Set up the default advisory sources • Start the ReactJS server
1.15 Environment setup
26 Chapter 1. Name Mataara Documentation, Release 0.1
Note: There is a Quickstart Guide if you are wanting to dive in or are familiar with development and just want a handy reference to stand up your dev environment.
For ease of development. Mataara supports the use of Vagrant to manage a virtual machine containing the server. The instructions below also assume the use of an Ubuntu-based host system.
Note: TODO - Add non-vagrant manual development environment setup steps
1.15.1 Cloning from Git
Clone the Mataara Server project from git using: git clone [email protected]:mataara/Mataara-Server.git mataara
1.15.2 Shared folder into vagrant
If you follow the instructions below, use Linux on your host computer and Vagrant, the mataara folder containing your git clone which be accessible from within the vagrant virtual machine at the path /vagrant.
1.15.3 Install dependencies
VirtualBox installation
The vagrant file is expecting to use VirtualBox to provide the virtualisation layer. Other alternatives, such as LXC are possible. (The VirtualBox Oracle VM VirtualBox Extension Pack is not required.)
Note: TODO - Add alternatives to VirtualBox to the Vagrantfile
To install your distribution provided version of VirtualBox use: sudo apt-get install virtualbox
Information on installing the latest upstream Oracle managed version of VirtualBox for a variety of Linux systems can be read at https://www.virtualbox.org/wiki/Linux_Downloads
Other packages
On Ubuntu install the dependencies using: sudo apt-get install vagrant fabric python3-pip python3-setuptools
1.15.4 Configure the development environment
Install git pre-commit hook and any necessary local dependencies:
1.15. Environment setup 27 Mataara Documentation, Release 0.1
fab setup_dev_env
Note: Fabric (fab) commands are run from the host.
1.15.5 Configure the Mataara server
The Mataara server uses a configuration settings file which is managed by puppet inside vagrant. When using vagrant this file is conf/archimedes.ini.vagrant and is accessible outside of the vagrant image for customising. Puppet uses hiera to load the settings that it manages from conf/hiera/config.yaml. Copy the example tem- plate:
cp conf/hiera/config.yaml.example conf/hiera/config.yaml
Now configure your site specific settings by editing the appropriate entries. Full details on all the options can be found in on the Server Configuration Settings page. The development environment comes with a single preconfigured user (vagrant:vagrant). If you are wanting to use an existing system for users Mataara currently supports LDAP and SAML for external authentication.
1.15.6 Install Mataara
Once the Mataara config file that will be used by puppet has been configured, the virtual machine itself can be setup. This may take awhile, particularly if you need to download an operating system image. To build and provision the initial vagrant virtual machine:
vagrant up
If the initial fails, or some component times out, it is probably sufficient to just tell vagrant to reprovision the virtual machine using:
vagrant provision
Once Vagrant is running the Mataara server should be available on the default IP at http://192.168.33.5:8000
Post-install configuration
If you have an existing Mataara database, you might want to copy the data across to kick-start your development. See Database backup and restore. At a minimum you will want to configure the default security advisory sources.:
fab setup_feeds
You will also want to setup at least one client. This will depend on a preferred site(or sites) that you are monitoring. See Getting Started in the Client section for options. See also: TODO - cross-link to client documentation
28 Chapter 1. Name Mataara Documentation, Release 0.1
Setting up a superuser
To access the admin web interface, which can be useful in development, you’ll need to convert a standard user to a superuser. First login through the web interface as normal using whatever authentication source you have configured. Now grant yourself superuser rights: vagrant ssh-c"archimedes setsuperuser --username=
1.15.7 Stand up your development server
You should now able to run the server. fab run
Note: fab run will not exit. This starts the process in the VM and watches the webserver logs. You should now be able to log into the Mataara server at http://192.168.33.5:8000 (or the IP address you specified in the config.yml file). By default there is a single user called vagrant with the password vagrant. • To explore more fabric commands see Fabric tasks. • To explore the server configuration settings in more detail see Server Configuration Settings.
1.16 Fabric tasks
Once you have the development environment setup, you can use various fabric tasks that have been defined to make your life easier.
1.16.1 Index of tasks
• bandit - runs security checks • build_wheels - build a wheels for Mataara and its dependencies • flake8 - run linting checks on python code • jslint - running linting checks on the JS code. • make_docs - use Sphinx to compile up an updated set of HTML documentation • migrate - runs the Django database migrations • new_tag - create and push a new git tag • pip_update • reload_server - reloads server and runs migrations. • run_checks - short had or running all the linting, security checks and tests • runjs - runs npm to server the frontend React code for development • setup_dev_env - setup your local development environment • setup_feeds - configure the default security advisory feeds • show_logs - displays the output from the Mataara logs.
1.16. Fabric tasks 29 Mataara Documentation, Release 0.1
• test - run the built-in unit tests
1.16.2 Bandit Security Checks
Bandit is a security linter for Python source code. You can run it manually using: fab bandit
1.16.3 Build Mataara Wheels
The server installation instructions are designed around the use of wheels for installation of Mataara. Sometimes it may be possible to install all necessary wheels using pip from a standard Python repository. To install a custom version of Mataara from source, this command can be used to build the Mataara wheel and those for all its dependencies (assuming that the target installation server is the same as the build server). The built wheels are placed in the directory wheels/. To build the wheels run: fab build_wheels
1.16.4 Creating new git tags
We use tags natively in git to manage deployments. They are created using the format:
%Y%m%d-%N
Tags will be created for the current date followed by -1 if it’s the first tag of the day. For subsequent tags the number will be incremented. There is an assumption that a maximum of 9 tags will be needed in any one day. To create the tag and upload it to your git server, run: fab new_tag
1.16.5 Flake8 Linting Checks
Flake8 is a generic linter for Python source code. It is a wrapper around other tools such as PyFlakes. The configuration file for flake8 is setup.cfg. Source code lines that need to be skipped because they are producing false positives need the following comment to be appended:
# noqa
1.16.6 Running the Server
This command is being deprecated as bringing up the development environment starts a backend server through su- pervisor. There is a short-cut for running the server using the integrated Django web service. (Note this interface is only suitable for development environments. In production it is important to use a full web server)
30 Chapter 1. Name Mataara Documentation, Release 0.1
To run the server: fab run
This will default to running the web site using the URL: http://192.168.33.5:8000/ The IP address can be configured in the Vagrantfile if you want to choose something else.
1.16.7 Setting up Initial Security Advisory Feeds
To initialise the default security advisory feeds run: fab setup_feeds
1.16.8 Setting up the Development Environment
Setup your local development environment - installs a git pre-commit hook to run check, and install locally on the host the required python packages: fab setup_dev_env
1.16.9 Running the tests
To run all both the Django Python server tests and the front end React JS tests: fab test The following options can be added: * fast - skip vagrant environment system checks * server_tests=” - to skip the server tests * frontend_tests=” - to skip the frontend tests
1.16.10 Updating Pip Dependencies
All of the Python packages installed by Pip and used on a production server have pinned versions. (We don’t bother to pin the versions of packages that are specific to development or the build process) By pinning packages, we can have confidence in the specific versions of all components on a server and we can control when we do the upgrades. Using this approach does require doing regular upgrades. This task takes the core requirements defined in requirements/requirements.in and updates requirements/common.txt to the latest pinned versions. To run the task: fab pip_update
1.17 Useful Commands
1.17.1 Standard Django Commands
There are many built-in Django commands that may be useful in a development environment. In typical documentation on the web you may see references to invoking these such as:
1.17. Useful Commands 31 Mataara Documentation, Release 0.1
python manage.py shell_plus
Inside our vagrant setup, these can be more easily invoked using the following without worrying about paths or where files are: archimedes shell_plus
A full list of standard commands, those added by packages, and by Mataara can be seen by just running: archimedes help
Detailed help on any individual command is available by running: archimedes help
Table 1: Common useful commands in development environments Command Source Description makemigrations django Creates any new migrations required migrate django Update database schema by applying migrations squashmigrations django Squash a set of migrations into a single one shell_plus django_extensions Python interactive shell with Django models loaded test django_nose Run our defined system tests collectstatic staticfiles Collect static files in a single location runserver staticfiles Start a lightweight development web server check django Check the project for potential problems celery djcelery Manage scheduled jobs getmail django_mailbox Retrieve email from any configured mailbox
Useful Celery Commands
Initiate a manual trigger of scheduled celery tasks: archimedes celery beat
View details of current celery workers: archimedes celery worker
1.17.2 Custom Mataara Commands
Table 2: Additional Mataara specific commands Command Source Description addadvisorysource core Add new advisory sources importadvisories drupal Import advisories from configured sources parseadvisories drupal Parse the advisories that have been imported setsuperuser core Assigned superuser rights to the specified user
TODO - The advisory commands are likely to move from a drupal category to a generic one
32 Chapter 1. Name Mataara Documentation, Release 0.1
1.18 Database Backup and Restore
During development it is often useful to have a database that contains real information. If you have an existing server with data it can easily be transferred to your development environment.
1.18.1 Create Database Dump
We’ll be using pg_dump to create the database dump into a plain-text SQL script file format. However, as this may be quite large we will also be compressing it. We will also create the backup such that when it is restored it will clean out any existing content and refresh the database tables.
Database backup in vagrant
Login to vagrant and create the backup in the vagrant directory that is shared with the host computer:
vagrant ssh pg_dump--clean--compress=9>/vagrant/mataara.sql.gz
Database backup on a server
Login to your server as the user that is running Mataara. Create the file and then transfer it off the server:
pg_dump--clean--compress=9> mataara.sql.gz
1.18.2 Restoring a Database Dump in Vagrant
Obtain your database backup and place it into the Mataara source code folder that is shared into the vagrant virtual machine. Before restoring the database it is important to shutdown any applications using it. To stop the automatic background processes run:
vagrant ssh supervisorctl stop all
Restore the database dump:
zcat/vagrant/
If there are errors during the import process it may be that the database backup was not created with the --clean parameter. In which case delete and recreate the database using:
dropdb vagrant createdb vagrant
Apply any missing migrations to the database models in Django so that it matching the restored database schema by running:
archimedes migrate
Restart the background tasks in Vagrant using:
1.18. Database Backup and Restore 33 Mataara Documentation, Release 0.1
supervisorctl start all
1.19 Setting up your development environment for ReactJS
The ReactJS environment can be installed and started with the following fabric task: # On the host fab runjs This may take a while depending on the speed of your machine and Internet connection and when completed it will keep the terminal session and automatically recompile as you make edits/updates. Installed packages will be in archimedes/frontend/node_modules. We’re using Webpack to bundle up the JavaScript files.
1.20 Report Processing
Reports are processed by calling
1.21 Mataara report format (JSON)
This file documents the format used for reports in the modern JSON format, with the intent of moving to a standardised, versioned API in future. See also: There is also a legacy XML-based report format that is only supported for Drupal 7.
1.21.1 Encryption
Encryption uses OpenSSL with an RC4 stream cipher. The HTTP endpoint receives a JSON blob with two fields containing BASE64-encoded values: ek Envelope key for the public key specified enc The encrypted data payload
1.21.2 JSON Format
All timestamps are expected to be in UNIX time, i.e. the number of seconds since 1 January 1970 UTC. AdminRole{} Details of the Drupal website administration role. RoleName Name of role (e.g. administrator). RoleID Drupal rid (role identifier) RoleUsers[] Set of users with this role. uid{} User’s unique identifier
34 Chapter 1. Name Mataara Documentation, Release 0.1
Name User’s Drupal site username. Mail User’s email address, if defined. Status Is the account marked as active inside Drupal? (0 = blocked, 1 = active). LastLogin Timestamp of the user’s last login to the Drupal site. Databases{} Set of databases that the Drupal site is connected to. key{} Database key (usually default). target{} Database target (usually default). Driver Database driver (e.g. mysql, pgsql). Database Name of the database to connect to. Host Hostname of the database server. DirectoryHash MD5 hash of the DRUPAL_ROOT directory and files inside to allow for detection of code changes. DrupalVersion Version number for the Drupal core, e.g. 7.34 Environment{} Drupal environment in use. Label user-friendly display name for the environment. Name Internal machine name for the environment. Modules[{}] List of installed Drupal modules. Module Internal machine name of module. Name User-friendly display name of module. Description Typically one-line description of module behaviour. Package Package name (used to group projects, e.g. Core for Drupal Core related modules). Project Machine name of project (used to group modules). Version Module version string (e.g. 7.x-2.3, 7.x-dev, 7.x-1.0-beta3, 7.x-1. 0-alpha4+1-dev). Url External URL for module, if provided. Nodes{} Summary of the Drupal content nodes in the site. Nodes Number of nodes in the site. Revisions Number of node revisions in the site. ReportExpiry Timestamp at which point the report should be considered stale and ignored if received after this time. ReportGenerated Timestamp when the report was generated. ReportType Type of report (e.g. drupal7). ServerHostName Server’s hostname. SiteKey Unique identifier for site. SiteSlogan An optional site description that may be converted in Drupal. SiteTitle Drupal site title. Themes[{}] List of installed Drupal front-end themes. Theme properties are the same as for Modules.
1.21. Mataara report format (JSON) 35 Mataara Documentation, Release 0.1
Users Number of users on site.
Example report
{ "Databases":{ "default":{ "default":{ "Host":"127.0.0.1", "Database":"vagrant", "Driver":"mysql" } } }, "ServerHostname":"archclient", "DrupalVersion":"7.51", "Environment":{ "Name":"production", "Label":"Production" }, "ReportGenerated": 1478730113, "ServerName":"http:\/\/archclient.vagrant-local\/", "ReportExpiry": 1478731913, "SiteRoot":"\/var\/www\/archclient", "SiteSlogan": false, "Nodes":{ "Revisions":"0", "Nodes":"0" }, "DirectoryHash":"3c4926153b2a5ffbceec2a464033de30", "Themes":[ { "Version":"7.51", "Theme":"bartik", "Url":"", "Description":"A flexible, recolorable theme with many regions.", "Name":"Bartik", "Project":"" } ], "AdminRole":{ "RoleUsers":[
], "RoleID":0, "RoleName":"" }, "Modules":[ { "Package":"", "Description":"Sets the environment variable for a site instance", "Url":"", "Version":"7.x-1.0", "Name":"Environment", "Module":"environment", "Project":"environment" }, (continues on next page)
36 Chapter 1. Name Mataara Documentation, Release 0.1
(continued from previous page) { "Package":"Core", "Description":"Controls the visual building blocks a page is constructed with.
˓→Blocks are boxes of content rendered into an area, or region, of a web page.", "Url":"", "Version":"7.51-dev", "Name":"Block", "Module":"block", "Project":"" }, { "Package":"Archimedes", "Description":"Client module for reporting to a Mataara Server", "Url":"", "Version":"7.x-dev", "Name":"Mataara Client", "Module":"archimedes_client", "Project":"" }, { "Package":"Core", "Description":"Logs and records system events to the database.", "Url":"", "Version":"7.51-dev", "Name":"Database logging", "Module":"dblog", "Project":"" }, { "Package":"Core", "Description":"Field API to add fields to entities like nodes and users.", "Url":"", "Version":"7.51-dev", "Name":"Field", "Module":"field", "Project":"" }, { "Package":"Core", "Description":"Stores field data in an SQL database.", "Url":"", "Version":"7.51-dev", "Name":"Field SQL storage", "Module":"field_sql_storage", "Project":"" }, { "Package":"Core", "Description":"Filters content in preparation for display.", "Url":"", "Version":"7.51-dev", "Name":"Filter", "Module":"filter", "Project":"" }, { "Package":"Core", (continues on next page)
1.21. Mataara report format (JSON) 37 Mataara Documentation, Release 0.1
(continued from previous page) "Description":"Allows content to be submitted to the site and displayed on
˓→pages.", "Url":"", "Version":"7.51-dev", "Name":"Node", "Module":"node", "Project":"" }, { "Package":"Core", "Description":"Handles general site configuration for administrators.", "Url":"", "Version":"7.51-dev", "Name":"System", "Module":"system", "Project":"" }, { "Package":"Core", "Description":"Defines simple text field types.", "Url":"", "Version":"7.51-dev", "Name":"Text", "Module":"text", "Project":"" }, { "Package":"Core", "Description":"Checks for available updates, and can securely install or
˓→update modules and themes via a web interface.", "Url":"", "Version":"7.51-dev", "Name":"Update manager", "Module":"update", "Project":"" }, { "Package":"Core", "Description":"Manages the user registration and login system.", "Url":"", "Version":"7.51-dev", "Name":"User", "Module":"user", "Project":"" }, { "Package":"", "Description":"Start with only a few modules enabled.", "Url":"", "Version":"7.51-dev", "Name":"Minimal", "Module":"minimal", "Project":"" } ], "SiteTitle":"Archimedes Client", "SiteKey":"d2e1e62ce90c3b4f2a6602209eb6ac29", (continues on next page)
38 Chapter 1. Name Mataara Documentation, Release 0.1
(continued from previous page) "Users":"2", "ReportType":"drupal7" }
1.22 Archimedes report format (legacy XML)
This file documents the legacy report format based on XML. This format is only supported by the Drupal 7 client. New installations should be using the new JSON based report format.
1.22.1 Transport Mechanism
Encryption uses OpenSSL with an RC4 stream cipher. The HTTP endpoint receives a JSON blob with two fields containing base64-encoded values: key Envelope key for the public key specified data The encrypted XML data payload The HTTP User-Agent is: Archimedes Client The reports.utils.legacy_decode() function fills in a few fields that the old report format is missing, or creates mappings to the same fields as the newer Drupal 7 reports, with a ReportType of drupalLegacy.
1.22.2 XML Format
The XML report format starts with a standard XML definition:
Everything is contained within a node tag. Node attributes are: xmlns monitor:node type The only supported type for this legacy format is drupal id The Drupal site’s SiteKey value datetime Report generation timestamp in ISO format, e.g. 2016-11-29T10:32:09+13:00 author Mailto link to the registered Drupal site email, e.g. mailto:[email protected]. com For example:
All fields inside node are defined in the form
1.22. Archimedes report format (legacy XML) 39 Mataara Documentation, Release 0.1
field_common_hash Site key. field_body Site slogan / description. field_servername Server’s hostname. field_webroot Drupal site root directory. field_dbname Database name (assumes only one?). field_dbhost XML definition : xmlns:node="monitor-plugin:node", node:title="$host", node:type="host" facet Contains the database hostname. Should match the title specified in the definition for the database host. field_users[] List of admin users. XML definition : xmlns:user="monitor-plugin:user", user:type="mail" value Specify user’s email address using the format mailto:$email field_c_dataset Node and user counts. XML definition for the following fields: xmlns:dataset="monitor-plugin:dataset", dataset:title="$title" Nodes Number of content nodes. Revisions Number of content node revisions. Users Number of users. field_drupal_mod[] List of Drupal modules. There are lots of attributes for modules. xmlns:node node="monitor-plugin:drupal-module" node:title Display name. node:body Description. node:field_name Module machine name. node:field_dru_pkg Package name. node:field_dru_proj Project name. node:field_mod_version Module version. node:field_mod_url Module URL, if supplied. node:type drupal_module facet Title of modules field_drupal_theme[] List of Drupal themes (identical format to _mod) field_site_env Drupal environment name. field_directory_hash Directory hash
Example report
40 Chapter 1. Name Mataara Documentation, Release 0.1
(continued from previous page)
˓→
˓→
˓→
˓→ ˓→value> ˓→"Allows administrators to customize the site navigation menu." node:field_name="menu ˓→" node:field_dru_pkg="Core" node:field_dru_proj="drupal" node:field_mod_version="7. ˓→52" node:field_mod_url="" node:type="drupal_module"> ˓→"Allows content to be submitted to the site and displayed on pages." node:field_ ˓→name="node" node:field_dru_pkg="Core" node:field_dru_proj="drupal" node:field_mod_ ˓→version="7.52" node:field_mod_url="" node:type="drupal_module"> ˓→value> ˓→"Core Panels display functions; provides no external UI, at least one other Panels ˓→module should be enabled." node:field_name="panels" node:field_dru_pkg="Panels" ˓→node:field_dru_proj="panels" node:field_mod_version="7.x-3.7" node:field_mod_url="" ˓→node:type="drupal_module"> ˓→"Administrative interface to views. Without this module, you cannot create or edit ˓→your views." node:field_name="views_ui" node:field_dru_pkg="Views" node:field_dru_ ˓→proj="views" node:field_mod_version="7.x-3.14" node:field_mod_url="" node:type= ˓→"drupal_module">
1.22. Archimedes report format (legacy XML) 41 Mataara Documentation, Release 0.1
(continued from previous page) ˓→mod_version="7.52" node:field_dru_proj="drupal" node:field_mod_url="" node:type= ˓→"drupal_theme"> ˓→"seven" node:field_mod_version="7.52" node:field_dru_proj="drupal" node:field_mod_ ˓→url="" node:type="drupal_theme">
1.23 Key Development Technologies
We’re using the following technologies: bandit Security static code analyser for Python code celery Asyncronous task queue/job queue used internally within Mataara and for scheduled jobs fabric Provides developers with shortcuts to standard tasks, such as running the server flake8 Perform Python checks for linting errors Gunicorn ‘Green Unicorn’ A Python WSGI HTTP server. Designed to be lightweight and speedy. pip Perform Python package management puppet Used by Vagrant to configure Mataara inside the virtual machine redis In-memory data store used by celery Sphinx Used to generate this documentation supervisor Used to control system services outside of the main Django application, such as celery Vagrant Used for managing and initialising the development environment inside VirtualBox VirtualBox Provides virtualisation of a development environment containing the server wheel Used for packing Python modules and applications for easy installation
1.24 Source code repository structure
1.24.1 Directories in version control archimedes Django application source code bin Scripts used in the vagrant virtual machine conf Sample and live configs used in vagrant manifests Puppet manifest used in the vagrant virtual machine
42 Chapter 1. Name Mataara Documentation, Release 0.1 puppet Puppet template used in the vagrant virtual machine requirements Python dependencies required in different environments
1.24.2 Directories not in version control
.vagrant Internal settings managed by Vagrant archimedes.egg-info Created if the wheelhouse has been built locally build Temporary directory that may be used as part of a build process dist Temporary directory that may be used as part of a build process reportkey Contains the encryption key configured in the virtual machine for receiving reports. venv Contains the ‘venv’ that Mataara is installed into when running in Vagrant.
1.24.3 Key files
.gitignore Defines what git should ignore and not check into the repository .gitlab-ci.yml Definitions file for Gitlab CI CHANGELOG.md Project changelog file fabfile.py Fabric tasks used by developers hiera_config.yaml Tells Hiera where to pull our config.yaml from. LICENSE Project open source license MANIFEST.in Used as part of the wheelbuilding process README.md Project README requirements.txt Shortcut to requirements/common.txt setup.cfg Configuration file for flake8 setup.py Python management file for Mataara Vagrantfile Configuration file for Vagrant archimedes/RELEASE Internal version and release number
1.24. Source code repository structure 43