IRMA Documentation Release 2.0.4

Quarkslab

Jan 24, 2018

Contents

1 Introduction 3 1.1 Purpose...... 3 1.2 File Analysis Process...... 3 1.3 Infrastructure Overview...... 4 1.4 Hardware requirements...... 5 1.5 Supported Analyzers...... 6

2 Automated Install 9 2.1 Requirements...... 9 2.2 Ansible scripts...... 9 2.3 Predeﬁned Environments...... 9 2.4 Using Debian repos...... 12

3 Manual Installation 13 3.1 Brain...... 13 3.2 Frontend...... 22 3.3 Probe...... 31

4 Database migration 41 4.1 Requirements...... 41 4.2 Content...... 41 4.3 Usage...... 42 4.4 Tips and tricks...... 44

5 To evolve IRMA 47 5.1 Adding a new probe...... 47

6 References 51 6.1 Disclaimer...... 51 6.2 License...... 51 6.3 Apache License, version 2.0...... 51 6.4 Authors...... 51

7 Frequently Asked Questions 53 7.1 Playing with tags...... 53 7.2 SSL settings...... 56 7.3 How to debug...... 57

i 7.4 How to migrate...... 60 7.5 API documentation...... 61 7.6 Connect to a vagrant box through ssh...... 63 7.7 Enable SSL using OpenSSL in ansible scripts...... 63 7.8 Speed up your Vagrant VMs...... 64

8 Resources 65

9 Screenshots 67 9.1 Command Line Interface...... 67 9.2 Web Interface...... 68

ii IRMA Documentation, Release 2.0.4

IRMA is an asynchronous & customizable analysis system for suspicious ﬁles. This guide will explain how to set up IRMA, use it and customize it at will.

Contents 1 IRMA Documentation, Release 2.0.4

2 Contents CHAPTER 1

Introduction

This is an introductory chapter to IRMA. It recalls IRMA’s overall architecture, hardware required to run it and the recommended order for installing the IRMA’s components.

1.1 Purpose

IRMA intends to be an open-source platform designed to help identifying and analyzing malicious files. However, today’s defense is not only about learning about a file, but it is also getting a fine overview of the incident you dealt with: where / when a malicious file has been seen, who submitted a hash, where a hash has been noticed, which anti-virus detects it, . . . An important value with IRMA comes from you keep control over where goes and who gets your data. Once you install IRMA on your network, your data stays on your network. Each submitted files is analyzed in various ways. For now, we focus our efforts on multiple anti-virus engines, but we are working on other “probes” (feel free to submit your own).

1.2 File Analysis Process

1. An analysis begins when a user uploads files to the Frontend. 2. Frontend checks for existing files and results in SQL. If needed, it stores the new files and calls asynchronously scan jobs on Brain. 3. Brain worker sends as much subtasks to Probe(s) as needed. 4. Probe workers process their jobs and send back results to Brain. 5. Brain sends results to Frontend.

3 IRMA Documentation, Release 2.0.4

1.3 Infrastructure Overview

A drawing is better than a lot of explanations (sometimes ;)

4 Chapter 1. Introduction IRMA Documentation, Release 2.0.4

1.4 Hardware requirements

IRMA platform is divided in three major components: the Frontend, the Brain and one or multiple Probes. These three components can be installed on a unique host or on multiple hosts, according to the kind of probes you are using. The Frontend and the Brain must be installed on a GNU/Linux system1. We recommend to use a Debian Stable distribution which is supported and known to work. According to the kind of probes and their dependencies, each analyzers can be installed on a separate hosts or share

1 Theorically, it should be possible, with some efforts, to make IRMA work on Microsoft Windows systems as most of the components used for the platform are known to work or to have equivalents on these systems.

1.4. Hardware requirements 5 IRMA Documentation, Release 2.0.4

the same host as far as they do not interfere with each other2. So forth, only Debian Stable and Microsoft Windows 8 and 10 hosts have been tested. We can not give you any speciﬁc numbers. On one hand we managed to run the whole IRMA platform on a single machine by hosting it with multiple systems inside virtual machines: this setup gives fairly high throughput as long as it has reasonable IO (ideally, SSDs), and a good amount of memory (our setup was an i7 cpu with 16 GB ram on regular drives (at least 200 GB required), on the other hand, a lighter version of the system with the three parts together3 was successfully installed on a single virtual machine (1 GB of Ram and 4 virtual processors). For a large company, in theory, given a single high-memory machine, with 16+ cores, and SSDs, you could run IRMA platform and bear the workload load with reasonable response time.

1.5 Supported Analyzers

We have mainly focused our efforts on multiple anti-virus engines but we are working on other kind of “probes”. We enumerate the analyzers that are bundled with IRMA probe application. Feel free to submit your own probes.

1.5.1 Antiviruses

So far, we have instrumented the following antiviruses from their CLI:

Probe Name Anti-Virus Name Platform ASquaredCmd Emsisoft Command Line Microsoft Windows CLI Avira Avira Microsoft Windows CLI AvastCoreSecurity Avast GNU/Linux CLI AVGAntiVirusFree AVG GNU/Linux CLI BitdefenderForUnices Bitdefender GNU/Linux CLI ClamAV ClamAV GNU/Linux CLI ComodoCAVL Comodo Antivirus for Linux GNU/Linux CLI DrWeb Dr.Web GNU/Linux CLI EsetNod32 Eset Nod32 Business Edition GNU/Linux CLI EScan eScan GNU/Linux CLI FProt F-Prot GNU/Linux CLI FSecure F-Secure GNU/Linux CLI GData G Data Antivirus Microsoft Windows CLI Kaspersky Kaspersky Internet Security Microsoft Windows CLI McAfeeVSCL McAfee VirusScan Command Line GNU/Linux - Microsoft Windows CLI Sophos Sophos GNU/Linux - Microsoft Windows CLI Symantec Symantec Endpoint Protection Microsoft Windows CLI VirusBlokAda VirusBlokAda GNU/Linux CLI Zoner Zoner Antivirus GNU/Linux CLI

1.5.2 External analysis platforms

So far, we query the following external analysis platforms:

2 For instance, we managed to host several GNU/Linux anti-viruses on an unique probe by preventing it to launch daemons at startup. This is difﬁcult for Microsoft systems on which it is not recommended to install multiple anti-viruses on a single host. 3 with a limited set of probes

6 Chapter 1. Introduction IRMA Documentation, Release 2.0.4

Probe Name Analysis Platform Description VirusTotal VirusTotal Report is searched using the sha256 of the ﬁle which is not sent

1.5.3 File database

So far, we query the following ﬁle databases:

Probe Database Description Name NSRL National Software Reference Li- collection of digital signatures of known, traceable software ap- brary plications

1.5.4 Metadata

So far, we implemented the following analyzers:

Probe Name Description StaticAnalyzer PE File analyzer adapted from Cuckoo Sandbox PEiD PE File packer analyzer Yara Checks if a ﬁle match yara rules

1.5. Supported Analyzers 7 IRMA Documentation, Release 2.0.4

8 Chapter 1. Introduction CHAPTER 2

Automated Install

IRMA platform can be easily installed with a set of ansible roles and playbooks. It will help you to build, install or maintain different setups.

2.1 Requirements

• Ansible 2.2.1.0;

2.2 Ansible scripts

Get IRMA ansible scripts on github:

$ git clone https://github.com/quarkslab/irma $ cd irma/ansible

2.3 Predeﬁned Environments

There are 2 different IRMA setups available. Dev/Testing will be installed in one or multiple virtual machines while production could be used to install IRMA on physical machines or virtual machines already setup:

2.3.1 Development environment

This environment has been designed to help you to modify IRMA’s components and redeploy and test them. In this setup, everything is installed in a single virtual machine with sources rsync-ed between the host and the guest.

9 IRMA Documentation, Release 2.0.4

Requirements

• Vagrant 1.8 or higher has to be installed • As the installation work only for Virtualbox, you will need to install it • Rsync to synchronize directories from host to VMs • Read the Ansible introduction

Run Vagrant and create your VMs

To initialize and provision the Virtualbox VM.

$ cd /ansible $ VM_ENV=your_environment_name vagrant up

The template will be downloaded automatically and conﬁgured using environments/dev.yml ﬁle.

Note: Optionally, if you want to use your own environment, create it in environments directory and run:

$ VM_ENV=your_environment_name vagrant up

Conﬁgure your .ini ﬁles

Note: You can bypass this step, as this provisioning is sync with default username and password used in (frontend|brain|probe) conﬁg ﬁles.

As your config/*.ini ﬁle are transferred from host to VMs, you will need to modify them individually in the frontend, probe and brain directories to match the user and password deﬁned in playbooks/group_vars/ *.

Modify your host and open IRMA frontend

Then, for proper use, update your /etc/hosts ﬁle and add:

172.16.1.30 www.frontend.irma

Then, with your web browser, IRMA allinone is available at www.frontend.irma.

Sync ﬁles between host and guest

Once rsync is installed inside your virtual machine and your environment is correctly set. You could easily sync your code with:

$ vagrant rsync # or vagrant rsync-auto to automatically initiates an rsync # transfer when changes are detected

Then, reload the modiﬁed application.

10 Chapter 2. Automated Install IRMA Documentation, Release 2.0.4

2.3.2 Production environment

This environment is used to install IRMA on production-ready Debian servers.

Requirements

• One or multiple 64-bit Debian 7 servers.

Preparing servers

Create an account that is going to be used to provision IRMA on the server via Ansible, or use one which has already been created. To speed up provisioning, you can: • Authorize your SSH key for password-less authentication (optional):

# On your local machine $ ssh-copy-id user@hostname # -i if you want to select your identity file

• If you do not want to have to type your password for sudo command, consider adding your user to sudoers, using visudo command (optional):

user ALL=(ALL) NOPASSWD: ALL

Conﬁgure for your installation

Modify settings in playbooks/group_vars/all especially the default_ssh_keys: section. You will need to add your public keys for SSH password-less connection to the default irma server user. Configuration file used by the brain, the frontend and the probes applications are generated with default values that are specified in playbooks/group_vars/brain, playbooks/group_vars/frontend and playbooks/ group_vars/probe respectively. Make sure to adapt xxx_deployment_configs variables accordingly to your installation. It is recommended to change all the default passwords defined in group_vars/* configuration files (password variables for most of them). Finally, you will need to customize the hosts/example file and adapt it to describe your own server infrastructure. There is three sections, one for each server role (frontend, brain, probe). Please refer to Ansible Inventory documentation for the expected syntax.

Run the Ansible Playbook

To run the main playbook with the hosts/example ﬁle you have deﬁned, use the following command. Ansible will ask you the sudo password (-K option).

$ cd /ansible $ ansible-playbook -i ./hosts/example playbooks/playbook.yml -u -

˓→K

To run one or more speciﬁc actions and avoid running all the playbook, you can use tags. For example, if you want to re-provision Nginx, run the same command, but append --tags=nginx. You can combine multiple tags separated with commas.

2.3. Predeﬁned Environments 11 IRMA Documentation, Release 2.0.4

Deploy a new version of IRMA

Assuming that you have already provisioned and deployed a version of IRMA, which you want to upgrade, you will need to run the deployment script:

$ ansible-playbook -i ./hosts/example ./playbooks/deployment.yml -u irma

Make sure to replace irma with the default user if you have changed it in the group_vars/all ﬁle.

Access your IRMA installation

Once the provisioning and the deployment have ﬁnished, you should be able to perform scans and get their results via the frontend hostname you speciﬁed.

2.4 Using Debian repos

If you planned to use only Debian ofﬁcial repository, you’ll need to change in playbooks/group_vars/all: default_use_debian_repo: yes # no is the default value

12 Chapter 2. Automated Install CHAPTER 3

Manual Installation

Warning: Manual installation needs an update since 1.5.3

as all repositories layout have changed. Try Automated install or ask for help on IRC. In some specific cases where automatic install is not possible, you should go with manual installation. Each major component of the IRMA platform comes with their own python-based application. As the Brain is the nerve center of the whole platform, it is recommended to install it first before installing other components. One can then install either the Frontend or the Probes he wants. Please refer to specific documentation for the manual installation process for each component.

3.1 Brain

The Brain is a python-based application that only dispatches analysis requests from different frontends1 to the available Probes. Analyses are scheduled by the Brain on Probes through Celery, an open source task framework for Python.

3.1.1 Architecture

Let us recall first the inner architecture of the Brain. It uses multiple technologies with a specific purpose each: • a Celery worker that handles scan requests from Frontends and results returned by the Probes. • a RabbitMQ server used by Celery as a backend and as a broker for task queues and job queues used to schedule tasks for Probes (for scan jobs) and the Frontend (for scan results). • an SFTP server where files to be scanned are uploaded by Frontends and downloaded by Probes,

1 This feature is not ready yet, we are currently working on its implementation.

13 IRMA Documentation, Release 2.0.4

3.1.2 Installing and Conﬁguring RabbitMQ

The following explains how to install and to setup RabbitMQ for your setup.

Install RabbitMQ

RabbitMQ server can be installed with this command on Debian:

$ sudo apt-get install rabbitmq-server

Conﬁguring RabbitMQ

RabbitMQ serves all components of IRMA platform. Each component has their own virtual host (i.e., message queue where to get data), a username and a password. To conﬁgure RabbitMQ for IRMA platform, you have to create users, vhosts and add permissions for each user to corresponding vhost. To easily create virtual hosts and users, we provide scripts in extras/ directory:

$ sh ./extras/scripts/rabbitmq/rmq_adduser.sh Usage: sudo rmq_adduser.sh

For instance, to create 3 users with the following parameters, one can do:

User- Password Virtual Command name Host brain brain-rmq- mqbrain sudo ./extras/scripts/rabbitmq/rmq_adduser.sh brain password brain-rmq-password mqbrain probe probe-rmq- mqprobe sudo ./extras/scripts/rabbitmq/rmq_adduser.sh probe password probe-rmq-password mqprobe fron- frontend- mqfron- sudo ./extras/scripts/rabbitmq/rmq_adduser.sh tend rmq- tend frontend frontend-rmq-password mqfrontend password

The script simply execute the following three commands:

$ sudo rabbitmqctl add_user $ sudo rabbitmqctl add_vhost $ sudo rabbitmqctl set_permissions -p ". *"". *"". *"

Warning: Important Make sure to note down the username, the password and the virtual host you just defined. You will be asked to retype them on each application configuration file (brain, frontend and probe)

Note: Disclaimer Please ensure that only trusted sources can communicate with your RabbitMQ server, by setting up ﬁrewall rules for instance, as your RabbitMQ may be exposed to Internet.

14 Chapter 3. Manual Installation IRMA Documentation, Release 2.0.4

Verifying RabbitMQ conﬁguration

We can verify that the RabbitMQ server has taken into account our modiﬁcations with some commands:

Checking for vhosts

$ sudo rabbitmqctl list_vhosts Listing vhosts ... mqbrain / mqfrontend mqprobe mqadmin ...done.

If the deﬁned virtual host are not listed by the above command, please execute once more the script.

Checking for users

$ sudo rabbitmqctl list_users Listing users ... probe[] brain[] frontend[] ...done.

If the deﬁned users are not listed by the above command, please execute once more the script.

Changing password

If you do not remember the password you just typed, you can change it with rabbitmqctl command:

$ sudo rabbitmqctl change_password brain brain-rmq-password Changing password for user "brain" ...... done.

Restarting the service

You may want to restart the service. Thus, the following command can be done:

$ sudo invoke-rc.d rabbitmq-server restart

3.1.3 Installing and conﬁguring sftp

A SFTP server is used to transfer ﬁles Frontends and meant to be analyzed by Probes. We describe in the following how to set it up.

3.1. Brain 15 IRMA Documentation, Release 2.0.4

Installing sshd sshd should already been installed but if not install it with:

$ sudo apt-get install openssh-sftp-server

Creating FTP speciﬁc users and groups

$ sudo groupadd sftpusers $ sudo useradd -g sftpusers -M -s /etc $ sudo useradd -g sftpusers -Match -s /etc $ sudo passwd frontend $ sudo passwd probe

Conﬁgure sshd

Edit /etc/sshd_config

Subsystem sftp internal-sftp Match User AllowTcpForwarding no ChrootDirectory /sftp/ ForceCommand internal-sftp -u2 PermitTunnel no X11Forwarding no Match User AllowTcpForwarding no ChrootDirectory /sftp ForceCommand internal-sftp -u2 PermitTunnel no X11Forwarding no

Create SFTP directories

The frontends need an account with /sftp//uploads as home directory and a single account is shared between probes ( uploads directory is needed as in chrooted environment home is not writeable). The later needs to access to all frontends, thus the associated home directory is simply /sftp/. For instance, a frontend named frontend, execute the following to create the directories:

$ sudo mkdir -pv /sftp/frontend/uploads mkdir: created directory `/sftp' mkdir: created directory `/sftp/frontend' mkdir: created directory `/sftp/frontend/uploads' $ sudo chown -R root:sftpusers /sftp $ sudo chmod 0750 /sftp $ sudo chmod 0750 /sftp/frontend $ sudo chown -R frontend:sftpusers /sftp/frontend/uploads $ sudo chmod -R 0775 /sftp/frontend/uploads

16 Chapter 3. Manual Installation IRMA Documentation, Release 2.0.4

Restart the service

You may want to restart the service:

$ sudo invoke-rc.d sshd restart

3.1.4 Installation

The Brain must be installed on a GNU/Linux distribution. With some efforts, it should be possible to run it on a Microsoft Windows system, but this has not been tested yet. This section describes how to get the source code of the application for the Brain and to install it.

Pre-requisites

We assume that you have a command line interface on your system with the following tools installed: • python 3.4.x and newer • python3-pip (see Install pip for the recommended way to install pip package manager) Furthermore, we assume that you have created an user irma that will be used to run the python application.

Note: On GNU/Linux, one can create the user (and the group) irma with:

$ sudo adduser --system --no-create-home --group irma

Installation on GNU/Linux

On GNU/Linux system, we will assume that the code for the Brain will be installed in /opt/irma/irma-brain/ current directory, which is the conﬁgured default installation directory. The source code is hosted on github.com. One can fetch an up-to-date version with the following commands. Let us note that there is a common submodule named common that is a soft-link (lib), do not forget the dereference option of cp (-L):

# If src repository not already cloned $ git clone https://github.com/quarkslab/irma /opt/irma/src $ cp -rL /opt/irma/src/brain /opt/irma/irma-brain/current

We need few dependencies for future steps: $ sudo apt-get install python3-virtualenv python3-dev then, we need to install python dependencies manually in a virtualenv :

$ virtualenv --system-site-packages /opt/irma/irma-brain/current/venv $ /opt/irma/irma-brain/current/venv/bin/pip install -r /opt/irma/irma-brain/current/

˓→requirements.txt [...]

If everything has gone well, you should have installed the python application on your system. The next step is to install the other components it relies on and ﬁnally to conﬁgure it for your platform.

3.1. Brain 17 IRMA Documentation, Release 2.0.4

3.1.5 Conﬁguration

The configuration file is located at config/brain.ini in the installation directory. Update it with your specific info.

Note: Detailed meaning of each ﬁeld in config/brain.ini:

Section Key Type Default Description log syslog integer 0 enable rsyslog (experimental) prefix string irma-brain: prefix to append to rsyslog entries debug boolean False | enable Debug log sql_debug boolean False | enable SQL debug log celery_options concurrency integer 0 number of concurrent workers (0 means nb of cores) soft_time_limit integer 300 (sec) time limit before task soft interrupt time_limit integer 1500 (sec) time limit before task is killed broker_brain host string hostname for the RabbitMQ server port integer 5672 port for the RabbitMQ server vhost string virtual host configured for brain username string username used for brain on the Rab- bitMQ server password string password used for brain on the Rab- bitMQ server queue string queue to poll new tasks on the Rab- bitMQ server broker_probe host string hostname for the RabbitMQ server port integer 5672 port for the RabbitMQ server vhost string virtual host configured for probes username string username used for probes on the RabbitMQ server password string password used for probes on the RabbitMQ server queue string queue to poll new tasks on the Rab- bitMQ server broker_frontend host string hostname for the RabbitMQ server port integer 5672 port for the RabbitMQ server vhost string virtual host configured for frontend username string username used for frontend on the RabbitMQ server password string password used for frontend on the RabbitMQ server queue string queue to poll new tasks on the Rab- bitMQ server sqldb dbms string sqlite dbapi engine dialect string sqlalchemy dialect username string database username password string database password host string database host dbname string /var/irma/ database name db/brain.db tables_prefix string database tables prefix Continued on next page

18 Chapter 3. Manual Installation IRMA Documentation, Release 2.0.4

Table 3.1 – continued from previous page Section Key Type Default Description ftp protocol string “sftp” choose File Transfer Protocol (“sftp” or “ftps”) ftp_brain host string hostname for the FTP server port integer 21 port for the FTP server auth string “password”| SFTP authentication method (“password” or “key”) key_path string sftp private key absolute path

username string username used by probe on the FTP server

password string password used by the probe on the FTP server interprocess_ path string /var/run/ Concurrency file lock lock lock/irma- brain.lock ssl_config activate_ssl boolean False Enable RabbitMQ ssl ca_certs string RabbitMQ SSL certs keyfile string RabbitMQ SSL keyfile certfile string RabbitMQ SSL certfile

Generate a SQLite database for scan tracking

You could easily generate the user database by running the following command. The path of the database is taken from the conﬁguration ﬁle and the folder where the database is going to be stored must be created beforehand.

Note: The default path for the database is /var/irma/db/ make sure it exists before creating user database.

$ cd /opt/irma/irma-brain/current/ $ ./venv/bin/python -m scripts.create_user usage: create_user with a string the rmqvhost used for the frontend the ftpuser used by the frontend example: create_user test1 mqfrontend frontend

To create an entry in the database for the frontend named frontend and which uses the mqfrontend virtual host on the RabbitMQ server, simply run the following commands:

$ ./venv/bin/python -m scripts.create_user frontend mqfrontend frontend

Note: There is a limitation due to SQLite. The folder where the database is stored, plus the database ﬁle must be writable by the user running the worker:

$ sudo chown irma:irma /var/irma/db/brain.db $ sudo chmod a+w /opt/irma/irma-brain

3.1. Brain 19 IRMA Documentation, Release 2.0.4

3.1.6 Installation Checks

Celery Workers

Before going further, you should check that the python applications manages to communicate with the RabbitMQ server through Celery. To ensure that, from the installation directory, execute both Celery workers: On GNU/Linux:

$ cd /opt/irma/irma-brain/current $ ./venv/bin/python -m brain.scan_tasks

------celery@brain v3.1.23(Cipater) ---- **** ------* *** * -- Linux-3.16.0-4-amd64-x86_64-with-debian-8.2 -- * - **** --- - ** ------[config] - ** ------.> app: scantasks:0x7fbd7ee4c350 - ** ------.> transport: amqp://brain:**@127.0.0.1:5672/mqbrain - ** ------.> results: amqp:// - *** --- * --- .> concurrency:2(prefork) -- ******* ------***** -----[queues] ------.> brain exchange=celery(direct) key=brain

[2016-07-15 15:00:36,155: WARNING/MainProcess] celery@brain ready.

This worker is responsible for splitting the whole scan job in multiples job per probe per ﬁle.

$ cd /opt/irma/irma-brain/current $ ./venv/bin/python -m brain.results_tasks

------celery@brain v3.1.23(Cipater) ---- **** ------* *** * -- Linux-3.16.0-4-amd64-x86_64-with-debian-8.2 -- * - **** --- - ** ------[config] - ** ------.> app: resultstasks:0x7fa68f9aa590 - ** ------.> transport: amqp://probe:**@127.0.0.1:5672/mqprobe - ** ------.> results: disabled:// - *** --- * --- .> concurrency:2(prefork) -- ******* ------***** -----[queues] ------.> results exchange=celery(direct) key=results

[2016-07-15 14:59:01,799: WARNING/MainProcess] celery@brain ready.

And this worker is responsible for collecting and tracking results. If your Celery worker does not output something similar to the above output, you should check twice the parameters in the application conﬁguration ﬁle you are using.

SFTP accounts

Try to login as frontend and upload a sample ﬁle in home dir (should raise an error as it is non writeable) then in uploads dir.

20 Chapter 3. Manual Installation IRMA Documentation, Release 2.0.4

$ sftp frontend@localhost frontend@localhost's password: Connected to localhost. sftp> put test Uploading test to /test remote open("/test"): Permission denied sftp> ls uploads sftp> cd uploads/ sftp> put test Uploading test to /uploads/test test

˓→100% 10 0.0KB/s 00:00

3.1.7 Running Brain applications at startup

We have ensured that the freshly installed Brain is ready to be integrated to your IRMA platform. Now, we can go a step further and make it launch automatically all daemons when the system starts up so you will not need to relaunch them manually every time. We are using supervisor to manage our celery daemons.

Installing Supervisor

Install it with pip (python2):

$ sudo pip install supervisor

We will create two new applications called scan_app and result_app.

Conﬁgure Scan APP

Create a ﬁle called scan_app located at /etc/supervisor/conf.d with the following content:

[program:scan_app] numprocs=1 stopwaitsecs= 600 killasgroup= True stderr_logfile= /var/log/supervisor/scan_app.log command= /opt/irma/irma-brain/current/venv/bin/python -m brain.scan_tasks user= irma autostart= True directory= /opt/irma/irma-brain/current stdout_logfile= /var/log/supervisor/scan_app.log

Conﬁgure Result APP

Create a ﬁle called result_app located at /etc/supervisor/conf.d with the following content:

[program:result_app] numprocs=1

3.1. Brain 21 IRMA Documentation, Release 2.0.4

stopwaitsecs= 600 killasgroup= True stderr_logfile= /var/log/supervisor/result_app.log command= /opt/irma/irma-brain/current/venv/bin/python -m brain.results_tasks user= irma autostart= True directory= /opt/irma/irma-brain/current stdout_logfile= /var/log/supervisor/result_app.log

Ensure supervisor will read our ﬁles by checking /etc/supervisor/supervisord.conf last lines should be:

[...] [include] files= /etc/supervisor/conf.d/ *

Restart supervisor:

$ sudo service supervisor restart

Restart applications if needed (should be done automatically):

$ sudo supervisorctl restart all scan_app: stopped result_app: stopped scan_app: started result_app: started

3.2 Frontend

The Frontend handles scan submission to the Brain, stores the results of the scanned ﬁles. These results can be displayed through a web graphical user interface or via the command line interface.

3.2.1 Architecture

Let us recall first the inner architecture of the Frontend. It uses multiple technologies with each a specific purpose: • A client through which an user submits a file and get the analysis results. There are two clients bundled in the repository: a web user interface and a command-line client. • A python-based restful API, served by a nginx web server and a uWSGI application server. It gets the results of a file scan by querying a database. • A worker that will handle scan submission to the Brain and store the results of analyzes scheduled by the Brain. The worker relies on Celery, a python-based distributed task queue. • An database server (PostgreSQL) is used to store results of analyzes made on each file submitted either by the web graphical interface or the CLI client.

3.2.2 Installation

The Frontend must be installed on a GNU/Linux system. With some efforts, it should be possible to run it on a Microsoft Windows system, but this has not been tested yet. This section describes how to get the source code of the application and to install it.

22 Chapter 3. Manual Installation IRMA Documentation, Release 2.0.4

Pre-requisites

Note: On GNU/Linux, one can create the user (and the group) irma with:

$ sudo adduser --system --no-create-home --group irma

Installation on GNU/Linux

On GNU/Linux system, we will assume that the code for the Frontend will be installed in /opt/irma/ irma-frontend/current directory, which is the conﬁgured default installation directory. The source code is hosted on github.com. One can fetch an up-to-date version with the following commands. Let us note that there is a common submodule named common that is a soft-link (lib), do not forget the dereference option of cp (-L):

$ git clone https://github.com/quarkslab/irma /opt/irma/src $ cp -rL /opt/irma/src/frontend /opt/irma/irma-frontend/current

We need few dependencies for future steps: $ sudo apt-get install python3-virtualenv python3-dev then, we need to install python dependencies manually in a virtualenv :

$ virtualenv --system-site-packages /opt/irma/irma-frontend/current/venv $ /opt/irma/irma-frontend/current/venv/bin/pip install -r /opt/irma/irma-frontend/

˓→current/requirements.txt [...]

Building the web client

The first step that should be performed when installing the frontend from the code source is to build the web client which is composed of static HTML files mixed with javascript web framework (in particular, AngularJS). One must install first some tools that are necessary to build the whole web client:

$ curl -sL https://deb.nodesource.com/setup | sudo bash - [...] $ sudo apt-get install -y nodejs [...] $ curl -sL https://www.npmjs.org/install.sh | sudo bash - [...]

Once the tools installed, you can build the static ﬁles for the web user interface with the following commands, executed from the installation directory:

3.2. Frontend 23 IRMA Documentation, Release 2.0.4

$ cd irma-frontend/current/web $ sudo npm install $ sudo node_modules/.bin/bower install --allow-root $ sudo node_modules/.bin/gulp dist

A new directory or an updated directory web/dist should appear now with HTML and javascript files that have been minified and “obfuscated” by gulp. For more details on the web interface, please refer to the dedicated chapter. If everything has gone well, you should have installed the python application on your system. The next step is to configure it for your platform and to install the other components it relies on.

3.2.3 Conﬁguration

The conﬁguration ﬁle is located at config/frontend.ini in the installation directory.

Note: Detailed meaning of each ﬁeld in config/frontend.ini:

Section Key Type Default Description log syslog integer 0 enable rsyslog (experimental) prefix string irma-frontend: prefix to append to rsyslog entries debug boolean False enable Debug log sql_debug boolean False enable SQL debug log sqldb username string database username password string database password host string database host dbname string database name tables_prefix string database tables prefix samples_storage path string Samples storage path celery_brain timeout integer 60 (sec) time before consid- ering that the brain has timed-out celery_frontend timeout integer 30 (sec) time before consid- ering that the frontend has timed-out celery_options concurrency integer 0 number of concurrent workers (0 means nb of cores) soft_time_limit integer 300 (sec) time limit before task soft interrupt time_limit integer 1500 (sec) time limit before task is killed beat_schedule string /var/irma/fronte celery beat schedule nd_beat_schedule file broker_brain host string hostname for the RabbitMQ server port integer 5672 port for the Rab- bitMQ server Continued on next page

24 Chapter 3. Manual Installation IRMA Documentation, Release 2.0.4

Table 3.2 – continued from previous page Section Key Type Default Description vhost string virtual host configured for brain username string username used for brain on the Rab- bitMQ server password string password used for brain on the Rab- bitMQ server queue string queue to poll new tasks on the Rab- bitMQ server broker_frontend host string hostname for the RabbitMQ server port integer 5672 port for the Rab- bitMQ server vhost string virtual host configured for this frontend username string username used for this frontend on the RabbitMQ server password string password used for this frontend on the RabbitMQ server queue string queue to poll new tasks on the Rab- bitMQ server ftp protocol string “sftp” choose File Transfer Protocol (“sftp” or “ftps”) ftp_brain host string hostname for the FTP server port integer 22 port for the FTP server auth string “password” SFTP authentication method (“password” or “key”) key_path string sftp private key absolute path username string username used by this frontend on the FTP server password string password used by this frontend on the FTP server cron_clean_file integer 0 remove file after X _age clean_db_file days (0 means dis- _max_age abled)

Continued on next page

3.2. Frontend 25 IRMA Documentation, Release 2.0.4

Table 3.2 – continued from previous page Section Key Type Default Description string 0 cron hour settings clean_db_cron _hour

string 0 cron minute settings clean_db_cron _minute

string * cron day of week clean_db_scan settings _day_of_week

cron_clean_file string “0” space’s _size clean_fs_max maxi- _size mum size (bytes) dedi- string * cated to clean_fs_size the file _cron_hour system cron hour settings clean_fs_size string 0 cron minute settings _cron_minute string * cron day of week settings clean_fs_size _cron_day_of_week interprocess_ lock path string /var/run/lock/ir ma- Concurrency file frontend.lock lock ssl_config activate_ssl boolean False Enable RabbitMQ ssl ca_certs string RabbitMQ SSL certs keyfile string RabbitMQ SSL keyfile certfile string RabbitMQ SSL certfile

Note: The default path for samples is /var/irma/samples/ make sure it exists with correct rights for irma user before launching your ﬁrst scan.

3.2.4 Installing and conﬁguring uWSGI

The restful API is served by an uWSGI application server. This section will explain how to install an uWSGI server and conﬁgure it for the Frontend.

26 Chapter 3. Manual Installation IRMA Documentation, Release 2.0.4

Installation

On Debian, installing uWSGI to serve a python application is pretty straightforward:

$ cd /opt/irma/irma-frontend/current $ ./venv/bin/pip install https://projects.unbit.it/downloads/uwsgi-lts.tar.gz [...]

Please refer to the documentation of your preferred distribution to install an uWSGI server on it.

3.2.5 Installing SQL server

Note: Since version 1.5.0, IRMA required a postgreSQL server as we are using JSONB column.

The Frontend relies on a PostgreSQL database to keep track of all scans info. On Debian, one can install a PostgreSQL server in few commands: Install it with the following dependencies:

$ sudo apt-get install postgresql-9.4 python3-psycopg2 [...]

You need to manually create the conﬁgured database and db user, the tables are automatically created on uwsgi startup.

3.2.6 Running Frontend applications at startup

We have ensured that the freshly installed Frontend is ready to be integrated to your IRMA platform. Now, we can go a step further and make it launch automatically all daemons when the system starts up so you will not need to relaunch them manually every time. We are using supervisor to manage our python daemons (uswgi for IRMA API and celery for IRMA frontend workers).

Installing Supervisor

Install it with apt:

$ sudo apt-get install python3-virtualenv python3-dev $ sudo pip install supervisor(python2 only)

We will create two new applications called frontend_api and frontend_app. Frontend_api is the python uwsgi server and frontend_app the frontend celery daemon.

Conﬁgure Frontend API

Create a ﬁle called frontend_api located at /etc/supervisor/conf.d with the following content:

[program:frontend_api] numprocs=1 redirect_stderr= True stopwaitsecs= 600 killasgroup= True

3.2. Frontend 27 IRMA Documentation, Release 2.0.4

stderr_logfile= /var/log/supervisor/frontend_api.log stopsignal= INT command= /opt/irma/irma-frontend/current/venv/bin/uwsgi -s 127.0.0.1:8081 --disable-

˓→logging --master --workers4 --need-app --chdir /opt/irma/irma-frontend/current --

˓→home /opt/irma/irma-frontend/current/venv --python-path /opt/irma/irma-frontend/

˓→current/venv --wsgi-file api/base.py --callable __hug_wsgi__ --lazy --offload-

˓→threads4 user= irma autostart= True stdout_logfile= /var/log/supervisor/frontend_api.log

Conﬁgure Frontend APP

Create a ﬁle called frontend_app located at /etc/supervisor/conf.d with the following content:

[program:frontend_app] numprocs=1 stopwaitsecs= 600 killasgroup= True stderr_logfile= /var/log/supervisor/frontend_app.log command= /opt/irma/irma-frontend/current/venv/bin/python -m api.tasks.frontend_app user= irma autostart= True directory= /opt/irma/irma-frontend/current stdout_logfile= /var/log/supervisor/frontend_app.log

Ensure supervisor will read our ﬁles by checking /etc/supervisor/supervisord.conf last lines should be:

[...] [include] files= /etc/supervisor/conf.d/ *

Restart supervisor:

$ sudo service supervisor restart

Restart applications if needed (should be done automatically):

$ sudo supervisorctl restart all frontend_api: stopped frontend_app: stopped frontend_app: started frontend_api: started

3.2.7 Installing and conﬁguring nginx

In the Frontend, we use a nginx web server to serve the uWSGI application and the static web site that query the API in order to get results of scanned ﬁles and to present them to the user.

Installation

On Debian, installing nginx is done with few commands:

28 Chapter 3. Manual Installation IRMA Documentation, Release 2.0.4

$ sudo apt-get install nginx [...]

Please refer to the documentation of your preferred distribution to install an uWSGI server on it.

Conﬁguration

We provide several template scripts in the extras/ repository located at the root of the installation directory. Tem- plates for nginx are located in extras/nginx/. Copy the ﬁle to sites-available in nginx conﬁguration folder:

$ sudo cp extras/nginx/frontend /etc/nginx/sites-available/

The template is conﬁgured for a default installation of the frontend in /opt/irma/irma-frontend/current. You may need to edit it for you setup. In particular, please ensure that the root directive points to the folder web/ dist (for a production ready version of the web site) or web/app (for a development version of the web site) in your installation directory.

Activate the web site

By default, all websites in apps-available are not activated. One can enable the website described in frontend conﬁguration ﬁle by creating a soft-link into the sites-enabled folder:

$ sudo ln -s /etc/nginx/sites-available/frontend /etc/nginx/sites-enabled/frontend

Note: The case of HTTPs A template to set up a HTTPs server with nginx is also provided in the extras/nginx folder. Here is the way to setup it:

$ sudo cp extras/nginx/frontend-https /etc/nginx/sites-available/ $ sudo ln -s /etc/nginx/sites-available/frontend-https /etc/nginx/sites-enabled/

˓→frontend-https $ sudo mkdir /etc/nginx/certificates/ $ cd /etc/nginx/certificates/

Generate the required Difﬁe Hellman Ephemeral Parameters:

$ sudo openssl dhparam -out dhparam.pem 4096

Finally, generate a self signed certiﬁcate:

$ sudo openssl req -x509 -nodes -days 365 -newkey rsa:4096 -subj"/CN= $(hostname --

˓→fqdn)/" -keyout frontend.key -out frontend.crt

Relaunch the service

The ﬁnal step is to relaunch the service:

$ sudo invoke-rc.d nginx restart

3.2. Frontend 29 IRMA Documentation, Release 2.0.4

3.2.8 Installation Checks

Celery Workers

Before going further, you should check that the python application manages to communicate with the RabbitMQ server and the Redis server through Celery. To ensure that, from the installation directory, execute the Celery worker: On GNU/Linux:

$ cd /opt/irma/irma-frontend/current $ ./venv/bin/python -m api.tasks.frontend_app

------celery@frontend v3.1.23(Cipater) ---- **** ------* *** * -- Linux-3.2.0-4-amd64-x86_64-with-debian-7.6 -- * - **** --- - ** ------[config] - ** ------.> app: frontend.tasks:0x1e18750 - ** ------.> transport: amqp://probe:**@brain:5672/mqfrontend - ** ------.> results: disabled - *** --- * --- .> concurrency:2(prefork) -- ******* ------***** -----[queues] ------.> frontend exchange=celery(direct) key=frontend

[2014-08-20 15:28:58,745: WARNING/MainProcess] celery@frontend ready.

SFTP accounts

Defaut File Transport Protocol since v1.4.0 is now SFTP, you can check whether the conﬁgured account is valid.

$ sftp @

FTP-TLS accounts

Additionnally, if you have conﬁgured IRMA to use FTP-TLS, you can check whether the conﬁgured account is valid. On Debian, this can be done with the ftp-ssl package:

30 Chapter 3. Manual Installation IRMA Documentation, Release 2.0.4

[SSL Cipher DHE-RSA-AES256-GCM-SHA384] 200 PBSZ=0 200 Data protection level set to "private" 331 User probe OK. Password required Password: frontend-ftp-password 230 OK. Current directory is / Remote system type is UNIX. Using binary mode to transfer files. ftp>

Restful API

One can verify that the restful API is up and running by querying a speciﬁc route on the web server or by checking the system logs:

$ curl http://localhost/api/v1.1/probes {"total":9, "data":["ClamAV", "ComodoCAVL", "EsetNod32", "FProt", "Kaspersky",

˓→"McAfeeVSCL", "NSRL", "StaticAnalyzer", "VirusTotal"]}

$ sudo cat /var/log/supervisor/frontend_api.log [...] added /opt/irma/irma-frontend/current/venv/ to pythonpath. *** uWSGI is running in multiple interpreter mode *** spawned uWSGI master process(pid: 3943) spawned uWSGI worker1(pid: 3944, cores:1) spawned uWSGI worker2(pid: 3945, cores:1) spawned uWSGI worker3(pid: 3946, cores:1) spawned uWSGI worker4(pid: 3947, cores:1) mounting frontend/api/base.py on /api mounting frontend/api/base.py on /api mounting frontend/api/base.py on /api mounting frontend/api/base.py on /api WSGI app0(mountpoint='/api') ready in0 seconds on interpreter 0x99a3e0 pid: 3945

˓→(default app) WSGI app0(mountpoint='/api') ready in0 seconds on interpreter 0x99a3e0 pid: 3946

˓→(default app) WSGI app0(mountpoint='/api') ready in0 seconds on interpreter 0x99a3e0 pid: 3944

˓→(default app) WSGI app0(mountpoint='/api') ready in0 seconds on interpreter 0x99a3e0 pid: 3947

˓→(default app)

3.3 Probe

The Probes are python-based application that host a single or multiple analyzers. Each analyzer listens on a speciﬁc work queue and waits for an analysis to be scheduled by the Brain through Celery, an open source task framework for Python. Python version should be at least 3.4 on linux, 3.5 on windows.

3.3.1 Architecture

Probes are mainly Celery workers that handle scan requests from Brain

3.3. Probe 31 IRMA Documentation, Release 2.0.4

3.3.2 Installation

Probes can be installed either on GNU/Linux and on Microsoft Windows systems. The installation procedure may differs between the two systems.

Downloading the source code from github.com

The source code is hosted on github.com. One can fetch an up-to-date version with the following commands. Let us note that there is a common submodule named common that is a soft-link (lib), do not forget the dereference option of cp1 (-L):

# If src repository not already cloned $ git clone https://github.com/quarkslab/irma /opt/irma/src $ cp -rL /opt/irma/src/probe /opt/irma/irma-probe/current

We assume that you have a command line interface on your system with the following tools installed: • python 3.4.x and newer on Linux • python 3.5.x and newer on Windows (see Python for Windows for prebuild MSI installer) • python3-pip (see Install pip for the recommended way to install pip package manager) Furthermore, we assume that you have created an user irma that will be used to run the python application.

Note: On GNU/Linux, one can create the user (and the group) irma with:

$ sudo adduser --system --no-create-home --group irma

Installation on Microsoft Windows

On windows system, we will assume that the code for the Probe will be installed at the root of the C:\ drive, namely in C:\irma\irma-probe\current.

$ C:\Python27\Scripts\pip.exe install virtualenv $ C:\Python27\Scripts\virtualenv C:\irma\irma-probe\current\venv

$ cd C:\irma\irma-probe\current\ $ .\venv\Scripts\pip.exe install -r requirements.txt [...]

Installation on GNU/Linux

On GNU/Linux system, we will assume that the code for the Probe will be installed in /opt/irma/irma-probe/ current directory. The source code is hosted on github.com. One can fetch an up-to-date version with the following commands. Let us note that there is a common submodule named common that is a soft-link (lib), do not forget the dereference option of cp (-L):

1 On Microsoft Windows, a Linux-like lightweight command line interface can be installed by installing MSYS or Git for windows.

32 Chapter 3. Manual Installation IRMA Documentation, Release 2.0.4

# If src repository not already cloned $ git clone https://github.com/quarkslab/irma /opt/irma/src $ cp -rL /opt/irma/src/probe /opt/irma/irma-probe/current

We need few dependencies for future steps: $ sudo apt-get install python3-virtualenv python3-dev then, we need to install python dependencies manually in a virtualenv :

$ virtualenv --system-site-packages --python=/usr/bin/python3 /opt/irma/irma-probe/

˓→current/venv $ /opt/irma/irma-probe/current/venv/bin/pip install -r /opt/irma/irma-probe/current/

˓→requirements.txt [...]

If everything has gone well, the python application is now installed on your system. The next step is to conﬁgure it for your platform and to enable the analyzers you need.

3.3.3 Conﬁguration

The conﬁguration ﬁle is config/probe.ini located in the installation directory.

Note: We recall in the following the meaning of each ﬁeld in config/probe.ini:

Section Key Type Default Description log syslog integer 0 enable rsyslog (experimental) prefix string irma-probe: prefix to append to rsyslog entries celery_options concurrency integer 0 number of concurrent workers (0 means nb of cores) soft_time_limit integer 300 (sec) time limit before task soft interrupt time_limit integer 1500 (sec) time limit before task is killed broker probe host string hostname for the RabbitMQ server port integer 5672 port for the RabbitMQ server vhost string virtual host configured for probes username string username used for probes on the RabbitMQ server password string password used for probes on the RabbitMQ server queue string queue to poll new tasks on the Rab- bitMQ server ftp_brain host string hostname for the FTP server port integer 21 port for the FTP server auth string “password”| SFTP authentication method (“password” or “key”) key_path string sftp private key absolute path

username string username used by probe on the FTP server password string password used by the probe on the FTP server

3.3. Probe 33 IRMA Documentation, Release 2.0.4

3.3.4 Enabling Analyzers

The python application auto-discovers available analyzers. As long as the requirements are fulﬁlled for an analyzer, it is automatically discovered and enabled. By default, no analyzers are available. The following enumerates the requirements to enable each analyzer bundled with the application.

ClamAV - GNU/Linux

To enable ClamAV on Debian Stable distribution, one should install several packages:

$ sudo apt-get install clamav-daemon [...] $ sudo freshclam [...] $ sudo service clamav-daemon restart [...]

ComodoCAVL - GNU/Linux

Comodo Antivirus for Linux can be downloaded from the Comodo’s download page. The following instruction enable to install the Debian package. By default, the binaries are installed in /opt/COMODO/ directory. As ComodoCAVL is not packaged for the current Debian Stable distribution, we must install it manually:

$ sudo apt-get install binutils $ ar x cav-linux_1.1.268025-1_iXXX.deb $ sudo tar xvf ~/data.tar.gz -C / [...]

Updates for the database can be performed with the following command:

$ XAUTHORITY="$HOME/.Xauthority" sudo /opt/COMODO/menu/comodo-updater [...]

Note: Dependencies to update the database To be able to update the database using the updater provided with Comodo Antivirus for Linux, some dependecies needed for the graphical interface may be missing from the distribution. On Debian Stable system, one can install them with:

$ sudo apt-get install libqt4-sql libqt4-network libqtgui4

EsetNod32 - GNU/Linux

One can request a trial version of Eset Nod32 Business Edition for Linux on the Eset download page. Once downloaded, the anti-virus can be installed with the following commands on Debian. Just follow the typical installation on the GUI:

$ sudo chmod u+x eset_nod32av_64bit_en.linux $ sudo apt-get install libgtk2.0-0 libc6-i386 $ sudo ./eset_nod32av_64bit_en.linux [...]

34 Chapter 3. Manual Installation IRMA Documentation, Release 2.0.4

Binaries should be installed in /opt/eset/esets directory. Updates are performed from the GUI:

$ XAUTHORITY="$HOME/.Xauthority" sudo /opt/eset/esets/bin/esets_gui

Note: Disabling the antivirus daemon To avoid the anti-virus to protect your system at startup, we deliberately disabled the script used to launch the anti-virus early at boot:

$ sudo service esets stop $ sudo mv /etc/init.d/esets /etc/init.d/esets.disable

F-Prot - GNU/Linux

A copy of F-PROT anti-virus for Linux workstations is available on the F-PROT download page. The binaries should be installed in /usr/local/f-prot to make the python application detect it automatically.

$ sudo tar xvf fp-Linux.x86.32-ws.tar.gz -C /usr/local/

To launch an update, a conﬁguration step is mandatory:

$ sudo cp /usr/local/f-prot/f-prot.conf.default /etc/f-prot.conf

An update is launched with:

$ sudo ./fpupdate ERROR: ld.so: object 'libesets_pac.so' from /etc/ld.so.preload cannot be preloaded:

˓→ignored. [...]

Note: Error If you see an error message like:

DownloadingWarning: Network - Connection failed(18), trying again... Downloading updateError: Update - Bad mergefile

Just relaunch the script.

Note: Dependencies to update the database To be able to update the database using the updater provided with Comodo install them with:

$ sudo apt-get install libc6-i386

McAfeeVSCL - GNU/Linux or Microsoft Windows

A free evaluation of McAfee VirusScan Command Line can be downloaded from the editor download page.

3.3. Probe 35 IRMA Documentation, Release 2.0.4

The binaries should be installed in /usr/local/uvscan/ on GNU/Linux system and must be installed in C:\VSCL on Windows Systems. Let us note that updates must be performed manually. Anti-virus databases and engines can be downloaded here. After downloading McAfee Virus Scan archive, create /usr/local/uvscan and extract the archive in it:

$ sudo mkdir /usr/local/uvscan $ sudo tar xvf vscl-XXX.tar.gz -C /usr/local/uvscan # replace using your values $ sudo chmod +x /usr/local/uvscan/uvscan

Extract also, using unzip program, the database:

$ sudo unzip avvepo7536dat.zip -d /usr/local/uvscan $ cd /usr/local/uvscan $ sudo unzip avvdat-XXXX.zip

Sophos - GNU/Linux or Microsoft Windows

A free evaluation of Sophos Endpoint Antivirus can be downloaded from the editor download page. You should receive by email a username and a password to authenticate for updates. It should be detected automatically by IRMA if the anti-virus is installed in its default installation directory. On GNU/Linux: • Download the archive for Linux, then execute:

$ tar zxf sav-linux-9-i386.tgz $ ./sophos-av/install.sh /opt/sophos-av --acceptlicence --autostart=False --

˓→enableOnBoot=False --automatic --ignore-existing-installation $ /opt/sophos-av/bin/savconfig set EnableOnStart false $ /opt/sophos-av/bin/savconfig set PrimaryUpdateSourcePath "sophos:" $ /opt/sophos-av/bin/savconfig set PrimaryUpdateUsername "" $ /opt/sophos-av/bin/savconfig set PrimaryUpdatePassword "" $ /opt/sophos-av/bin/savupdate

Kaspersky - Microsoft Windows

A free evaluation of Kaspersky Internet Security can be requested on the editor download page. It should be detected automatically if the anti-virus is installed in its default installation directory.

Symantec - Microsoft Windows

The procedure to install a trial version of Symantec Endpoint Protection is particularly tedious. We will not document its installation.

G Data - Microsoft Windows

A trial version of G Data is available on the editor download page html>. It should be detected automatically if the anti-virus is installed in its default installation directory.

36 Chapter 3. Manual Installation IRMA Documentation, Release 2.0.4

VirusTotal - GNU/Linux or Microsoft Windows

The VirusTotal analyzer can be installed easily by downloading the python packages it depends on and by modifying its conﬁguration ﬁle. From the installation directory, one can execute: On GNU/Linux:

$ pip install -r modules/external/virustotal/requirements.txt [...] then update conﬁguration ﬁle located at modules/external/virustotal/config.ini.

Note: Meaning of the fields in the configuration file

Section Option Type Default Description VirusTotal apikey string api_key used to query VirusTotal API VirusTotal private boolean use private api (need a private api key)

NSRL - GNU/Linux

The National Software Reference Library can be downloaded on the NIST’s web page. The provided files are stored in the RDS (Reference Data Set) format. To use this analyzer, one must build first the database. We use LevelDB as fast key-value storage library. To build the dabatase, one must install first the dependencies:

$ pip install -r modules/database/nsrl/requirements.txt

A (not optimized and very slow) helper script is provided to build the database:

$ mkdir /home/irma/leveldb $ python -m modules.database.nsrl.nsrl create -t os NSRLOS.txt /home/irma/leveldb/os_

˓→db $ python -m modules.database.nsrl.nsrl create -t manufacturer NSRLMfg.txt /home/irma/

˓→leveldb/mfg_db $ python -m modules.database.nsrl.nsrl create -t product NSRLProd.txt /home/irma/

˓→leveldb/prod_db $ python -m modules.database.nsrl.nsrl create -t file NSRLFile.txt /home/irma/leveldb/

˓→file_db

Finally, one must indicate to the analyzer where to find the files for the database by filling the configuration file located at modules/database/nsrl/config.ini.

Note: Meaning of the fields in the configuration file

Section Option Type Default Description NSRL nsrl_os_db string location of the OS database nsrl_mfg_db string location of the Manufacturer database nsrl_ﬁle_db string location of the File database nsrl_prod_db string location of the Product database

3.3. Probe 37 IRMA Documentation, Release 2.0.4

Note: Error If you see an error message like: fatal error: Python.h: No such file or directory

Then you’ll need to install python3-dev package (for Debian like systems).

Note: leveldb.LevelDBError: IO error: /home/irma/leveldb/ﬁle_db/LOCK: Permission denied If you encounter this problem, you likely have a problem with unix permissions. Please ensure that the folder is owned by the user running the probes. On supervisord-based installation (default for vagrant/ansible scripts), the folder owner should be set to nobody. For init.d-based installation, it should be irma instead.

StaticAnalyzer - GNU/Linux or Microsoft Windows

The PE File analyzer adapted from Cuckoo Sandbox can be installed easily. One need to install the python packages it depends on. From the installation directory, one can execute: On GNU/Linux:

$ pip install -r modules/metadata/pe_analyzer/requirements.txt [...]

On Microsoft Windows, you need cygwin to successfully install the requirements (see python3-magic documentation for installation details):

$ C:\Python27\Scripts\pip.exe install -r modules/metadata/pe_analyzer/requirements.txt [...]

Yara - GNU/Linux or Microsoft Windows

Please refer to modules/metadata/yara/README.md ﬁle for the documentation.

Guide on Debian (credits to Carbonn)

1. Installing dependencies

$ sudo apt-get install libtool automake bison

2. Installing Yara python modules

$ git clone https://github.com/plusvic/yara.git $ autoreconf -i --force $ ./configure $ make $ sudo make install $ python setup.py build $ sudo python setup.py install $ sudo ldconfig

38 Chapter 3. Manual Installation IRMA Documentation, Release 2.0.4

3. Conﬁguring for IRMA

$ mkdir /opt/irma/yara_rules/ $ cat /opt/irma/yara_rules/yara_rules.yar << EOF # Insert rule below inside the file rule silent_banker : banker { meta: description = "This is just an example" thread_level = 3 in_the_wild = true

strings: $a = {6A 40 68 00 30 00 00 6A 14 8D 91} $b = {8D 4D B0 2B C1 83 C0 27 99 6A 4E 59 F7 F9} $c = "UVODFRYSIHLNWPEJXQZAKCBGMT"

condition: $a or $b or $c } EOF

3.3.5 Installation Checks

Celery Workers

$ cd /opt/irma/irma-probe/current $ ./venv/bin/python -m probe.tasks WARNING:root: *** [plugin] Plugin failed to load: Kaspersky miss dependencies: win32 ˓→(PlatformDependency). WARNING:root: *** [plugin] Plugin failed to load: Sophos miss dependencies: win32 ˓→(PlatformDependency). WARNING:root: *** [plugin] Plugin failed to load: Symantec miss dependencies: win32 ˓→(PlatformDependency). WARNING:root: *** [plugin] Plugin failed to load: NSRL miss dependencies: leveldict ˓→(ModuleDependency). See requirements.txt for needed dependencies WARNING:root: *** [plugin] Plugin failed to load: VirusTotal miss dependencies: virus_ ˓→total_apis(ModuleDependency). See requirements.txt for needed dependencies WARNING:root: *** [plugin] Plugin failed to load: StaticAnalyzer miss dependencies: ˓→pefile(ModuleDependency). See requirements.txt for needed dependencies

------celery@irma-probe v3.1.13(Cipater) ---- **** ------* *** * -- Linux-3.2.0-4-amd64-x86_64-with-debian-7.6 -- * - **** --- - ** ------[config] - ** ------.> app: probe.tasks:0x1e18750 - ** ------.> transport: amqp://probe:**@brain:5672/mqprobe - ** ------.> results: redis://brain:6379/1 - *** --- * --- .> concurrency:1(prefork) -- ******* ----

3.3. Probe 39 IRMA Documentation, Release 2.0.4

--- ***** -----[queues] ------.> ClamAV exchange=celery(direct) key=ClamAV .> ComodoCAVL exchange=celery(direct) key=ComodoCAVL .> EsetNod32 exchange=celery(direct) key=EsetNod32 .> FProt exchange=celery(direct) key=FProt .> McAfeeVSCL exchange=celery(direct) key=McAfeeVSCL

[2014-08-19 18:25:06,469: WARNING/MainProcess] celery@irma-probe ready.

The equivalent command on Microsoft Windows is:

C:\> cd "C:\IRMA\irma-probe\current" C:\> .\venv\Scripts\python.exe -m probe.tasks [...]

If your Celery worker does not output something similar to the above output, you should check twice the parameters in the application conﬁguration ﬁle you are using. Let us note that the Celery worker gives us useful information on the analyzer that are enabled. As each analyzer connects to a dedicated queue, we can deduce, in this example, that the analyzers ClamAV, ComodoCAVL, EsetNod32, FProt and McAfeeVSCL are enabled and ready to serve.

SFTP accounts

Defaut File Transport Protocol since v1.4.0 is now SFTP, you can check whether the conﬁgured account is valid.

$ sftp @

FTP-SSL accounts

Additionnally, if you have conﬁgured IRMA to use FTP-ssl, you can check whether the conﬁgured account is valid. On Debian, this can be done with the ftp-ssl package:

$ sudo apt-get install ftp-ssl [...] $ ftp-ssl Connected to brain. 220------Welcome to Pure-FTPd[privsep][TLS] ------220-You are user number1 of 50 allowed. 220-Local time is now 18:55. Server port: 21. 220-This is a private system - No anonymous login 220-IPv6 connections are also welcome on this server. 220 You will be disconnected after 15 minutes of inactivity. Name(brain:root): probe-ftp 500 This security scheme is not implemented 234 AUTH TLS OK. [SSL Cipher DHE-RSA-AES256-GCM-SHA384] 200 PBSZ=0 200 Data protection level set to "private" 331 User probe OK. Password required Password: probe-ftp-password 230 OK. Current directory is / Remote system type is UNIX. Using binary mode to transfer files. ftp>

40 Chapter 3. Manual Installation CHAPTER 4

Database migration

IRMA uses Alembic to manage and perform databases migration.

Note: Alembic is a useful tool to manage migration, but can’t surpass local engine implementation of SQL. As SQLite doesn’t manage schema modiﬁcations such as ALTER_COLUMN, the whole migration system of IRMA won’t support it. The preferred database engine is PostgreSQL. You can still use SQLite, but you will be on your own for migrations.

Warning: Please note that most of the manipulations on this can and sometimes will alter your data. If you are not sure about what you are doing, and even if you are sure, make backup.

4.1 Requirements

• Alembic

4.2 Content

Database migrations are managed in the frontend and brain IRMA components. The ﬁles/directories used are: alembic.ini extras/migration/ +- env.py +- script.py.mako +- versions/ +- .py

41 IRMA Documentation, Release 2.0.4

+- .py +- ...

Note: All the commands below will assert to be executed on top of this file system, as Alembic needs the alembic. ini configuration file. You could also use the -c .

4.3 Usage

Alembic manage a ‘revision’ for each database evolution. These revisions are used to upgrade or downgrade the database schema. The command:

$ alembic current

. . . shows the current revision of the database. The command to get the history of the latest alembic migrations is:

$ alembic history --verbose

4.3.1 Create database from scratch with Alembic

Conﬁguration and creating database

Alembic will use the information in the [sqldb] section of the conﬁguration ﬁles (respectively config/ frontend.ini or conf/brain.ini for the repositories of the frontend or the brain components). Make sure they are accurate. The database must already exist. This step is quite simple, the SQL command usually being: sql$ CREATE DATABASE ;

Update your schema with Alembic

If you use a virtualenv, activate it. Then enter:

$ alembic upgrade head

Alembic applies each revision one after the other. At the end of the process, if no error occurs, your database should be updated.

Note: You can update the database one revision at a time, or up to a speciﬁc revision. See the revisions section for further information.

42 Chapter 4. Database migration IRMA Documentation, Release 2.0.4

4.3.2 If you already have a database WITHOUT Alembic

Alembic stores its current revision number in database. If your database doesn’t have this information, you are very likely to encounter errors when using Alembic, as it will try to create already existing tables. The easiest solution is to destroy your database and go for a fresh install. Although, if you don’t want to lose your data, you could update the Alembic information manually. You will need to: 1. Get the exact current Alembic revision of your database. Each migration ﬁle has a Revision ID in its header. Investigate the successive revisions to know which one matches your current database state. 2. Once you known your Alembic revision, run:

$ alembic stamp

3. Your database is now synchronized with Alembic! You should be able to use Alembic to upgrade/downgrade your database now. Be aware that if the revision number you provided is false, you could encounter massive errors while attempting to upgrade/downgrade your database.

4.3.3 Generating a new revision

Creating a new revision can be done with the command:

$ alembic revision -m

This command produces a new _.py file in the extras/migration/ versions/ directory. This file contains two functions upgrade and downgrade, respectively used to upgrade the database to the revision, or downgrade from it. These two functions are empty and must be completed with the desired modifications (see the alembic documentation). A revision could be produced automatically, from database metadata defined in the IRMA SQL objects description through sqlalchemy, with the command:

$ alembic revision --autogenerate -m

These SQL objects are defined in: • frontend/models/sqlobjects.py for the frontend, • brain/models/sqlobjects.py for the brain. Alembic scripts in IRMA repositories are already configured to use metadata defined in these files. You should be able to use the --autogenerate option without further modifications.

Note: IRMA configuration allows to prefix table names through configuration. Our revision files use the function /config/parser.py:prefix_table_name to generate table names rather than keeping alembic-generated plain string names. A good practice would be to keep using this function in revision files.

Warning: Alembic easily detects changes such as adding/removing columns, but could be blind on thin, inner modiﬁcations. Re-reading the auto-generated script is a strongly recommended step before actually performing the migration. See the alembic documentation for more information.

4.3. Usage 43 IRMA Documentation, Release 2.0.4

Warning: Database modiﬁcations using ALTER_COLUMN (such as changing the type of a column) can’t be performed on SQLite databases. Be aware of this limitation if you absolutely want to use migration scripts with this SQL engine.

4.3.4 Migrating between revisions

Once the revision is properly described, the migration is performed with:

$ alembic upgrade head

Alembic allows to migrate the database to any revision, relatively to the current revision or absolutely. Several exam- ples:

$ alembic upgrade +4 $ alembic downgrade base $ alembic upgrade +3

4.4 Tips and tricks

Note: Don’t trust Alembic too much. It is nothing more than a tool, without any comprehension on the code. Cautiously read the revision scripts it generates.

Note: Database migration is hardly ever a painless step. Be sure to: 1. save your data before performing a migration, 2. test your application after the migration to ensure its compatibility with the new data schemes.

Note: With a PostgreSQL database, the Float type is tolerated but the real type name used by the database is Real. It means that SQL objects described in sqlalchemy with Float columns will be properly applied in database, but at each autogenerate revision, alembic will see Real type in database, against Float type in the code metadata, and so will perform each time a useless alter_column from Real to Float. This problem could be avoided (with PostgreSQL) by declaring Real instead of Float. See this page for more information on PostgreSQL numeric types.

Note: Alembic can’t directly deal with many somehow complex operations, such as type migration with no trivial cast. In these cases, the operation must be manually described with a raw SQL command (which could be database-dependent). For instance, alembic can’t perform the migration from real to datetime:

> alembic.alter_column('table','column', existing_type=sqlalchemy.REAL(),

44 Chapter 4. Database migration IRMA Documentation, Release 2.0.4

type_=sqlalchemy.DateTime(), existing_nullable=False)

. . . because of an error a column "column" cannot be cast automatically to type timestamp with time zone. A proper migration for PostgreSQL would be (in Python):

> alembic.execute('ALTER TABLE"table" ALTER COLUMN"column" TYPE TIMESTAMP WITHOUT

˓→TIME ZONE USING to_timestamp(column)')

And the reverse code to downgrade the migration could be:

> alembic.execute('ALTER TABLE"table" ALTER COLUMN"column" TYPE REAL USING

˓→extract(epoch from column)')

Note: Rather than managing migrations directly with Alembic, we could generate SQL migration revision to be used directly on database with the command:

$ alembic upgrade --sql > migration.sql

Note: Deleting a revision R is simple: • downgrade the database to the revision before R-1 the revision you want to delete; • if any, edit the script of the following revision R+1 and update the down_revision variable to match the revision number of revision R-1; • delete the script of the revision R you want to delete; • upgrade your database. The deleted revision want be applied any more.

4.4. Tips and tricks 45 IRMA Documentation, Release 2.0.4

46 Chapter 4. Database migration CHAPTER 5

To evolve IRMA

5.1 Adding a new probe

5.1.1 Writing a Plugin for the probe

Note: To be a valid probe module, IRMA expects it to have a predeﬁned structure. To save time, one can get a minimal working structure from the skeleton plugin. The new plugin is stored in the appropriate sub-directory of the directory probe/modules according to the type of the new probe (antivirus, metadata, external. . . ).

For a probe that is not a antivirus

1. Copy the directory skeleton to the new module (appropriate localisation). Example with a module my_module with metadata type :

$cp -r probe/modules/custom/skeleton/ probe/modules/metadata/my_module

2. If there are packages to install, specify them in the file requirements.txt. Otherwise remove the file 3. Adjust the file plugin.py according to the module : • Adjust the class’s name with the name of your probe • Fill in the fields of the class :- _plugin_name_ = [the plugin name] - _plugin_display_name_ = [the field _name of the class of the probe] - _plugin_version_ = [the version number] - _plugin_category = [the type of the probe: IrmaProbeType.] - _plugin_description = [uick description] - _plugin_dependencies = [list of dependencies: platform, binary or/and file] => if used import from lib.plugins PlatformDependency, BinaryDependency or/and FileDepen- dency – _mimetype_regexp = [mimetype corresponding] 4. Implement the functions corresponding to the type of the plugin

47 IRMA Documentation, Release 2.0.4

For an antivirus

In the case of an antivirus, it is a little different because an Antivirus class was created to avoid code’s duplication. You can use the skeleton below: plugin.py:

# # Copyright (c) 2013-2018 Quarkslab. # This file is part of IRMA project. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License in the top-level directory # of this distribution and at: # # http://www.apache.org/licenses/LICENSE-2.0 # # No part of the project, including this file, may be copied, # modified, propagated, or distributed except according to the # terms contained in the LICENSE file. from.skeleton import Skeleton from lib.plugins import PluginBase, PluginLoadError from lib.irma.common.utils import IrmaProbeType class SkeletonPlugin(PluginBase, Skeleton):

# ======# plugin metadata # ======_plugin_name_="Skeleton" _plugin_display_name_= Skeleton._name _plugin_author_="IRMA (c) Quarkslab" _plugin_version_="1.0.0" _plugin_category_="custom" _plugin_description_="Plugin skeleton" _plugin_dependencies_=[] _mimetype_regexp= None

# ======# constructor # ======

def __init__(self): self.module= Skeleton() skeleton.py:

48 Chapter 5. To evolve IRMA IRMA Documentation, Release 2.0.4

# http://www.apache.org/licenses/LICENSE-2.0 # # No part of the project, including this file, may be copied, # modified, propagated, or distributed except according to the # terms contained in the LICENSE file. import logging

# Choose the class you need to inherit from from modules.antivirus.base import AntivirusUnix, AntivirusWindows log= logging.getLogger(__name__)

# Inhererit from AntivirusUnix or AntivirusWindows according to your plateform class Skeleton(Antivirus): name="Skeleton for Antivirus"

# ======# Constructor and destructor stuff # ======

def __init__(self, *args, **kwargs): # class super class constructor super().__init__(*args, **kwargs)

# do your initialization stuff

# Finally initialize all the attributes self._init_attributes()

The recipe is the same, the files with the corresponding module name and differents fields need to be updated. The attributes in Antivirus._attributes are meant to be defined by the instanciation. One can either: • leave it blank, in this case the super class will assign it a default value (eg. "unavailable" for self. version); • define it directly (eg. self.scan_path = Path("/opt/skeleton/skeleton")); • define a function to be called to assign it (eg. def get_scan_path(self): ...), the super class will take care of calling it and handling exceptions.

5.1.2 Testing the new plugin

Before testing, module’s necessary stuff (binaries, ﬁles, etc) must be provisioned to the VM.

$cd ansible $vagrant rsync $vagrant ssh $sudo su deploy $cd /opt/irma/irma-probe/current $venv/bin/python -m extras.tools.run_module

This last command lists available modules. Now, if the new module is available, its launching can be done:

$venv/bin/python -m extras.tools.run_module my_module file

5.1. Adding a new probe 49 IRMA Documentation, Release 2.0.4

5.1.3 Automatic provisioning

Creating a new role

Create a new directory with this structure: cd ansible tree roles/quarsklab.my_module roles/quarkslab.my_module/ +-- defaults | +-- main.yml +-- tasks +-- main.yml tasks/main.yml is the default entry point for a role containing Ansible tasks. In this file, write the instruction to install the module. Add the file tasks/update.yml to write the informations for the update if necessary. In defaults/main.yml it is usual to store default variables for this role. If there are particular instructions, for example how to obtain a licence for a antivirus, add a README file.

Invoking the module role

Modify playbooks/provisioning.yml : add the module

-name : my_module hosts: my_module roles: - { role: quarkslab.my_module, tags: 'my_module'}

If a task update was deﬁned, add the module in playbooks/updating.yml :

-name : my_module hosts: my_module roles: - { role: quarkslab.module, tags: 'my_module', task_from : update}

Deﬁning hosts

Modify the environment to add the new probe. For example for the allinone_dev :

$ cat environments/allinone_dev.yml [ ... snip ...] virustotal: - brain.irma my_module: - brain.irma "probe:children": - clamav - comodo - mcafee - static-analyzer - virustotal - my_module

50 Chapter 5. To evolve IRMA CHAPTER 6

References

6.1 Disclaimer

IRMA is distributed as it is, in the hope that it will be useful, but without any warranty neither the implied mer- chantability or ﬁtness for a particular purpose. Whatever you do with this tool is uniquely your own responsibility.

6.2 License

IRMA source code is licensed under Apache License, version 2.0. The full license text can be found below (Apache License, version 2.0).

6.3 Apache License, version 2.0

6.4 Authors

51 IRMA Documentation, Release 2.0.4

52 Chapter 6. References CHAPTER 7

Frequently Asked Questions

7.1 Playing with tags

Tags are available in IRMA from version 1.3.0

7.1.1 Creating a tag

You could create tags by using the command line tools

>>> from irmacl.helpers import * >>> tag_list() []

>>> tag_new("archive") {u'text': u'archive', u'id': 1}

>>> tag_list() [Tag archive [1]] or directly from your terminal by using curl and posting a json with ‘text’ key:

$ curl -H "Content-Type: application/json; charset=UTF-8" -X POST -d '{"text":"

˓→tag>"}' http://172.16.1.30/api/v1.1/tags

Note: There is currently no way to create a tag directly from the web IHM.

7.1.2 Tagging a File

Directly in web IHM, once you are on a ﬁle details page:

53 IRMA Documentation, Release 2.0.4

Just click the tag bar and you will see all available tags. You could add multiple tags.

It is also possible to add a tag through command line tools:

54 Chapter 7. Frequently Asked Questions IRMA Documentation, Release 2.0.4

>>> from irmacl.helpers import * >>> file_tag_add? Signature: file_tag_add(sha256, tagid, verbose=False) Docstring: Add a tag to a File

:param sha256: file sha256 hash :type sha256: str of (64 chars) :param tagid: tag id :type tagid: int :return: No return

>>> file_tag_add("346ae869f7c7ac7394196de44ab4cfcde0d1345048457d03106c1a0481fba853",1)

7.1.3 Searching by tag

You could specify one or more tags while searching for ﬁles too:

choose your tag list then hit the search button:

or by command line:

>>> from irmacl.helpers import * >>> file_search(tags=[1]) (1, [])

7.1. Playing with tags 55 IRMA Documentation, Release 2.0.4

7.2 SSL settings

7.2.1 Making RabbitMQ running SSL

Certiﬁcates generation

See rabbitmq detailled guide on how to generate server and clients certiﬁcates here

Update server settings on brain

Copy the RabbitMQ conﬁguration template provided in “./extras/rabbitmq” to “/etc/rabbitmq” Restart RabbitMQ:

$ sudo service rabbitmq-server restart

Allow irma to use SSL with RabbitMQ

Note: Frontend configuration file is /config/frontend.ini Brain configuration file is /config/brain.ini Probe configuration file is /config/probe.ini

Update PORT in configuration file from 5672 to 5671. Except if you change the default port defined in the the RabbitMQ configuration template provided in the precedent paragraph.

[broker_xxxx] host= 127.0.0.1 port= 5671

Update activate_ssl switch in conﬁguration ﬁle:

[ssl_config] activate_ssl= yes ca_certs= keyfile= certfile=

Put the SSL certificates (ca_cert, key_file, cert_file) in /ssl Update ca_certs, keyfile and certfile in configuration file according to the filenames in “./ssl”

[ssl_config] activate_ssl= yes ca_certs= keyfile= certfile=

Note: If you are switching to ssl from an already running no_ssl version, please do the following on irma-brain RabbitMQ server:

56 Chapter 7. Frequently Asked Questions IRMA Documentation, Release 2.0.4

$ sudo rabbitmqctl stop_app $ sudo rabbitmqctl reset $ sudo rabbitmqctl start_app # create again the RabbitMQ vhosts, usernames and passwords: $ sudo ./extras/scripts/rabbitmq/rmq_adduser.sh probe probe mqprobe $ sudo ./extras/scripts/rabbitmq/rmq_adduser.sh brain brain mqbrain $ sudo ./extras/scripts/rabbitmq/rmq_adduser.sh frontend frontend mqfrontend

7.3 How to debug

7.3.1 Switch debug log on

Conﬁguration ﬁle for frontend, brain and probe is located by default in the config folder and is named respectively frontend.ini, brain.ini and probe.ini. To turn on debug log just add the following line:

[log] syslog=0 debug=1 and restart all related applications. To turn on SQL debug log (warning: its verbose) just add the following line:

[log] syslog=0 debug=1 sql_debug=1 and restart all related applications.

7.3.2 Debug a probe

Open a session on the probe machine and change directory to the irma-probe location. Try the run_module tool on a ﬁle to see what analyzer is detected and what is its output on a ﬁle.

$ sudo su deploy $ cd /opt/irma/irma-probe/current $ ./venv/bin/python -m extras.tools.run_module

[...] usage: run_module.py[-h][-v] {Unarchive,StaticAnalyzer,ClamAV,VirusTotal} filename [filename ...] run_module.py: error: too few arguments

Here 4 probes are automatically detected. Now try one on a ﬁle:

$ ./venv/bin/python -m extras.tools.run_module ClamAV requirements.txt {'database':{'/var/lib/clamav/bytecode.cvd':{'ctime': 1458640823.285298, 'mtime': 1458640823.069295, 'sha256':

˓→'82972e6cc5f1204829dba913cb1a0b5f8152eb73d3407f6b86cf388626cff1a1'},

7.3. How to debug 57 IRMA Documentation, Release 2.0.4

'/var/lib/clamav/daily.cvd':{'ctime': 1458640822.8932924, 'mtime': 1458640822.6692889, 'sha256':

˓→'9804c9b9aaf983f85b4f13a7053f98eb7cca5a5a88d3897d49b22182b228885f'}, '/var/lib/clamav/main.cvd':{'ctime': 1458640821.6972747, 'mtime': 1458640813.9771628, 'sha256':

˓→'4a8dfbc4c44704186ad29b5a3f8bdb6674b679cecdf83b156dd1c650129b56f2'}}, 'duration':0.0045299530029296875, 'error': None, 'name': 'Clam AntiVirus Scanner', 'platform': 'linux2', 'results': None, 'status':0, 'type': 'antivirus', 'version': '0.99'}

And check the output.

7.3.3 Debug Ansible Provisioning

To debug errors while provisioning (same goes with deployment) with following typical command:

$ ansible-playbook --private-key=~/.vagrant.d/insecure_private_key --inventory-file=.

˓→vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory -u vagrant

˓→playbooks/provisioning.yml

Example output:

TASK[Mayeu.RabbitMQ : add rabbitmq user and set privileges] ******************* [DEPRECATION WARNING]: Using bare variables is deprecated. Update your playbooks so

˓→that the environment value uses the full variable syntax('{{rabbitmq_users_definitions}}'). This feature will be removed in a future release. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. failed:[brain.irma](item={u'vhost': u'mqbrain', u'password': u'brain', u'user': u

˓→'brain'})=>{"failed": true, "item":{"password": "brain", "user": "brain", "vhost

˓→": "mqbrain"}, "module_stderr":"", "module_stdout": "Traceback (most recent call

˓→last):\r\n File \"/tmp/ansible_wKXoO5/ansible_module_rabbitmq_user.py\", line 302,

˓→in \r\n main()\r\n File \"/tmp/ansible_wKXoO5/ansible_module_rabbitmq_

˓→user.py\", line 274, in main\r\n if rabbitmq_user.get():\r\n File \"/tmp/

˓→ansible_wKXoO5/ansible_module_rabbitmq_user.py\", line 155, in get\r\n users =

˓→self._exec(['list_users'], True)\r\n File \"/tmp/ansible_wKXoO5/ansible_module_

˓→rabbitmq_user.py\", line 150, in _exec\r\n rc, out, err = self.module.run_

˓→command(cmd + args, check_rc=True)\r\n File \"/tmp/ansible_wKXoO5/ansible_modlib.

˓→zip/ansible/module_utils/basic.py\", line 1993, in run_command\r\n File \"/usr/lib/

˓→python2.7/posixpath.py\", line 261, in expanduser\r\n if not path.startswith('~

˓→'):\r\nAttributeError: 'list' object has no attribute 'startswith'\r\n", "msg":

˓→"MODULE FAILURE", "parsed": false}

You could ﬁrst increase ansible verbosity by adding -vvv option (-vvvv on windows for winrm debug), it will help is the problem is linked to arguments.

$ ansible-playbook -vvv --private-key=~/.vagrant.d/insecure_private_key --

˓→inventory-file=.vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory -u

˓→vagrant playbooks/provisioning.yml TASK[Mayeu.RabbitMQ : add rabbitmq user and set privileges] *******************

58 Chapter 7. Frequently Asked Questions IRMA Documentation, Release 2.0.4

task path: /home/alex/repo/irma-ansible/roles/Mayeu.RabbitMQ/tasks/vhost.yml:13 [DEPRECATION WARNING]: Using bare variables is deprecated. Update your playbooks

˓→so that the environment value uses the full variable syntax('{{rabbitmq_users_definitions}}'). This feature will be removed in a future release. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. <127.0.0.1> ESTABLISH SSH CONNECTION FOR USER: vagrant <127.0.0.1> SSH: EXEC ssh -C -q -o ForwardAgent=yes -o Port=2222 -o

˓→'IdentityFile="/home/alex/.vagrant.d/insecure_private_key"' -o

˓→KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-

˓→keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o

˓→ConnectTimeout=10 127.0.0.1 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.

˓→ansible/tmp/ansible-tmp-1468570550.09-211613386938202 `" && echo ansible-tmp-

˓→1468570550.09-211613386938202="` echo $HOME/.ansible/tmp/ansible-tmp-1468570550.09-

˓→211613386938202 `" ) && sleep 0'"'"'' <127.0.0.1> PUT /tmp/tmpiysJ6l TO /home/vagrant/.ansible/tmp/ansible-tmp-

˓→1468570550.09-211613386938202/rabbitmq_user <127.0.0.1> SSH: EXEC sftp -b - -C -o ForwardAgent=yes -o Port=2222 -o

˓→'IdentityFile="/home/alex/.vagrant.d/insecure_private_key"' -o

˓→KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-

˓→keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o

˓→ConnectTimeout=10 '[127.0.0.1]' <127.0.0.1> ESTABLISH SSH CONNECTION FOR USER: vagrant <127.0.0.1> SSH: EXEC ssh -C -q -o ForwardAgent=yes -o Port=2222 -o

˓→'IdentityFile="/home/alex/.vagrant.d/insecure_private_key"' -o

˓→KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-

˓→keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o

˓→ConnectTimeout=10 -tt 127.0.0.1 '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"

˓→'"'"'"'"'"'"'"'echo BECOME-SUCCESS-rbeeckncuxenewcwkayivqiwvarchlrd; LANG=fr_FR.UTF-

˓→8 LC_ALL=fr_FR.UTF-8 LC_MESSAGES=fr_FR.UTF-8 /usr/bin/python /home/vagrant/.ansible/

˓→tmp/ansible-tmp-1468570550.09-211613386938202/rabbitmq_user; rm -rf "/home/vagrant/.

˓→ansible/tmp/ansible-tmp-1468570550.09-211613386938202/" > /dev/null 2>&1'"'"'"'"'"'"

˓→'"'"' && sleep 0'"'"'' failed:[brain.irma](item={u'vhost': u'mqbrain', u'password': u'brain', u'user

˓→': u'brain'})=>{"failed": true, "invocation":{"module_name": "rabbitmq_user"},

˓→"item":{"password": "brain", "user": "brain", "vhost": "mqbrain"}, "module_stderr

˓→":"", "module_stdout": "Traceback (most recent call last):\r\n File \"/tmp/

˓→ansible_Qo3lZl/ansible_module_rabbitmq_user.py\", line 302, in \r\n

˓→main()\r\n File \"/tmp/ansible_Qo3lZl/ansible_module_rabbitmq_user.py\", line 274,

˓→in main\r\n if rabbitmq_user.get():\r\n File \"/tmp/ansible_Qo3lZl/ansible_

˓→module_rabbitmq_user.py\", line 155, in get\r\n users = self._exec(['list_users

˓→'], True)\r\n File \"/tmp/ansible_Qo3lZl/ansible_module_rabbitmq_user.py\", line

˓→150, in _exec\r\n rc, out, err = self.module.run_command(cmd + args, check_

˓→rc=True)\r\n File \"/tmp/ansible_Qo3lZl/ansible_modlib.zip/ansible/module_utils/

˓→basic.py\", line 1993, in run_command\r\n File \"/usr/lib/python2.7/posixpath.py\",

˓→ line 261, in expanduser\r\n if not path.startswith('~'):\r\nAttributeError:

˓→'list' object has no attribute 'startswith'\r\n", "msg": "MODULE FAILURE", "parsed

˓→": false}

In this particular case, verbose doesnt add much information as the problem is linked to ansible scripts. Lets go one level deeper so. Ansible output the temporary script executed on guest (highlighted in previous code block), but delete it just after execution. To further debug it we will set ansible to keep remote ﬁles and the debug session will now takes place inside the guest.

$ ANSIBLE_KEEP_REMOTE_FILES=1 ansible-playbook -vvv --private-key=~/.vagrant.d/

˓→insecure_private_key --inventory-file=.vagrant/provisioners/ansible/inventory/

˓→vagrant_ansible_inventory -u vagrant playbooks/provisioning.yml

7.3. How to debug 59 IRMA Documentation, Release 2.0.4 in debug log get the temporary ansible path to remote script:

/usr/bin/python /home/vagrant/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275/

˓→rabbitmq_user

$ vagrant@brain:~/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275$ ls rabbitmq_user

$ vagrant@brain:~/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275$ python

˓→rabbitmq_user explode Module expanded into: /home/vagrant/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275/debug_dir

$ vagrant@brain:~/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275$ ls debug_

˓→dir/ ansible ansible_module_rabbitmq_user.py args

$ vagrant@brain:~/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275$ python

˓→rabbitmq_user execute Traceback(most recent call last): File "/home/vagrant/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275/debug_

˓→dir/ansible_module_rabbitmq_user.py", line 302, in main() File "/home/vagrant/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275/debug_

˓→dir/ansible_module_rabbitmq_user.py", line 274, in main if rabbitmq_user.get(): File "/home/vagrant/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275/debug_

˓→dir/ansible_module_rabbitmq_user.py", line 155, in get users= self._exec(['list_users'], True) File "/home/vagrant/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275/debug_

˓→dir/ansible_module_rabbitmq_user.py", line 150, in _exec rc, out, err= self.module.run_command(cmd + args, check_rc=True) File "/home/vagrant/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275/debug_

˓→dir/ansible/module_utils/basic.py", line 1993, in run_command args=[ os.path.expandvars(os.path.expanduser(x)) for x in args if x is not None

˓→] File "/usr/lib/python2.7/posixpath.py", line 261, in expanduser if not path.startswith('~'): AttributeError: 'list' object has no attribute 'startswith'

You could now add debug to source ﬁles and properly understand where the problem is. In our example case, it is an ansible problem related to module_rabbitmq_user present in 2.1.0.0 see githubPR

7.4 How to migrate

Note: If you need help to connect to your box through ssh, see vagrant FAQ

This part is only useful to someone willing to manually upgrade from an older version of IRMA.

60 Chapter 7. Frequently Asked Questions IRMA Documentation, Release 2.0.4

7.4.1 Install alembic

$ sudo su deploy $ cd /opt/irma/irma-frontend/current $ ./venv/bin/pip install alembic $ export PYTHONPATH=.:$PYTHONPATH $ alembic history

430a70c8aa21 -> eb7141efd75a(head), version1.3.0 2cc69d5c53eb -> 430a70c8aa21, version1.2.1 -> 2cc69d5c53eb, DB revision creation

7.4.2 from 1.2.1 to 1.3.0

Fix nginx conﬁguration

Introducing multiversion API means python code should receive the api version parameter. in ﬁle /etc/nginx/sites- available/irma-frontend.conf replace: rewrite ^/api/v1/(.+)/$1 break; by: rewrite ^/api/(.+)/$1 break; and restart nginx

Migrate Database

First you should tell alembic you are at version 1.2.1:

$ ./venv/bin/alembic stamp 430a70c8aa21 then upgrade model and data:

$ ./venv/bin/alembic upgrade head

Regenerate IHM to regenerate IHM do the following:

$ sudo su deploy $ cd /opt/irma/irma-frontend/current/web $ ./node_modules/.bin/bower update $ ./node_modules/.bin/gulp dist

Its done.

7.5 API documentation

There is a dynamic documentation for IRMA API available on your instance

7.5. API documentation 61 IRMA Documentation, Release 2.0.4

It allows you to read documentation but also try request and see server response. Give it a try.

You could see detailed information about one speciﬁc API route:

and by clicking on the Try it button, see the server response:

62 Chapter 7. Frequently Asked Questions IRMA Documentation, Release 2.0.4

7.6 Connect to a vagrant box through ssh

If you don’t already have it download vagrant insecure_private_key

$ wget https://raw.githubusercontent.com/mitchellh/vagrant/master/keys/vagrant -O

˓→insecure_private_key

Then change rights on the key otherwise ssh will complains and connect to your vagrant box

$ chmod 700 insecure_private_key $ ssh [email protected] -i insecure_private_key

7.7 Enable SSL using OpenSSL in ansible scripts

If you want to activate SSL on the frontend server, you’ll need: • modify frontend_openssl variables in group_vars/frontend: frontend_openssl: True # Default is false frontend_openssl_dh_param: # put the DH file locations frontend_openssl_certificates:[] # an array of files {source, destination} # to copy to the server

• Uncomment (and customize) the nginx_sites variable in the group_vars/frontend, a commented example is available. Then, provision or re-provision your infrastructure. Ansible will only change ﬁle related to OpenSSL and Nginx conﬁgurations.

7.6. Connect to a vagrant box through ssh 63 IRMA Documentation, Release 2.0.4

7.8 Speed up your Vagrant VMs

Install this softwares: • vagrant-cachier (more info on vagrant-cachier)

$ vagrant plugin install vagrant-cachier

• vagrant-vbguest (more info on vagrant-vbguest)

$ vagrant plugin install vagrant-vbguest

64 Chapter 7. Frequently Asked Questions CHAPTER 8

Resources

• Project website • IRC (irc.freenode.net, #qb_irma) • Twitter (@qb_irma)

65 IRMA Documentation, Release 2.0.4

66 Chapter 8. Resources CHAPTER 9

Screenshots

9.1 Command Line Interface

A sample script can be found in frontend repository. Add your own frontend address before testing it.

67 IRMA Documentation, Release 2.0.4

9.2 Web Interface

Some screenshots of the irma user interface shipped with frontend package.

68 Chapter 9. Screenshots IRMA Documentation, Release 2.0.4

9.2. Web Interface 69 IRMA Documentation, Release 2.0.4

70 Chapter 9. Screenshots IRMA Documentation, Release 2.0.4

9.2. Web Interface 71 IRMA Documentation, Release 2.0.4

72 Chapter 9. Screenshots