Maintenance Branches
Total Page:16
File Type:pdf, Size:1020Kb
intelmq Release 3.0.2 IntelMQ Community Sep 10, 2021 CONTENTS 1 User guide 3 2 Connecting with other systems 185 3 Getting involved 199 4 Licence 243 5 Funded by 245 6 Indices and tables 247 Index 249 i ii intelmq, Release 3.0.2 IntelMQ is a solution for IT security teams (CERTs & CSIRTs, SOCs abuse departments, etc.) for collecting and processing security feeds (such as log files) using a message queuing protocol. It’s a community driven initiative called IHAP (Incident Handling Automation Project) which was conceptually designed by European CERTs/CSIRTs during several InfoSec events. Its main goal is to give to incident responders an easy way to collect & process threat intelligence thus improving the incident handling processes of CERTs. CONTENTS 1 intelmq, Release 3.0.2 2 CONTENTS CHAPTER ONE USER GUIDE 1.1 Introduction 1.1.1 About IntelMQ is a solution for IT security teams (CERTs & CSIRTs, SOCs abuse departments, etc.) for collecting and processing security feeds (such as log files) using a message queuing protocol. It’s a community driven initiative called IHAP (Incident Handling Automation Project) which was conceptually designed by European CERTs/CSIRTs during several InfoSec events. Its main goal is to give to incident responders an easy way to collect & process threat intelligence thus improving the incident handling processes of CERTs. Incident Handling Automation Project • URL: <http://www.enisa.europa.eu/activities/cert/support/incident-handling-automation> • Mailing-list: <[email protected]> Several pieces of software are evolved around IntelMQ. For an overview, look at the IntelMQ Ecosystem. IntelMQ can be used for - automated incident handling - situational awareness - automated notifications - as data collector for other tools - etc. IntelMQ’s design was influenced by AbuseHelper however it was re-written from scratch and aims at: • Reducing the complexity of system administration • Reducing the complexity of writing new bots for new data feeds • Reducing the probability of events lost in all process with persistence functionality (even system crash) • Use and improve the existing Data Harmonization Ontology • Use JSON format for all messages • Provide easy way to store data into Log Collectors like ElasticSearch, Splunk, databases (such as PostgreSQL) • Provide easy way to create your own black-lists • Provide easy communication with other systems via HTTP RESTful API It follows the following basic meta-guidelines: • Don’t break simplicity - KISS • Keep it open source - forever • Strive for perfection while keeping a deadline • Reduce complexity/avoid feature bloat • Embrace unit testing 3 intelmq, Release 3.0.2 • Code readability: test with inexperienced programmers • Communicate clearly 1.1.2 Usage Various approaches of installing intelmq are described in Installation. The Configuration and Management gives an overview how a intelmq installation is set up and how to configure and maintain the setup. There is also a list of available Feeds as well as a detailed description of the different Bots intelmq brings with it. If you know additional feeds and how to parse them, please contribute your code or your configuration (by issues or the mailing lists). For support questions please use the IntelMQ Users Mailinglist. IntelMQ Manager Check out this graphical tool to easily manage an IntelMQ system. 1.1.3 Contribute • Subscribe to the IntelMQ Developers Mailinglist • IRC: server: irc.freenode.net, channel: #intelmq • Via GitHub issues • Via Pull requests (please have a look at the Developers Guide first) 1.2 Hardware Requirements Do you ask yourself how much RAM do you need to give your new IntelMQ virtual machine? The honest answer is simple and pointless: It depends ;) Contents • Hardware Requirements – IntelMQ and the messaging queue (broker) ∗ Disk space ∗ Background on memory – Additional components ∗ EventDB ∗ IntelMQ cb mailgen 4 Chapter 1. User guide intelmq, Release 3.0.2 1.2.1 IntelMQ and the messaging queue (broker) IntelMQ uses a messaging queue to move the messages between the bots. All bot instances can only process one message at a time, therefore all other messages need to wait in the queue. As not all bots are equally fast, the messages will naturally “queue up” before the slower ones. Further, parsers produce many events with just one message (the report) as input. The following estimations assume Redis as messaging broker which is the default for IntelMQ. When RabbitMQ is used, the required resources will differ, and RabbitMQ can handle system overload and therefore a shortage of memory. As Redis stores all data in memory, the data which is processed at any point in time must fit there, including overheads. Please note that IntelMQ does neither store nor cache any input data. These estimates therefore only relate to the processing step, not the storage. For a minimal system, these requirements suffice: • 4 GB of RAM • 2 CPUs • 10 GB disk size Depending on your data input, you will need the twentiethfold of the input data size as memory for processing. When using Redis persistence, you will additionally need twice as much memory for Redis. Disk space Disk space is only relevant if you save your data to a file, which is not recommended for production setups, andonly useful for testing and evaluation. Do not forget to rotate your logs or use syslog, especially if you use the logging level “DEBUG”. logrotate is in use by default for all installation with deb/rpm packages. When other means of installation are used (pip, manual), configure log rotation manually. See Logging. Background on memory For experimentation, we used multiple Shadowserver Poodle reports for demonstration purpose, totaling in 120 MB of data. All numbers are estimates and are rounded. In memory, the report data requires 160 MB. After parsing, the memory usage increases to 850 MB in total, as every data line is stored as JSON, with additional information plus the original data encoded in Base 64. The further processing steps depend on the configuration, but you can estimate that caches (for lookups and deduplication) and other added information cause an additional size increase of about 2x. Once a dataset finished processing in IntelMQ, it is no longer stored in memory. Therefore, the memory is only needed to catch high load. The above numbers result in a factor of 14 for input data size vs. memory required by Redis. Assuming some overhead and memory for the bots’ processes, a factor of 20 seems sensible. To reduce the amount of required memory and disk size, you can optionally remove the raw data field, see Removing raw data for higher performance and less space usage in the FAQ. 1.2. Hardware Requirements 5 intelmq, Release 3.0.2 1.2.2 Additional components If some of the optional components of the IntelMQ Ecosystem are in use, they can add additional hardware requirements. Those components do not add relevant requirements: • IntelMQ API: It is just an API for intelmqctl. • IntelMQ Manager: Only contains static files served by the webserver. • IntelMQ Webinput CSV: Just a webinterface to insert data. Requires the amount of processed data to fit in memory, see above. • Stats Portal: The aggregation step and Graphana require some resources, but no exact numbers are known. • Malware Name Mapping • Docker: The docker layer adds only minimal hardware requirements. EventDB When storing data in databases (such as MongoDB, PostgreSQL, ElasticSearch), it is recommended to do this on separate machines for operational reasons. Using a different machine results in a separation of stream processing to data storage and allows for a specialized system optimization for both use-cases. IntelMQ cb mailgen While the Fody backend and frontend do not have significant requirements, the RIPE import tool of the certbund-contact requires about 8 GB of memory as of March 2021. 1.3 Installation Contents • Installation – Requirements – Install Dependencies ∗ Ubuntu / Debian ∗ CentOS 7 / RHEL 7 ∗ CentOS 8 ∗ openSUSE 15.2 / 15.3 ∗ Docker (beta) – Installation ∗ Native Packages ∗ PyPi ∗ Docker with docker-compose (recommended) ∗ Docker without docker-compose 6 Chapter 1. User guide intelmq, Release 3.0.2 ∗ Additional Information Please report any errors you encounter at https://github.com/certtools/intelmq/issues For upgrade instructions, see Upgrade instructions. For setting up a development environment see the Developers Guide section Development Environment. For testing pre-releases see also the Developers Guide section Testing Pre- releases. 1.3.1 Requirements The following instructions assume the following requirements. Python versions >= 3.6 are supported. Supported and recommended operating systems are: • CentOS 7 and 8 • Debian 10 Buster and 11 Bullseye • openSUSE Leap 15.2, 15.13 • Ubuntu: 18.04, 20.04 • Docker Engine: 18.x and higher Other distributions which are (most probably) supported include RHEL, Fedora, openSUSE Tumbleweed and FreeBSD 12. A short guide on hardware requirements can be found on the page Hardware Requirements. 1.3.2 Install Dependencies If you are using native packages, you skip this section as all dependencies are installed automatically. Ubuntu / Debian apt install python3-pip python3-dnspython python3-psutil python3-redis python3-requests ,!python3-termstyle python3-tz python3-dateutil apt install redis-server Optional dependencies: apt install bash-completion jq apt install python3-pymongo python3-psycopg2 CentOS 7 / RHEL 7 yum install epel-release yum install python36 python36-devel python36-requests yum install gcc gcc-c++ yum install redis 1.3. Installation 7 intelmq, Release 3.0.2 CentOS 8 dnf install epel-release dnf install python3-dateutil python3-dns python3-pip python3-psutil python3-pytz python3- ,!redis python3-requests redis Optional dependencies: dnf install bash-completion jq dnf install python3-psycopg2 python3-pymongo openSUSE 15.2 / 15.3 zypper install python3-dateutil python3-dnspython python3-psutil python3-pytz python3- ,!redis python3-requests python3-python-termstyle zypper install redis Optional dependencies: zypper in bash-completion jq zypper in python3-psycopg2 python3-pymongo Docker (beta) ATTENTION Currently you can’t manage your botnet via intelmqctl documentation.