SYSADMIN Security Lessons: Rsyslog Making sure your logs work Secure Logging Might as well do it properly – rsyslog. BY KURT SEIFRIED drop-in replacement for sysklogd, and if you’re running a recent Debian, Ubuntu, or Fedora, it’s the default logging pack- was looking at my server backups syslog listen to the network and accept age. The bad news is that you will most the other day and suddenly realized syslog messages from remote hosts. likely need to upgrade it if you want the II had no backups of the logfiles – Problem solved, right? more advanced features. (Fedora ships you know, all the stuff in /var/log/ that Not really. The main problem is that version 2.0.6; the latest stable release is you ignore until something breaks. Also, syslog uses UDP for message transport, 5.2.0.) I realized I had no idea whether logging and UDP doesn’t guarantee delivery. So, For those of you on a different flavor was actually working, so maybe I should for example, if you accidentally firewall of Linux, chances are your vendor ships go about fixing this. syslog, the messages could simply be rsyslog, and you can simply install it dropped without any warning. If you with yum or whatever package manager Why Logging Locally Isn’t want to send logs across networks you you use (e.g., emerge -va rsyslog on Gen- Secure or Reliable don’t trust (e.g., the Internet), you can’t too Linux). Downloading the source and By definition, attackers who gain access be sure that an attacker hasn’t injected compiling it is not hard; however, you to your system can monkey around with log messages, because spoofing UDP is should make sure that you add support the logfiles, modifying them or simply much easier than TCP in that no three- for: deleting them altogether if they’re not way handshake or sequence ID is worried about being quiet. This can leave needed. Also, if an attacker is really on ./configure --enable-gnutls ^^ you with no log of how they broke in or the ball, he or she can alter messages in --enable-mysql what they did, or even that you have suf- transit, without being detected (the mes- make fered a break in. Also, your logging can sages are not protected in any way). make install simply break and stop writing logs, and, unless you’re checking those logs, you’ll Why Not SSL Wrap It? Of course, you will need the gnutls-devel never notice. Trick question. Applications like Stunnel and mysql-devel packages installed on This old problem has a simple solu- only play nicely with TCP-based ser- your system for this to work. Once com- tion: You just log to a remote host that is vices. If you’re utterly determined not to piled, you need to install rsyslog. This designed to hold the logfiles securely. run rsyslog for some reason, I suppose can be tricky because you will need to The syslog -r option will specifically let you could set up a VPN (e.g., remove the existing rsyslog or sysklogd OpenVPN), but that still won’t guaran- package forcibly and ignore the resulting tee message delivery because dependency complaints (initscripts, syslog will still be using vixie-cron, cronie, etc.) that you will re- UDP. ceive. Future updates of these packages could also be an issue because a depen- Rsyslog to the dency is missing. Unfortunately, most Rescue vendors are shipping very old versions The good news of rsyslog. Once you have rsyslog com- is that rsys- piled and installed, you will need to con- log [1] is a figure it securely. Guaranteed Delivery The first thing rsyslog does right is use TCP, which is a much more reli- able transport than UDP. The second thing rsyslog does is pro- vide application-level acknowl- edgment of messages. Thus, it pro- vides a guarantee of message deliv- ery, so even strange TCP errors won’t cause messages to disappear silently. To do this, it supports RELP (Reliable Event 54 ISSUE 115 JUNE 2010 Security Lessons: Rsyslog SYSADMIN Logging Protocol); configuration is triv- communications. If you look in the gtls Because rsyslog is a module and sup- ial on a client: Network Stream Driver documentation ports multiple inputs (and outputs), you file, you will see that the mode you want can easily run it with support for multi- *.* :omrelp:10.1.2.3:2514 to use is x509/name. This setting will ple client types (UDP, TCP, RELP, etc.). cause rsyslog to validate the certificate So, the first goal of ensuring that remote and the name before allowing the client I’m Stuck with an Older logging actually results in remote logging to communicate. Also, you should con- Rsyslog taking place is accomplished. Another figure this on the clients to ensure that If you are truly stuck with an older rsys- advantage is that a central rsyslog server an attacker can’t execute a man-in-the- log, the good news is that you can at will correctly report the origin of a mes- middle attack and impersonate a server least use Stunnel to SSL wrap it. Red Hat sage even if the sender is behind a NAT (and thus harvest potentially sensitive has a knowledge base article with de- machine with other rsyslog clients (in log messages). tailed instructions [4]. The process is other words, you will be able to tell Note that support for the x509/name pretty much like wrapping any other ser- them apart, which doesn’t work so well configuration directive only appeared in vice: You set up Stunnel on the server to with other logging packages). version 3.19.4 and later, so you’ll need accept connections, and on the client, to update Fedora (rsyslog 2.0.6), Debian you connect to the server and configure Secure Delivery (rsyslog 3.18.6), and Ubuntu 9.0.4 (rsys- a local port that is forwarded to the Secure delivery is essential; otherwise, log 3.18.6), but not Ubuntu 9.10 (rsyslog server. Rsyslog on the client machine an attacker can modify messages in tran- 4.2.0), to name a few versions of Linux. connects to the local port that sends the sit or inject fake messages and cause all data off to the server. sorts of problems (like hard disk write Off-Peak Message Delivery error warnings every day at 3am that Real-time remote logging has one prob- In Conclusion cause your pager to go off). Also, you lem, which is that network traffic will be The rsyslog package is a lot more reliable don’t want an attacker eavesdropping on steady and in some cases quite heavy and secure than syslog or sysklogd. Ad- messages. For example, if a user acciden- (someone scans servers for weak ac- ditionally, you can log to a database, tally enters her password instead of her counts, a sudden slew of errors, etc.). If send SNMP alerts, and browse events username, that password will be logged you have branch offices or remote loca- with a nice web interface [5] [6]. n (and sent to whatever remote systems tions that you want to tie into a central your logs go to), potentially exposing the logging system, you could find yourself INFO password to an attacker with access to taking up a significant portion of upload [1] Rsyslog: http://​­www.​­rsyslog.​­com/ your network traffic. To address this, bandwidth during business hours (not [2] Encrypting Syslog Traffic with TLS rsyslog supports TLS (Transport Layer all the world has high-speed Internet like (SSL): http://​­www.​­rsyslog.​­com/ Security) natively; all you need to do is Japan and Norway yet). Fortunately, doc-rsyslog_tls.​­html edit rsyslog.conf [2]. rsyslog addresses this with off-peak mes- [3] Delivery during off-peak hours: This brings up about the only flaw I sage delivery (Listing 1). http://​­wiki.​­rsyslog.​­com/​­index.​­php/ can find with rsyslog: the documenta- This will send log data to 10.1.2.3 be- OffPeakHours tion. Generally speaking, the documen- tween 10pm and 4am; otherwise, it will [4] Wrapping rsyslog with Stunnel: tation is good, but the examples are not spool the logfile locally for later trans- http://​­kbase.​­redhat.​­com/​­faq/​­docs/ always the best. In this case, the primary missions. The ActionQueueSaveOnShut- DOC-18564 example shows the $InputTCPServer- down is important; without it, you will [5] “The sys admin’s daily grind: RSys- StreamDriverAuthMode variable set to lose data if you shut down rsyslog, be- log” by Charly Kühnast, "anon", meaning no client authentica- cause it will not write data in memory to Linux Magazine June 2008, pg. 63, tion is taking place, which sort of defeats the spool. The other benefit of off-peak http://​­www.​­linux-magazine.​­com/ the whole point of using TLS to secure message delivery is that you can stagger Issues/​­2008/​­91/​­WHERE-TO-NEXT delivery times for servers so your central [6] “The sys admin’s daily grind: ph- Listing 1: Off-Peak logging server doesn’t get flooded by cli- pLogCon” by Charly Kühnast, Linux Message Delivery ents [3]. Magazine, July 2008, pg. 69, http:// www.​­linux-magazine.​­com/​­Issues/ 01 # reliably transmit messages Transitioning to Rsyslog 2008/​­92/​­MILKING-MACHINE-2.​­0 02 # during off-peak hours (10p to 4a) What if you want to start using rsyslog, 03 $ModLoad omrelp but you have older syslog clients that Kurt Seifried is an 04 $WorkDirectory /rsyslog/work # Information Secu- where to place the spool files? you can’t yet upgrade? That’s easy; run a rity Consultant spe- 05 $ActionQueueType LinkedList central rsyslog server with support for UDP messages and upgrade the clients cializing in Linux 06 $ActionQueueDequeueTimeBegin 22 when you can.