IBM Front cover Secure Your E-mail Server on IBM Eserver i5 with Linux Understanding security issues for network and e-mail server Linux open source solutions to secure your e-mail server Linux-based ISV solutions to secure your e-mail server Yessong Johng Alex Robar Colin McNaught Senthil Kumar ibm.com/redbooks Redpaper International Technical Support Organization Secure Your E-mail Server on IBM Eserver i5 with Linux October 2005 Note: Before using this information and the product it supports, read the information in “Notices” on page vii. First Edition (October 2005) This edition applies to IBM i5/OS V5R3, SUSE LINUX Enterprise Server 9, and Red Hat Enterprise Linux AS Version 4. © Copyright International Business Machines Corporation 2005. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Contents Notices . vii Trademarks . viii Preface . ix The team that wrote this Redpaper . ix Become a published author . xi Comments welcome. xi Part 1. Open Source Solutions for Network Security . 1 Chapter 1. Understanding and planning e-mail server security. 3 1.1 Concepts: Securing e-mail servers . 4 1.1.1 Linux-based firewall . 6 1.1.2 E-mail security . 7 1.2 Scenarios: Securing e-mail server . 8 1.2.1 Open source protection and open source mail delivery . 8 1.2.2 Open source protection and Domino . 9 1.2.3 ISV protection, open source filtering, and open source mail delivery . 10 1.3 Planning: Securing e-mail server . 11 1.3.1 OSS versus ISV solutions for network security mechanisms . 11 1.3.2 Direct I/O for firewall . 11 1.3.3 Choice of e-mail server . 12 1.3.4 Support contracts . 12 1.3.5 De-militarized zone . 13 1.3.6 Planning worksheet. 13 1.4 Types of attacks and protection mechanisms . 15 Chapter 2. Linux installation . 17 2.1 Linux installation overview . 18 2.1.1 Required or helpful tools . 18 2.1.2 Installation notes . 18 2.2 Setting up the partitions . 20 2.2.1 Creating a logical partition using the HMC . 20 2.2.2 Set up the i5/OS partition virtual I/O . 40 2.2.3 Working with network servers . 44 2.3 Installing Linux . 50 2.3.1 Installing SLES9 . 50 2.3.2 Installing RHEL4 . 80 Chapter 3. Locking down the Linux firewall partition . 107 3.1 Hardening Linux . 108 3.1.1 Bastille Linux. 108 3.1.2 Removing unnecessary servers . 156 3.1.3 Altering insecure defaults . 156 3.2 iptables rules . 159 3.2.1 Understanding iptables . 160 3.2.2 Initial iptables setup . 162 3.3 grsecurity kernel patch . 166 3.4 Security-Enhanced Linux (SELinux) . 171 © Copyright IBM Corp. 2005. All rights reserved. iii 3.5 Snort . 171 3.5.1 Installing libpcap 0.9.0-096 . 171 3.5.2 Installing Perl Compatible Regular Expressions (PCRE) 5.0 . 172 3.5.3 Installing Snort 2.3.1 . 172 3.5.4 Configuring Snort . 173 3.6 Rootkit hunter . 175 Chapter 4. E-mail Security tools installation and configuration. 177 4.1 Postfix . 178 4.1.1 Preparing to install Postfix . 178 4.1.2 Updating Postfix . 181 4.1.3 Postfix configuration files . 183 4.1.4 Configuring Postfix . 184 4.2 qmail . 185 4.2.1 Overview of qmail installation . 186 4.2.2 Preparing to install qmail. 186 4.2.3 Installing qmail . 188 4.2.4 Configuring qmail . 195 4.3 Clam AntiVirus . 202 4.3.1 Installing Clam Antivirus . 203 4.3.2 Configuring Clam AntiVirus for Postfix . 208 4.3.3 Configuring Clam Antivirus for qmail. 211 4.3.4 Adding Clam to system boot . 213 4.4 SpamAssassin . 213 4.4.1 Installing SpamAssassin . 214 4.4.2 Overview: Configuration of SpamAssassin. 219 4.4.3 Configuring SpamAssassin for Postfix . 222 4.4.4 Configuring SpamAssassin for qmail . 223 4.4.5 Adding SpamAssassin process.