Squid Proxy Server 3.1 Beginner's Guide Improve the performance of your network using the caching and access control capabilities of Squid Kulbir Saini yl open source distilled [i&BSttKte,P- it, I community experience PUBLISHING BIRMINGHAM - MUMBAI Table of Contents Preface 1 Chapter 1: Getting Started with Squid Z Proxy server 7 Reverse proxy 9 Getting Squid 9 Time for action - identifying the right version 10 Methods of obtaining Squid 11 Using source archives 11 Time for action - downloading Squid 11 Obtaining the latest source code from Bazaar VCS 12 Time for action - using Bazaar to obtain source code 13 Using binary packages 14 Installing Squid 14 Installing Squid from source code 14 Compiling Squid 14 Uncompressing the source archive 15 Configure or system check 15 Time for action - running the configure command 25 Time for action - compiling the source 26 Time for action - installing Squid 27 Time for action - exploring Squid files 27 Installing Squid from binary packages 29 Fedora, CentOS or Red Hat 30 Debian or Ubuntu 30 FreeBSD 30 OpenBSD or NetBSD 30 Dragonfly BSD 30 Gentoo 30 Arch Linux 31 Summary 32 Table of Contents Chapter 2: Configuring Squid 33 Quick start 34 Syntax of the configuration file 34 Types of directives 35 HTTP port 37 Time for action - setting the HTTP port 37 Access control lists 38 Time for action - constructing simple ACLs 39 Controlling access to the proxy server 40 HTTP access control 40 Time for action - combining ACLs and HTTP access 41 HTTP reply access 42 ICP access 43 HTCP access 43 HTCP CLR access 43 Miss access 43 Ident lookup access 43 Cache peers or neighbors 44 Declaring cache peers 44 Time for action-adding a cache peer 44 Quickly restricting access to domains using peers 45 Advanced control on access using peers 46 Caching web documents 46 Using main memory (RAM) for caching 46 In-transit objects or current requests 47 Hot or popular objects 47 Negatively cached objects 47 Specifying cache space in RAM 47 Time for action - specifying space for memory caching 48 Maximum object size in memory 48 Memory cache mode 49 Using hard disks for caching 49 Specifying the storage space 49 Time for action - creating a cache directory 51 Configuring the number of sub directories 52 Time for action - adding a cache directory 52 Cache directory selection 53 Cache object size limits 53 Setting limits on object replacement 54 Cache replacement policies 54 Least recently used (LRU) 54 Greedy dual size frequency (GDSF) 54 Least frequently used with dynamic aging (LFUDA) 55 Table of Contents Tuning Squid for enhanced caching 55 Selective caching 55 Time for action - preventing the caching of local content 55 Refresh patterns for cached objects 56 Time for action - calculating the freshness of cached objects 57 Options for refresh pattern 58 Aborting the partial retrievals 60 Caching the failed requests 61 Playing around with HTTP headers 61 Controlling HTTP headers in requests 61 Controlling HTTP headers in responses 62 Replacing the contents of HTTP headers 62 DNS server configuration 62 Specifying the DNS program path 63 Controlling the number of DNS client processes 63 Setting the DNS name servers 63 Time for action - adding DNS name servers 64 Setting the hosts file 64 Default domain name for requests 64 Timeout for DNS queries 64 Caching the DNS responses 65 Setting the size of the DNS cache 65 Logging 66 Log formats 66 Log file rotation or log file backups 66 Log access 66 Buffered logs 66 Strip query terms 67 URL rewriters and redirectors 67 Other configuration directives 67 Setting the effective user for running Squid 68 Configuring hostnames for the proxy server 68 Hostname visible to everyone 68 Unique hostname for the server 68 Controlling the request forwarding 68 Always direct 69 Never direct 69 Hierarchy stoplist 69 Broken posts 70 TCP outgoing address 70 Table of Contents PID filename 71 Client netmask 71 Summary 73 Chapter 3: Running Squid 75 Command line options 75 Getting a list of available options 76 Time for action - listing the options 77 Getting information about our Squid installation 78 Time for action - finding out the Squid version 78 Creating cache or swap directories 78 Time for action - creating cache directories 78 Using a different configuration file 79 Getting verbose output 79 Time for action - debugging output in the console 80 Full debugging output on the terminal 81 Running as a normal process 82 Parsing the Squid configuration file for errors or warnings 82 Time for action - testing our configuration file 82 Sending various signals to a running Squid process 83 Reloading a new configuration file in a running process 83 Shutting down the Squid process 84 Interrupting or killing a running Squid process 84 Checking the status of a running Squid process 84 Sending a running process in to debug mode 85 Rotating the log files 85 Forcing the storage metadata to rebuild 86 Double checking swap during rebuild 86 Automatically starting Squid at system startup 87 Adding Squid command to /etc/rc.local file 87 Adding init script 87 Time for action - adding the init script 87 Summary 89 Chapter 4: Getting Started with Squid's Powerful ACLs and Access Rules 91 Access control lists 92 Fast and slow ACL types 92 Source and destination IP address 92 Time for action - constructing ACL lists using IP addresses 93 Time for action - using a range of IP addresses to build ACL lists 94 Source and destination domain names 96 Time for action - constructing ACL lists using domain names 97 Destination port 98 Table of Contents Time for action - building ACL lists using destination ports 99 HTTP methods 101 Identifying requests using the request protocol 102 Time for action - using a request protocol to construct access rules 102 Time-based ACLs 103 URL and URL path-based identification 104 Matching client usernames 105 Proxy authentication 106 Time for action - enforcing proxy authentication 107 User limits 108 Identification based on various HTTP headers 109 HTTP reply status 111 Identifying random requests 112 Access list rules 112 Access to HTTP protocol 112 Access to other ports 114 Enforcing limited access to neighbors 115 Time for action - denying miss_access to neighbors 115 Requesting neighbor proxy servers 116 Forwarding requests to remote servers 117 Ident lookup access 117 Controlled caching of web documents 118 URL rewrite access 118 HTTP header access 119 Custom error pages 119 Maximum size of the reply body 120 Logging requests selectively 120 Mixing ACL lists and rules - example scenarios 121 Handling caching of local content 121 Time for action - avoiding caching of local content 121 Denying access from external networks 122 Denying access to selective clients 122 Blocking the download of video content 123 Time for action - blocking video content 123 Special access for certain clients 123 Time for action - writing rules for special access 124 Limited access during working hours 124 Allowing some clients to connect to special ports 125 Testing access control with squidclient 126 Table of Contents_ Time for action - testing our access control example with squidclient 128 Time for action - testing a complex access control 129 Summary 132 Chapter 5: Understanding Log Files and Log Formats 133 Log messages 134 Cache log or debug log 134 Time for action - understanding the cache log 134 Access log 137 Understanding the access log 137 Time for action - understanding the access log messages 137 Access log syntax 139 Time for action - analyzing a syntax to specify access log 139 Log format 140 Time for action - learning log format and format codes 140 Log formats provided by Squid 142 Time for action - customizing the access log with a new log format 142 Selective logging of requests 143 Time for action - using accessjog to control logging of requests 144 Referer log 144 Time for action - enabling the referer log 145 Time for action - translating the referer logs to a human-readable format 145 User agent log 146 Time for action - enabling user agent logging 147 Emulating HTTP server-like logs 147 Time for action - enabling HTTP server log emulation 147 Log file rotation 148 Other log related features 148 Cache store log 149 Summary 150 Chapter 6: Managing Squid and Monitoring Traffic 151 Cache manager 151 Installing the Apache Web server 152 Time for action - installing Apache Web server 152 Configuring Apache for providing the cache manager web interface 152 Time for action - configuring Apache to use cachemgr.cgi 153 Accessing the cache manager web interface 153 Configuring Squid 154 Log in to cache manger 154 General Runtime Information 156 IP Cache Stats and Contents 157 FQDN Cache Statistics 158 Table of Contents HTTP Header Statistics 159 Traffic and Resource Counters 160 Request Forwarding Statistics 161 Cache Client List 162 Memory Utilization 163 Internal DNS Statistics 164 Log file analyzers 165 Calamaris 165 Installing Calamaris 166 Time for action - installing Calamaris 166 Using Calamaris to generate statistics 167 Time for action - generating stats in plain text format 167 Time for action - generating graphical reports with Calamaris 168 Summary 171 Chapter 7: Protecting your Squid Proxy Server with Authentication 173 HTTP authentication 174 Basic authentication 174 Time for action - exploring Basic authentication 174 Database authentication 176 Configuring database authentication 177 NCSA authentication 178 Time for action - configuring NCSA authentication 178 NIS authentication 179 LDAP authentication 179 SMB authentication 179 PAM authentication 180 Time for