USENIX Association
Proceedings of the 17th Large Installation Systems Administration Conference
San Diego, CA, USA October 26Ð31, 2003
THE ADVANCED COMPUTING SYSTEMS ASSOCIATION
© 2003 by The USENIX Association All Rights Reserved For more information about the USENIX Association: Phone: 1 510 528 8649 FAX: 1 510 548 5738 Email: [email protected] WWW: http://www.usenix.org Rights to individual papers remain with the author or the author's employer. Permission is granted for noncommercial reproduction of the work for educational or research purposes. This copyright notice must be included in the reproduced paper. USENIX acknowledges all trademarks herein. A Secure and Transparent Firewall Web Proxy Roger Crandell, James Clifford, and Alexander Kent – Los Alamos National Laboratory
ABSTRACT The LANL transparent web proxy lets authorized external users originating from the Internet to securely access internal intranet web content and applications normally blocked by a firewall. Unauthenticated access is still, of course, denied. The proxy is transparent in that no changes to browsers, user interaction, or intranet web servers are necessary. The proxy, a few thousand lines of C running on Linux, has been operating within Los Alamos National Laboratory’s firewall since 1999.
Introduction This paper focuses on how these needs were addressed through the design of an innovative and Our goal was to provide a transparent firewall user-friendly proxy using available technologies and web proxy, a proxy allowing authorized and secure open source software. access to the many firewall-protected intranet web servers at Los Alamos National Laboratory (LANL). Goals Our offsite users required access to internal news, phone directories, web email, the online library, and Controlled access to intranet web servers from administrative services. Additionally, they need to the Internet was the goal. VPN access into the intranet download documentation and software. External users was available, but only provided access to users who often find themselves in the situation where they are had the proper VPN client software installed. In addi- forced to use available computers at the sites they are tion, a VPN solution was seen as overkill for those visiting. Non-standard access methods, including users requiring just web-based intranet access. Clearly, browser configuration modification or installing vir- there was an accessibility gap needing to be filled. tual private networking (VPN) software was often What was needed was an access mechanism that impractical or undesirable. At the same time, it was worked with any browser, anywhere. We desired sim- very important to maintain the security of our intranet plified access while also ensuring client transparency, servers and their data. Only requests from authorized encrypting all data transmitted over the Internet, pro- users would be forwarded and all information sent viding user-based authentication, and requiring no outside the firewall had to be encrypted. We wanted modifications to existing intranet web servers. The the web proxy to function within the framework of the authentication method must continue to work in the existing firewall. same manner as the existing firewall. A web single We considered several approaches. sign on was determined to be a requirement. Once an • We could make exclusive use of virtual private individual authenticated to the web firewall proxy, networking to access internal web servers. they would not need to authenticate again to internal • A proxy could be designed to work only for a web servers. The authentication on the web proxy single web server, making the design easier. would provide authentication credentials necessary to • Selected web services could be relocated out- access all internal servers. side the firewall. Design • Rules to allow direct access through the fire- wall to internal web servers could be allowed. Meeting the transparency requirement was our These rules could be permanent or dynamic in first task. Accomplishing this requires all inbound nature. Dynamic rules would be installed auto- intranet web requests to traverse a path allowing eval- matically following authentication and removed uation of the connection and forwarding as necessary. after a specific time period. Rules would be This evaluation could be done with a broadcast net- based upon external IP address and internal work firewall DMZ where all the incoming HTTP web server IP address. requests would be visible. The proxy could then inter- • We could not allow external access to our inter- cept the requests from the broadcast medium. Another nal web servers. option is to use a stateful Linux firewall [14] which • A transparent web proxy could be provided. would also act as the proxy. However, we chose a We concluded a transparent web proxy would meet all third option: using a router with policy routing [18]. our requirements. The next question was how to The policy routing rules route packets based upon des- implement it in a secure fashion while still providing tination port numbers and sends all HTTP (port 80) simple and reliable operation. and HTTPS (port 443) requests to the proxy server.
2003 LISA XVII – October 26-31, 2003 – San Diego, CA 23 A Secure and Transparent Firewall Web Proxy Crandell, Clifford, & Kent
The standalone proxy server makes use of the Linux proxy. The certificate has the form *.lanl.gov, Los operating system with netfilter [14] and iptables rule Alamos’ domain name. This simple redirection from set. Through the insertion of iptables rules such as: HTTP to HTTPS guarantees all data requests from our iptables -t nat -A PREROUTING \ external users are encrypted. -ieth0 -p tcp -d INTERNAL_NET \ Once the port 80 to 443 redirect has been issued, -dport 80 -j REDIRECT -to-ports 80 knowledge of the original port requested is lost. All iptables -t nat -A PREROUTING \ web requests arrive at the proxy on port 443, even the -ieth0 -p tcp -d INTERNAL_NET \ ones that started out on port 80. The proxy must make -dport 443 -j REDIRECT -to-ports 443 a decision about which intranet web server port it the proxy server can accept connections for an entire net- should connect to following authentication. Keeping work and respond using the IP address of the intended state information about previous requests was deemed destination server. Several router vendors also provide to be problematic, and a simpler approach was needed. layer-4-based policy routing that provides similar trans- It was decided to first attempt a port 443 connection to parent redirection. This policy routing is commonly used the intranet server, and look at the HTTP return code for outbound transparent HTTP proxying [4]. [7]. If the return code was 404, indicating the content The second task was to force the use of SSL is not found, the proxy then attempts a connection to encryption on all incoming HTTP connections. This is port 80 on the intranet server. If the return code is accomplished by running a redirector daemon on the again 404, this is returned to the web browser. If the proxy server. The redirector listens on port 80 and return code is not 404, the proxy forwards the server’s redirects port 80 connection requests to port 443 on data to the web browser. Figures 1 and 2 indicate the the same requested server. To accommodate the fact different possibilities described above. Obviously, the that the proxy masquerades as many web servers, a proxy does not support intranet web servers running wildcard X.509 [13] certificate is installed on the on non-standard ports.
User Agent Firewall Intranet (Browser) on web Web Server Internet proxy
https: Intranet (cookie) https: Request
https: Response,, return code = 404 https: Response https: Response, return, code not 404 https: Response
Figure 1: An http request with valid authentication cookie, internal server content on port 80.
User Agent Firewall Intranet (Browser) on web Web Server Internet proxy
https: Intranet (cookie) https: Request
https: Response,, return code = 404 https: Response https: Response,, return code = 404 https: Response, return code = 404
Figure 2: An http request with valid authentication cookie, content unavailable on internal server.
24 2003 LISA XVII – October 26-31, 2003 – San Diego, CA Crandell, Clifford, & Kent A Secure and Transparent Firewall Web Proxy
Third, the proxy needs a web authentication sys- must be able to tell if an HTTPS request comes from tem to tell whether or not to forward the user’s HTTPS an authenticated user session. Second, if the HTTPS requests. One method we have seen is to remember request does not include valid authentication creden- authenticated users by source IP address. The draw- tials, the proxy needs the URL of a web site that han- back to this method is you allow access to an IP dles an initial, interactive user login. address, not necessarily a specific user. Connections There are two popular methods for keeping track from remote sites using network address translation of authenticated web sessions: one camp uses Kerberos (NAT) [14], for example would allow multiple IP tickets [16], (RFC 2478) [10], and one uses cookies [6, addresses access because NAT changes all internal IP 7, 8], RFC 2965 [9]. Since the authentication system address to a single external IP address. We needed to determines what is allowed through the proxy, careful authenticate individual users rather than IP addresses. design [17] and implementation is crucial. The authentication methods used by the web fire- LANL had an existing cookie based central web wall vendors we investigated [1, 2, 3, 4, 5] were authentication service with the following security features: deemed unacceptable. Their methods were either • Two-factor CRYPTOCard [11] tokens with unacceptable from a security standpoint, or their one-time passwords are used instead of tradi- authentication scheme did not fit into our existing tional reusable passwords. authentication methods. Consideration was given to • The cookie sent to the browser is not persistent. the HTTP CONNECT [12] method, which tunnels • The cookie string is random and contains no HTTPS and bypasses normal application layer func- intrinsic value or information beyond being rea- tionality. This method presents a set of security issues, sonably unique and hard to guess. which we choose to avoid. Final design choices were • The browser is instructed to send the cookie two-factor authentication, a transparent proxy, and a only to ‘‘*.lanl.gov’’ sites and only within an secure authentication methodology. Implemented cor- SSL session. rectly, these would guarantee reliable access only to • The cookie is bound to the browser’s IP authenticated users and not IP addresses. address. This authentication service, at a minimum, must • Every new HTTPS request causes the cookie to provide the following capabilities. First, the proxy be re-verified. Central Intranet Web Authentication Server Server
Firewall Firewall
Layer 4 Web DMZ Switching Firewall Router Proxy
Firewall
Web Outer Login Internet Public Server Network User Agent (Browser) Figure 3: Web firewall architecture.
2003 LISA XVII – October 26-31, 2003 – San Diego, CA 25 A Secure and Transparent Firewall Web Proxy Crandell, Clifford, & Kent
• To address the risk posed by a user walking then masquerades as the internal web server when away from a logged in session at an Internet responding to the external web client. kiosk, cookies have a maximum server-side The third part is the web authentication system. lifetime after which the user must login anew. LANL’s web authentication system is single sign-on. • Cookies are also invalidated (server-side) if The session cookie obtained when the user authenti- they are not used (idle time out). cates works with all participating intranet servers. • The user can voluntarily log out. User authentication is two-factor authentication using These features lent themselves well to the authentication CRYPTOCards [11]. This type of authentication needs of the proxy so this preexisting system was used. makes use of one time passcodes. Following success- ful authentication, the password generated from the Implementation CRYPTOCard may not be used again. Our cookie Our actual proxy implementation is comprised of attributes are: several parts. See Figure 3, Web Firewall Architecture Name = "SessionID", for details. VALUE = "Authentication random number", Domain =".lanl.gov", Path ="/", The first part requires the use of policy based Max age = 0, routing to deliver HTTP/HTTPS messages to the Secure = "1". proxy server. The firewall Demilitarized Zone Net- All other attributes are default values. The Max age work (DMZ) [15] is contained within a commercial attribute keeps the cookie from being stored on the router that provides layer-4 policy routing. Incoming web browser and causes the destruction of the cookie packets addressed to URLs behind the firewall are when the web client application is terminated. The policy routed to the proxy server using layer-4 rules. secure attribute keeps the cookie from being transmit- All destination IP traffic addressed to any IP address ted in a non SSL request. behind the firewall going to ports 80 or 443 is policy Cookies provide a security plus: the ability to routed to the web proxy. revoke the authentication cookie at any time. Figure 4, The second part is the web proxy server plat- Initial HTTP Request and Authentication, describes an form. We use the Linux operating system. Netfilter initial connection without a valid authentication cookie. and iptables are part of the operating system. Netfilter Figure 5, HTTP Request with Valid Authentication runs within the kernel. Iptables provides the configu- Cookie, describes a normal connection request that has ration interface to netfilter [14]. As described earlier, a valid cookie. When an HTTPS request is forwarded using iptables redirect, all packets policy routed to the though the proxy, the cookie is passed along too. web proxy are redirected to the web proxy. The proxy Checking to see if a cookie is valid on the intranet
User Agent Firewall Web Central (Browser) on web Login Authentication Internet proxy Server Server http: Intranet Request Redirect to https: Intranet https: Intranet (no cookie) Redirect https: Web login (w/referrer) https: Web login (w/referrer) Login Form Submit Login Form Authenticate User Authenticate OK Redirect https: Intranet (set cookie) https: Intranet (cookie)
Note: Dashed lines are the result of redirects Figure 4: Initial HTTP request.
26 2003 LISA XVII – October 26-31, 2003 – San Diego, CA Crandell, Clifford, & Kent A Secure and Transparent Firewall Web Proxy servers can be tricky if the cookie is associated with the web proxy, scans are multiplied by the number of browser ’s IP address. The intranet web server sees the intranet web server addresses which are policy routed proxy as the user agent, but the cookie is associated the proxy. Under the right conditions, a simple scan with the end user’s IP address. Our solution is to trust turns into a denial of service attack on the web proxy. any cookie with a non-intranet address originating from For example, suppose the proxy is set up to the proxy. Integrating the proxy with other authentica- respond for and entire class B network to avoid the tion systems may require a different approach. administrative chore of tracking intranet web servers. Our web authentication system uses two Linux A network scan of the site’s intranet web servers will servers: a login web server to provide the interface send 64K requests to the proxy. This generates a bit of allowing the entry of a CRYPTOCard-generated pass- extra work, but the proxy can keep up as long as the word, and a back-end authentication server to validate requests are sequential and use 3-way TCP hand- the password. shakes. If the scan involves a burst of SYN packets The fourth and primary component of the system sent to 64K addresses, it turns into a SYN flood is the web proxy daemons. They handle redirection of attack. Connection queues for the proxy’s daemon fill HTTP to HTTPS requests and the actual connection up so additional connection requests are dropped. proxying. The proxy consists of approximately 2500 One solution is to throttle connection requests lines of C code. The first daemon, called REDIRD, from individual source IP addresses. Since the proxy listening on port 80, redirects port 80 intranet requests is responding for many destination addresses, the rule to port 443. The second daemon, WFD, handles the to limit connections applies to any destination IP port 443 connections. WFD handles header parsing to address. A second alternative is to dynamically block collect the authentication cookie and response codes, source IP addresses sending malicious packets. connections to intranet web servers, connections to the Finally, to minimize the problem, one can limit the central authentication server to verify cookies, and bi- number of addresses the proxy responds for to real directional forwarding of allowed browser and intranet intranet servers that need to be accessed from the web server traffic. Internet. This also improves security. Reliability In general, issues that affect other Internet ser- Performance vice availability will apply to the proxy. The usual The proxy software makes light demands on the hardware, operating system, server software, network- hardware. The application does some network I/O, ing and power concerns are relevant and are not dis- some string manipulation, and SSL encryption. A cussed here. Attacks from the Internet, however, pre- modestly configured PC should be able to keep up sent some different problems for the proxy. with most site’s workloads. Our proxy runs on a 800 The good news is the proxy does not need to be MHz Intel Pentium with 512 Mbytes of memory and directly addressable from the Internet. Policy routing 100 Mb/s Ethernet and enough disk storage for a mini- directs only port 80 and port 443 traffic destined for mal Linux operating system and logging. With this intranet hosts to the proxy. The proxy itself cannot be hardware and current workloads, there is no user scanned or attacked in the usual ways. detectable difference in response in fetching web The bad news is that the proxy must respond for pages through the proxy and from within the intranet. all intranet web servers. When someone scans your The web proxy currently proxies for one class B intranet address space for ports 80 or 443, the packets address range. There are approximately 1000 individual are routed to the proxy. The impact to a single host users who access the web proxy. The proxy typically from this type of scan is minimal. In the case of the handles 10,000 to 12,000 authenticated requests per day.
User Agent Firewall Intranet (Browser) on web Web Internet proxy Server https: Intranet (cookie) https: Request https: Response https: Response
Figure 5: HTTP request with valid authentication cookie.
2003 LISA XVII – October 26-31, 2003 – San Diego, CA 27 A Secure and Transparent Firewall Web Proxy Crandell, Clifford, & Kent
The average number of simultaneous connections is 12 will cause problems. When the user connects to the with peaks to 36. We have seen no performance issues server on port 80, and the proxy has established the since installing the proxy. Port 80 and 443 scans of the session with HTTP keepalive active, and then the user class B address space do on occasion cause a slowdown selects a link on the returned page sending them to the in the proxy’s response to valid connections. HTTPS server on the same web server, the proxy will The best way to improve performance is the continue to proxy the request to port 80. It will receive same as improving availability: keep the Internet noise an HTTP 404 response code and can recover from this away from the proxy. Also, a layer-4 switch or policy error for any request types except posts. In these cases router using proxy load balancing would allow for the proxy saves the original request and retransmits multiple proxies, resulting in scalable performance the saved request to port 443. However, due to the for- and improved availability for large applications. mat nature and potential large size of post requests, the original post is not kept and the request will fail. It is Topology not conceivable to save and reissue the post requests. The web proxy service consists of three pieces: a In practice, however, this has not actually been a prob- web login server, an authentication server (e.g., a Ker- lem for us because our web server system administra- beros key distribution center-KDC), and the proxy itself. tors do not operate their servers in this manner. Where you place these components in your firewall As written, the proxy only supports standard architecture involves some security trade offs. The HTTP ports 80 and 443. Other ports could be supported, weblogin server, a real web server, is the only compo- but would require additional code. One way to address nent that needs to be addressed directly from the internet. this would be to maintain a configuration file with a list Our weblogin server is outside the firewall, of servers and associated non-standard ports. where it is subject to attack. If compromised, the For those who are so inclined, one can use self- intruder can capture names and passwords of users on the weblogin server and use them to get access signed certificates, as opposed to paying for certifi- through the web proxy. Intruder access is limited to cates signed by recognized organizations. Doing so firewall services and some users’ accounts. will generate a warning message on the browser, requiring the user to purposely accept the self-signed Placing the weblogin server inside the firewall certificate. with a special rule permitting access from the Internet creates a bigger problem if the server is compromised. Revocable authentication credentials is a valu- The intruder is no longer limited to firewall services for able security tool. We limit the authentication creden- certain users; you have an intruder inside your intranet. tial lifetime. When the cookie expires, whoever is sit- Our authentication server is inside the firewall, but is ting at the browser has to login again. Inactive ses- not directly accessible from the Internet. With the sions are closed. If the proxy does not hear from the weblogin server outside the firewall and the authentica- browser within the idle time out interval, the user must tion server inside, we have a special rule (hole) that login again. Finally, the mindful user can explicitly allows them to talk. Capturing the weblogin server logout or simply exit the browser; both remove the allows the intruder to attack the authentication server. authentication cookie from the web client. Issues Even with all the built in security protections, it is still possible a session could be compromised. Good Forcing the use of SSL was recognized to pre- logs are critical for detection and recovery. The proxy sent issues to user agents that the end user must deal logs all successful transactions with the source and with. Some user agents will complain about wildcard destination IP addresses, the user’s ID, and the HTTP certificates and the user must specifically accept the request. The logs can be fed to a real time anomaly certificate. We have observed some vendor proprietary detection system and reviewed manually. Previous systems using HTTP protocols that have been coded login successes, failures and logouts can be displayed never to accept wildcard certificates. Another issue after a successful weblogin so the user can report arises when system administrators have image source unexplained activity. tags that are absolute, rather than relative. When the image tag for example is http://www.foo- Security Analysis bar.net.com/image, the image will not always display properly. Because we have forced the user agent to use Given the trusted nature of the web proxy, poten- SSL, it should comment that it is about to fetch an tial security risks are of particular concern. Below is insecure document, and give the user the option to an attempt to identify and show mitigation efforts for proceed or not. Instead, some agents will silently such concerns. ignore the image tag, resulting in an incomplete page First, we must deal with network-based risks. being displayed to the external web client. Given that the web proxy is an inbound firewall ser- Web servers that run both secure (HTTPS port vice proxy, the following three network-based risks 443) and insecure (HTTP port 80) on the same server are salient [15]:
28 2003 LISA XVII – October 26-31, 2003 – San Diego, CA Crandell, Clifford, & Kent A Secure and Transparent Firewall Web Proxy
• Session hijacking design and code reviews to help remove both logic and • Packet sniffing programming errors. Formalized system configuration • False authentication (we use a combination cfengine/cvs approach) including Session hijacking is best mitigated by the use of a minimized system install, helps to ensure a potential SSL/TLS connections for all data movement beyond compromise has few other applications to leverage. the initial referral from port 80. Since the proxy pre- Helping to ensure least privileged access for the proxy sents a properly (VeriSign) signed wildcard certificate, to the internal network, the router ACLs only allow the users can be relatively assured that a man-in-the-mid- proxy to have TCP ports 80 and 443 access, again miti- dle situation is not ongoing. As indicated in the SANS gating exposure should the proxy application or server SSL Man-in-the-Middle Attacks paper [19], using be compromised. Minimizing vulnerabilities on internal older Internet Explorer browsers can make it easier to web servers also reduces the risks of a compromised accomplish man-in-the-middle attacks. proxy. This mitigation is done with regular vulnerability scans of the internal network web servers. One should Packet sniffing is also well protected against by check the code using tools like Flawfinder and Rough the encryption provided by the SSL/TLS connections. Auditing Tool for Security (RATS). These tools will Usernames, passwords, and subsequent authentication possibly identify security problems with the code. The cookies are all contained within the encrypted channel. proxy also performs off host system logging. To date, The proxy does continue to support shorter key lengths there have been no known compromises of the web for older international browsers, mostly from a capabil- proxy system. ity need. Currently, we feel the capability need out- weighs of the risks of the reduced encryption quality, The last aspect of security concern involves end- though such decision may be reversed in the near future. user behavior. User behavior can make or break any good security stance. A major security concern with Finally, false authentication risks are well miti- the web proxy is user error. The proxy allows authen- gated by several aspects of the system. The use of the ticated users access intranet services. The service is short-lived authentication cookies, the relationship of designed for users coming from public systems they the cookie to originating IP address, and the random do not control like kiosks, cyber cafes, or conference content of the cookie mitigate the risk of stolen or terminal rooms. Many users do not understand web forged authentication cookies. The use of two-factor cookie management or are forgetful. Walking away generated one-time passwords significantly lowers the from a ‘‘logged in’’ browser is a real possibility and risk of stolen passwords. concern. The ability to revoke credentials is an impor- The transparent nature of the proxy may give tant feature, as is their max age attribute of zero. some level of protection through its obscurity and dif- ficulty in understanding the proxy’s relationship to the Conclusion web servers it sits in front. While the obscurity benefit Users have received the proxy positively; with may be minimal, it can definitely make scan probes of the LANL proxy handling over 10,000 authenticated the internal network look unusual. One negative con- HTTPS requests per day. The fact that the proxy is cern related to this transparent aspect is potential transparent, not requiring any browser preferences to denial of service to the proxy, both intentional and be set, is particularly beneficial. This feature allows unintentional. Since the server answers for a wide users to access LANL intranet web sites from any- range of addresses (class B in our case) on port 80 and where they have access to a web browser. In addition, 443, there is significant potential for overwhelming the per-user authentication has worked well. We have the server. Experience has shown this can be mitigated encountered no technical problems with the use of through the use of SYN cookies [20] and keeping the authentication cookies and no security issues with this port 80 redirector lean and simple since it receives the mechanism have been detected. The proxy’s simplicity majority of denial attacks. Active methods of intrusion and non-intrusive nature into the end data exchange detection and automatic host blocking have also miti- has allowed it to work effectively with Java, gated the impact of scans. Javascript, and all other web-based application exten- A second concern is logic and programming sions currently used and foreseeable. Overall, the errors. These are the standard inherent low-level sys- proxy provides a highly flexible yet secure mechanism tem risks relevant to any security-centric application. for Internet users to access intranet web content. These risks include logic errors (e.g., unintended granting of authentication cookies) and programming Author Information errors (e.g., buffer overruns). The resulting unautho- James (Jim) Clifford is the Network Service rized access could be either unintended access to the Team Leader and a Systems Software Engineer for the protected websites or administrative access to proxy Network Engineering Group at Los Alamos National itself. Obviously such risk must be addressed. Laboratory. His interests include Internet technology, A multi-tiered mitigation approach to such errors Linux, and practical computer security. Jim has a BS has been taken. First, the proxy has undergone both peer from the University of Michigan. He may be reached
2003 LISA XVII – October 26-31, 2003 – San Diego, CA 29 A Secure and Transparent Firewall Web Proxy Crandell, Clifford, & Kent view e-mail: [email protected] or smail mail: MS B255, [18] Marsh, Matthew G., Policy Routing, http://www. Los Alamos National Laboratory, Los Alamos, NM policyrouting.org/PolicyRoutingBook/ONLINE/ 87545. TO C . h t m l . Alexander (Alex) Kent is the Deputy Group [19] http://www.sans.org/rr/papers/60/480.pdf . Leader for the Network Engineering Group at Los [20] Lemon, Jonathan, ‘‘Resisting SYN flood DoS Alamos National Laboratory. His primary develop- attacks with a SYN cache,’’ Usenix BSDCon, ment projects include Laboratory-wide authentication 2002. and user account systems, network information propa- gation and architecture, and the Los Alamos firewall system. Alex has an BS and MS in CS from New Mexico Tech and an MBA from the University of New Mexico. He may be reached at
30 2003 LISA XVII – October 26-31, 2003 – San Diego, CA