TLScompare: Crowdsourcing Rules for HTTPS Everywhere Wilfried Mayer Martin Schmiedecker SBA Research, Austria SBA Research, Austria [email protected] [email protected] ABSTRACT Strict Transport Security [11] can be used to enforce HTTPS For billions of users, today's Internet has become a criti- for specified domains, only few website operators deploy cal infrastructure for information retrieval, social interaction it [17]. This leads to numerous security- and privacy-related and online commerce. However, in recent years research has problems, as an active adversary can read, modify or redi- shown that mechanisms to increase security and privacy like rect traffic at will during a man-in-the-middle (MITM) at- HTTPS are seldomly employed by default. With the excep- tack. This can be either a malicious ISP [13], a local at- tion of some notable key players like Google or Facebook, tacker on the same network using ARP spoofing [14], or a the transition to protecting not only sensitive information malicious certificate authority [19]. In a recent paper it was flows but all communication content using TLS is still in demonstrated by Gmail that not only HTTPS is commonly targeted by active attackers. Also attacks against other pro- the early stages. While non-significant portion of the web 1 can be reached securely using an open-source browser ex- tocols which use TLS , e.g., SMTP are encountered in the tension called HTTPS Everywhere by the EFF, the rules wild [9]. fueling it are so far manually created and maintained by a small set of people. Server administrators are responsible for deploying HTTPS In this paper we present our findings in creating and val- for a given website, and there is little that users can do to idating rules for HTTPS Everywhere using crowdsourcing protect their communication content if servers do not sup- approaches. We created a publicly reachable platform at port HTTPS. However, many websites employ HTTPS for tlscompare.org to validate new as well as existing rules at a fraction of their pages, resulting in a valid certificate and large scale. Over a period of approximately 5 months we proper HTTPS deployment { many times serving the same obtained results for more than 7,500 websites, using multi- webroot over both protocols. The browser extension HTTPS ple seeding approaches. In total, the users of TLScompare Everywhere redirects otherwise unencrypted communication spent more than 28 hours of comparing time to validate ex- content to the encrypted counterpart where available, with isting and new rules for HTTPS Everywhere. One of our a set of rules written in regular expressions. So far these key findings is that users tend to disagree even regarding rules are manually maintained, and while everyone can cre- binary decisions like whether two websites are similar over ate his/her own rules, this approach clearly doesn't scale port 80 and 443. to the current size of the web. Therefore we present our platform in this paper, which tries to scale rule-creation and Keywords -verification using crowdsourcing. HTTPS, TLS, HTTPS Everywhere, crowdsourcing, privacy, The remainder of this paper is divided into the following security sections: Section 2 gives the essential background on current HTTPS deployment and HTTPS Everywhere. Section 3 in- 1. INTRODUCTION troduces our approach for crowdsourcing rule generation. Section 4 presents our implementation for the collection of One of the major problems in today's Internet is the lack these rules. Section 5 discusses our findings before we con- of wide-spread HTTPS deployment: while it is a well-under- clude in Section 6. stood concept and many websites protect portions of their communication content with clients using HTTPS, e.g., lo- gin forms, only a minority of websites serve their entire 2. BACKGROUND content over HTTPS. While protocol extensions like HTTP Today Transport Layer Security (TLS) is responsible for the majority of encrypted online communication, not only for HTTPS but also numerous other protocols. Compared to plaintext communication it provides confidentiality, au- thenticity and integrity. Currently, TLS 1.2 [8] is the most Copyright is held by the International World Wide Web Conference Com- recent version of the SSL/TLS protocol family, with TLS mittee (IW3C2). IW3C2 reserves the right to provide a hyperlink to the 1.3 on the horizon. Many of the problems of TLS have been author’s site if the Material is used in electronic media. 1 WWW’16 Companion, April 11–15, 2016, Montréal, Québec, Canada. In this paper we use the term TLS to refer to all incarna- ACM 978-1-4503-4144-8/16/04. tions of the TLS and SSL protocols, if not specified other- http://dx.doi.org/10.1145/2872518.2888606. wise. 471 discussed in the literature [7, 21], and multiple guidelines <ruleset name="example.org"> on how to securely deploy TLS using best-practice methods <target host="example.org" /> have been published [22, 3]. <rule from="^http:" to="https:" /> </ruleset> While HTTPS Everywhere is a solution that has to be in- stalled in client browsers, there are also server-side solutions Listing 2: Trivial rewrite rule available to prevent cleartext communication despite server- side redirects from TCP port 80 to TCP port 443. HTTP Files with 1 rule 13,871 Strict Transport Security (HSTS) [11] for example was stan- Files with 2{10 rules 4,496 dardized to enforce the use of HTTPS. Some browsers go Files with more than 10 rules 44 even one step further by shipping the software with prede- Total rules 26,522 fined HSTS lists to enforce HTTPS before the first use [16]. Trivial rules 7,528 Rules without reference ($) 19,267 Rules with more than one reference ($) 417 2.1 HTTPS Everywhere Files with no exclusion 17,005 HTTPS Everywhere2 is a browser extension for Firefox, Files with one exclusion 1,136 Chrome and Opera which is developed by the EFF and the Files with more exclusion 272 Tor project. It ensures that a secure connection is used if Files with no securecookie 8,597 available by rewriting URLs in the browser before attempt- Files with one securecookie 9,215 ing to start a connection over HTTP. This is helpful in the Files with more securecookie 601 particular use case: a secure channel is available, but not used by default, and the server operator does not by default Table 1: Ruleset statistics employ HTTPS for all users and communication content. The rules for HTTPS Everywhere are written using regular expressions which specify the target domains as well as the 2.2 Related Work secure counterparts that ought to be used instead. Listing 1 Recent publications regarding TLS in general focus firstly presents a simple rule for the domain torproject.org. Not on the insecure deployment of TLS in the field, and sec- only does it rewrite all connections to http://torproject.org, ondly on new cryptographic attacks and their possible im- but also all valid subdomains with the * wildcard. pact. Problems range from the use of old and deprecated SSL/TLS versions [12], to the use of insecure cryptographic <ruleset name="Tor Project"> primitives like export-grade ciphers [18, 6] or weak RSA keys <target host="torproject.org" /> in certificates [4], to the use of TLS implementations with <target host="*.torproject.org" /> known bugs like Heartbleed [10]. To the best of our knowl- <!-- excluded content --> edge we are the first to discuss the creation of rules for <rule HTTPS Everywhere using automatic and semi-automatic from="^http://([^/:@\.]+\.)?torproject.org/" means. to= "https://$1torproject.org/" /> </ruleset> 3. DESIGN We split the design of automated rule generation in two Listing 1: HTTPS Everywhere Rule for torproject.org3 parts. First, suitable rules are generated with a rather sim- ple algorithm. Second, these rules are validated by match- These rules are manually developed and maintained using ing the similarity of the encrypted and unencrypted content a public git repository. At the time of writing the repository with crowdsourcing methods. contains approx. 18,400 files that specify rewrite rules. The 3.1 Automated Rule Generation process of manually creating rules is justified by the possi- ble complexity of rewrite rules for particular deployments, Many domains are deployed with a simple HTTPS con- yet simple rewrite algorithms do sometimes not know how figuration. For these domains rules can be automatically to handle these deployments correctly. During the compi- generated with tool support. HTTPS Everywhere provides lation of the browser extension, these xml files are merged a script to generate trivial rules for these domains (called into a single SQLite file. Table 1 shows that the majority of make-trivial-rule). Rulefiles generated with this script the existing rules of HTTPS Everywhere are in fact really include one or more targets and the simple rule as listed in simple. 13,871 files contain only one rewrite rule and of all Listing 2. We can identify hosts that support HTTP and rules 73% do not reference the http URL (i.e., are not using HTTPS via Internet-wide scanning techniques. Hosts that $ in the regular expression). 28% use the trivial rewrite rule use HSTS or server-side redirects can also be efficiently iden- \http:" to \https:" as listed in Listing 2 and only 1.6% use a tified and excluded from further processing. We then use a more complicated rule with two references. The \exclusion" rather easy approach of rule generation by creating trivial element is used very seldom. rules for all domains. We also include often-used subdo- mains such as \www" and \secure". Also, the rank from the Alexa Top 1 Million ranking [1] is taken into consideration. The most important requirement for new HTTPS Every- 2https://www.eff.org/https-everywhere where rules is that we want to avoid corrupt rules under all 3https://github.com/EFForg/https-everywhere/blob/ circumstances.