TLScompare: Crowdsourcing Rules for HTTPS Everywhere Wilfried Mayer Martin Schmiedecker SBA Research, Austria SBA Research, Austria [email protected] [email protected] ABSTRACT stood concept and many websites protect portions of their For billions of users, today's Internet has become a criti- communication content with clients using HTTPS, e.g., lo- cal infrastructure for information retrieval, social interaction gin forms, only a minority of websites serve their entire and online commerce. However, in recent years research has content over HTTPS. While protocol extensions like HTTP shown that mechanisms to increase security and privacy like Strict Transport Security [11] can be used to enforce HTTPS HTTPS are seldomly employed by default. With the excep- for specified domains, only few website operators deploy tion of some notable key players like Google or Facebook, it [17]. This leads to numerous security- and privacy-related the transition to protecting not only sensitive information problems, as an active adversary can read, modify or redi- flows but all communication content using TLS is still in rect traffic at will during a man-in-the-middle (MITM) at- the early stages. While non-significant portion of the web tack. This can be either a malicious ISP [13], a local at- can be reached securely using an open-source browser ex- tacker on the same network using ARP spoofing [14], or a tension called HTTPS Everywhere by the EFF, the rules malicious certificate authority [19]. In a recent paper it was fueling it are so far manually created and maintained by a demonstrated by Gmail that not only HTTPS is commonly targeted by active attackers. Also attacks against other pro- small set of people. 2 In this paper we present our findings in creating and val- tocols which use TLS , e.g., SMTP are encountered in the idating rules for HTTPS Everywhere using crowdsourcing wild [9]. approaches. We created a publicly reachable platform at tlscompare.org to validate new as well as existing rules at Server administrators are responsible for deploying HTTPS large scale. Over a period of approximately 5 months we for a given website, and there is little that users can do to obtained results for more than 7,500 websites, using multi- protect their communication content if servers do not sup- ple seeding approaches. In total, the users of TLScompare port HTTPS. However, many websites employ HTTPS for spent more than 28 hours of comparing time to validate ex- a fraction of their pages, resulting in a valid certificate and isting and new rules for HTTPS Everywhere. One of our proper HTTPS deployment { many times serving the same key findings is that users tend to disagree even regarding webroot over both protocols. The browser extension HTTPS binary decisions like whether two websites are similar over Everywhere redirects otherwise unencrypted communication port 80 and 443.1 content to the encrypted counterpart where available, with a set of rules written in regular expressions. So far these rules are manually maintained, and while everyone can cre- Keywords ate his/her own rules, this approach clearly doesn't scale HTTPS, TLS, HTTPS Everywhere, crowdsourcing, privacy, to the current size of the web. Therefore we present our security platform in this paper, which tries to scale rule-creation and -verification using crowdsourcing. 1. INTRODUCTION The remainder of this paper is divided into the following One of the major problems in today's Internet is the lack sections: Section 2 gives the essential background on current of wide-spread HTTPS deployment: while it is a well-under- HTTPS deployment and HTTPS Everywhere. Section 3 in- troduces our approach for crowdsourcing rule generation. 1This is a preprint of the paper to appear at the Work- shop on Empirical Research Methods in Information Secu- Section 4 presents our implementation for the collection of rity (ERMIS) 2016, to be held with WWW'16. these rules. Section 5 discusses our findings before we con- clude in Section 6. 2. BACKGROUND Today Transport Layer Security (TLS) is responsible for the majority of encrypted online communication, not only for HTTPS but also numerous other protocols. Compared 2In this paper we use the term TLS to refer to all incarna- ACM ISBN 978-1-4503-2138-9. tions of the TLS and SSL protocols, if not specified other- DOI: wise. to plaintext communication it provides confidentiality, au- more complicated rule with two references. The \exclusion" thenticity and integrity. Currently, TLS 1.2 [8] is the most element is used very seldom. recent version of the SSL/TLS protocol family, with TLS 1.3 on the horizon. Many of the problems of TLS have been discussed in the literature [7, 21], and multiple guidelines <ruleset name="example.org"> on how to securely deploy TLS using best-practice methods <target host="example.org" /> have been published [22, 3]. <rule from="^http:" to="https:" /> </ruleset> While HTTPS Everywhere is a solution that has to be in- stalled in client browsers, there are also server-side solutions Listing 2: Trivial rewrite rule available to prevent cleartext communication despite server- side redirects from TCP port 80 to TCP port 443. HTTP Strict Transport Security (HSTS) [11] for example was stan- Files with 1 rule 13,871 dardized to enforce the use of HTTPS. Some browsers go Files with 2{10 rules 4,496 even one step further by shipping the software with prede- Files with more than 10 rules 44 fined HSTS lists to enforce HTTPS before the first use [16]. Total rules 26,522 Trivial rules 7,528 Rules without reference ($) 19,267 2.1 HTTPS Everywhere Rules with more than one reference ($) 417 3 HTTPS Everywhere is a browser extension for Firefox, Files with no exclusion 17,005 Chrome and Opera which is developed by the EFF and the Files with one exclusion 1,136 Tor project. It ensures that a secure connection is used if Files with more exclusion 272 available by rewriting URLs in the browser before attempt- Files with no securecookie 8,597 ing to start a connection over HTTP. This is helpful in the Files with one securecookie 9,215 particular use case: a secure channel is available, but not Files with more securecookie 601 used by default, and the server operator does not by default employ HTTPS for all users and communication content. Table 1: Ruleset statistics The rules for HTTPS Everywhere are written using regular expressions which specify the target domains as well as the secure counterparts that ought to be used instead. Listing 1 presents a simple rule for the domain torproject.org. Not 2.2 Related Work only does it rewrite all connections to http://torproject.org, but also all valid subdomains with the * wildcard. Recent publications regarding TLS in general focus firstly on the insecure deployment of TLS in the field, and sec- ondly on new cryptographic attacks and their possible im- <ruleset name="Tor Project"> pact. Problems range from the use of old and deprecated <target host="torproject.org" /> SSL/TLS versions [12], to the use of insecure cryptographic <target host="*.torproject.org" /> primitives like export-grade ciphers [18, 6] or weak RSA keys <!-- excluded content --> in certificates [4], to the use of TLS implementations with <rule known bugs like Heartbleed [10]. To the best of our knowl- from="^http://([^/:@\.]+\.)?torproject.org/" edge we are the first to discuss the creation of rules for to= "https://$1torproject.org/" /> HTTPS Everywhere using automatic and semi-automatic </ruleset> means. Listing 1: HTTPS Everywhere Rule for torproject.org4 3. DESIGN We split the design of automated rule generation in two These rules are manually developed and maintained using parts. First, suitable rules are generated with a rather sim- a public git repository. At the time of writing the repository ple algorithm. Second, these rules are validated by match- contains approx. 18,400 files that specify rewrite rules. The ing the similarity of the encrypted and unencrypted content process of manually creating rules is justified by the possi- with crowdsourcing methods. ble complexity of rewrite rules for particular deployments, yet simple rewrite algorithms do sometimes not know how 3.1 Automated Rule Generation to handle these deployments correctly. During the compi- lation of the browser extension, these xml files are merged Many domains are deployed with a simple HTTPS con- into a single SQLite file. Table 1 shows that the majority of figuration. For these domains rules can be automatically the existing rules of HTTPS Everywhere are in fact really generated with tool support. HTTPS Everywhere provides simple. 13,871 files contain only one rewrite rule and of all a script to generate trivial rules for these domains (called rules 73% do not reference the http URL (i.e., are not using make-trivial-rule). Rulefiles generated with this script $ in the regular expression). 28% use the trivial rewrite rule include one or more targets and the simple rule as listed in \http:" to \https:" as listed in Listing 2 and only 1.6% use a Listing 2. We can identify hosts that support HTTP and HTTPS via Internet-wide scanning techniques. Hosts that 3https://www.eff.org/https-everywhere use HSTS or server-side redirects can also be efficiently iden- 4https://github.com/EFForg/https-everywhere/blob/ tified and excluded from further processing. We then use a master/src/chrome/content/rules/Torproject.xml rather easy approach of rule generation by creating trivial rules for all domains. We also include often-used subdo- mains such as \www" and \secure". Also, the rank from the Alexa Top 1 Million ranking [1] is taken into consideration. The most important requirement for new HTTPS Every- where rules is that we want to avoid corrupt rules under all circumstances.