Bulletproof Hosting: Ecosystem and Registry-Based Approaches

Pierre Tholoniat∗1,2, Ian Chai†1, Amira Dhalla1, Damin Jung1, Stephen Owusu1, and John Sakellariadis1 1Columbia University 2Ecole Polytechnique

February 22, 2020

Abstract establishment of a global registry that builds on previous successful models, to unite and extend existing Bulletproof hosts are domain or web hosting legal, policy, and technical efforts. providers that allow customers to host illegal or malicious content on their servers. In this paper, we use a cross-disciplinary approach to tackle this issue from 1 Introduction a technical, legal and policy perspective. The problem is emblematic of the issues that Bulletproof hosting facilitates both petty and sophis- plague traditional methods of regulating the global ticated crime, in addition to the distribution of illicit commons. Both legal and policy regimes operate content. While there is a wide variety of bulletproof within national boundaries, while hosts and perpetra- hosting services (BPHs), this paper focuses only on tors operate trans-nationally with agility, paying lit- those hosting providers that enable the most “toxic” tle regard to national borders and bureaucracy, there- content types: child exploitation, malware, exploit fore stymieing enforcement efforts. While there have kits and C&C components. been successes, hosts still exploit legal and policy la- The paper proceeds in three sections. First, we of- cunae around the world. From the technical perspec- fer a brief background on bulletproof hosting and tra- tive, traditional content blocking methods, such as ditional solutions, and argue that solutions should be- IP blacklisting or DNS blocking, are no longer suffi- gin with targeting high-toxicity hosts. The following cient to contain bulletproof hosts. We expose chal- sections emphasize the problems and possible solu- lenges raised by modern networking, such as cloud tions to BPHs from a unique perspective: combining services, encryption or distributed architectures, and that of policy, technology, and law. From a policy we show the interest of technical methods such as perspective, we explore the evolution of BPHs over SNI filtering, machine learning over encrypted traf- the last ten years in order to justify and explain the fic or peer-to-peer traffic analysis. We also underline need for new solutions. The technical section empha- technical difficulties that BPH registries face, such sises the need for high-fidelity attribution that avoids as accountability issues, privacy leaks, identification collateral damage, and it offers a roadmap for how to and prescription problems. accomplish this. This section concludes by explor- We study registry-based solutions and propose the ing several further technical strategies to maximise the effectiveness of the registry. From a legal per- ∗ [email protected] spective, we cover existing approaches, and cover an †[email protected] array of solutions which would be natural and effec-

1 tive extensions of the current regime. Ultimately, we ing providers, bulletproof hosts do not impose strict advocate voluntary ’soft-law’ approaches to minimise policies about what type of content clients can host. space for BPHs, and the strengthening of interna- Many hosts also enable their clients to evade international law and cooperative norms, both of which will tional and domestic law enforcement by hiding their be necessary to capitalise on an international registry. infrastructure, ignoring take-down requests, or ex- We end with highlighting the value and feasibility of ploiting jurisdictional loopholes. Both legal and pol- such a registry, in addition to recommending a variety icy regimes operate within national boundaries, while of lower-level solutions for mitigating the availability hosts and perpetrators operate trans-nationally with of BPHs. agility, paying little regard to national borders and An online registry could be used to identify, verify, bureaucracy, therefore stymieing enforcement efforts. and track abusive hosts across the globe. It builds While there have been successes, hosts still exploit le- on the model of previous work and similar registries, gal and policy lacunae around the world. This is em- such as Spamhaus, Green of Team Cymru, StopBad- blematic of the issues that plague traditional methods ware, and apwg.org. In addition, collective action of regulating the global commons. by private and public sectors globally is required to Not all bulletproof hosts are created equal. Bullet- tackle the problem. We study these and the model of proof hosts differ with regards to the types of content the IAEA as an example of a successful international they are willing to host; their complicity with illegal organisation that is predicated on collective action. activity; their reliance on abusive content to turn a Like the IAEA, the agency would be neutral and in- profit; popularity; price and quality of their services; vested with the authority to investigate complaints geolocation; and primary jurisdiction. For example, brought by constituents. In so doing, it would fill the the following is an extract from an advertisement for capability and credibility gaps that undermine exist- one of the biggest BPHs, that was posted to a cyber- ing approaches to mitigating the availability of high crime forum[1]: toxicity BPHs. Through information sharing with constituent -Based in Asia and Europe. states, the private and non-profit sectors, and an in- -It is allowed to host: ordinary sites, door- house technical staff, the registry could wield the au- way pages, satellites, codecs, adware, tds, thority to verify and attribute abusive content. In ad- warez, pharma, spyware, exploits, zeus, dition, its activity would pressure non-compliant or IRC, etc. -Passive SPAM is allowed (you complicit entities to do a better job of self-regulation, can spam sites that are hosted by us). -Web while enabling well-intentioned but under-resourced spam is allowed (Hrumer, A-Poster . . . .) hosting services or jurisdictions to do the same. Fi- -Forbidden: Any outgoing Email spam, DP, nally, the registry could adjudicate cases brought by porn, phishing (exclude phishing email, so- member-states against the most malicious abettors. cial networks) Like the IAEA, its judgements could be used to val- idate or invalidate cross-border action in defense of There is a server with instant activation un- the global commons. der botnets (zeus) and so on. The prices will pleasantly please you! The price depends on the specific content!!!! 2 Bulletproof hosting Unfortunately, the wide variety of bulletproof hosts 2.1 Background can blur the distinctions between malicious entities and unwilling abettors of illegal activity. For exam- Bulletproof hosting refers to domain or web hosting ple, one of the individuals indicted for the banking providers that allow customers to host illegal or ma- malware known as Gozi was charged with conspiracy licious content on their servers. Unlike regular host- for his role in providing bulletproof hosting despite

2 resistant messaging applications, like Telegram Figure 1: Malicious content stored in bulletproof and Signal, have come to resemble bulletproof hosts and their corresponding toxicity. hosts. Both applications use elaborate techniques to sidestep DNS, IP, and DPI blocking [5]. Likewise, civil liberties organizations, such as the Hermes Cen- ter for Transparency and Digital Human Rights [6], act as de facto BPHs by providing technical tools to help activists set up whistle-blowing sites. While disaggregating “good” hosters from “bad” hosters represents a serious concern, this paper’s focus on the most toxic content means that it tends not to engage other civil rights and liberties issues, such as the protection of free-speech. Few would dis- agree with stringent measures being taken against child exploitation, malware, exploit kits, and C&C components, it is on these that we focus.

2.2 Technical Overview 2.2.1 Networking protocol stack having no knowledge of the malware itself [2]. Mali- At their core, these various bulletproof hosting ser- cious bulletproof hosts do not hesitate to exploit this vices are delivered by a physical or virtual server that uncertainty to avoid calls for greater oversight. They executes programs such as web services (for illegal on- argue that they do not know they are supporting line services), file services (in the case of file-sharing criminal activity or that they cannot monitor their or video hosting) or mail services (for spam). networks without violating the privacy of their cus- The role of a bulletproof host is to connect such a tomers, which they are unwilling to do. We find that malicious server to the outside world through the In- this argument is untenable, as evident from the con- ternet. The server is connected to the network of an tent and privacy policies of such content delivery net- Internet Service Provider (ISP) that associates the works as Cloudflare. Cloudflare terminated 5428 do- physical server with an IP address. The ISP’s net- mains over 8 years, but only after engaging the users work is interconnected with other ISP’s networks, in it found in flagrant violation of its anti-child sexual order to exchange packets through routing protocols. abuse material[3]. Finally, the server’s IP address is associated with a Given the diversity of bulletproof hosts and the domain name through a system of DNS, that consists unique challenges presented by each, this paper fo- of a hierarchy of domain name servers. cuses only on the most pernicious content hosters. To access a bulletproof-hosted service, a client (ei- To do so, it relies on a toxicity-based classification ther a malicious individual or a compromised ma- system developed by TrendMicro [4] (see Figure1). chine) will provide the host’s domain name to a DNS Here, toxic is defined as having strong negative ef- service to obtain the corresponding IP address, con- fects on others. Our paper focuses on solutions for nect to the server through its IP address and finally mitigating the availability of content with a toxicity exchange packets with the malicious machine. level of “high” and “very high”: Child exploitation, This technical overview presents the bulletproof C&C components, Exploits, and Malware. hosting problem as a communication chain. To sim- It is worth noting that the very meaning of bullet- plify how the issue may be resolved technically, this proof hosting is evolving. In recent years, censorship- chain can theoretically be broken in 5 different places:

3 in the host’s physical machine, in the host’s ISP, Deep packet inspection. Contrary to shallow in the routing and DNS system, in the client’s ISP packet inspection methods, deep packet inspection or in the client’s machine. Technical solutions exist (DPI) methods rely on the analysis of the specific at every level of the protocol stack, and are worth information carried inside packets[8]. Since such an analysing. Solutions include: i) physically taking analysis is more computationally costly than mere down the malicious server; ii) cutting the server header filtering, packets are usually acquired by copy- from its ISP; iii) preventing the host from register- ing the flow of packets directed to one port or split- ing a given domain name or disrupting the routing ting the network traffic. It is possible to use a broad of traffic; iv) implementing blocking at the client’s range of analysis techniques on the content of pack- ISP level; or, finally v) filtering traffic directly on ets, making DPI a versatile tool used for corporate the client’s machine (e.g. antivirus or Parental Con- network security, service analysis or lawful intercep- trol). We will describe traditional content blocking tion in addition to content blocking. techniques for BPH that follow this simple networking stack, before moving to the technical challenges Multi-level actions. The best traditional techni- raised by modern bulletproof hosting such as virtual cal approaches to tackling bulletproof hosting com- servers, content delivery networks or encryption. bine these different actions. For example, the botnet Gameover Zeus used to be partially coordi- 2.2.2 Traditional content blocking tech- nated through a system of deterministically gener- niques ated pseudo-random domain names hosted in different jurisdictions. To take the control center down, the Shallow packet inspection. The most common FBI reverse-engineered the domain name generator content blocking techniques rely on the analysis and and obtained a temporary restraining order blocking blocking of packet headers, at the transport layer, specific ranges of domain names from being registered networking layer or application layer of the protocol through US registrars. However, this strategy did not stack. Those methods are also called shallow packet work for “.ru” domains that are not under American inspection, because they do not inspect the payload jurisdiction. Instead of acting at the DNS level, the of packets. FBI obtained American ISPs to install a filtering sys- Packet dropping is one way to block content from a tem to stop compromised machines from connecting given list of IP addresses of malicious websites. It is to this domain name[9]. implemented by ISPs that drop the packets heading towards or from any of the blacklisted IP addresses, making their content unavailable. However, packet Limits. The techniques cited above are the sim- dropping method can lead to overblocking and col- ple and fundamental approaches to content filtering. lateral damage, as we will expose in Section 3.3.1. However, in a real-world setting they face technical DNS poisoning is another method in which the in- challenges that we describe in Section 3.3. formation in the DNS server is manipulated so that it will send the user to a modified IP address or the 2.3 Legal Overview hostnames of websites to be blocked will not give the Here, we set out the background of legal issues sur- correct IP address [7]. rounding BPHs, along with existing legal solutions. Content filtering systems are employed to block en- In Section 3, we outline the legal bases for effective tire websites and specific items like an image or a web action against BPHs. Crucially, despite the transna- page. To accomplish this, it checks whether or not tional nature of transactions over the internet, do- the URL is to be blocked and makes sure that the con- mestic law remains the primary mode of regulation. tent is not accessible. This content-focused method is less likely to wreak collateral havoc but is easier to evade, as described in section 3.3.1.

4 2.3.1 Content takedown and law enforce- national authorities[12][13]. ment This is exacerbated by the fact that many countries do not have the intelligence or enforcement capabil- Mitigating the availability of content hosted by BPHs ities to identify and punish bulletproof hosts, or the can be carried out in a number of ways: i) through appropriate legal and diplomatic agreements to facili- takedown orders, which require a court order from a tate either extradition or intelligence sharing. Hence, judge; ii) through takedown requests, which are sub- while enforcement action will remain an essential tool mitted to content providers by private entities or indi- for fighting cybercrime, it is probably better suited viduals; and iii) through forcible takedowns, wherein for dealing with persistent threat actors posing di- domestic or international law enforcement authori- rect dangers to a a clearly defined legal entity. Even ties seize or destroy malicious infrastructure. Each then, it is only viable in states that have robust legal approach faces drawbacks. regimes, and relatively efficient court systems. Com- bating shared threats to the global digital commons Take-down orders Legal and bureaucratic con- therefore requires a different solution that can move straints slow the pace of takedown orders and en- quick enough across national boundaries to combat forcement action. For example, though Microsoft has such hosts. successfully used court orders to wrest control of web infrastructure used by sophisticated Advanced Per- Voluntary takedowns Voluntary takedowns offer sistent Threat groups, the processes took multiple a second approach to removing pernicious content. years because Microsoft had to collect sufficient digi- This type of takedown can be carried out at different tal forensics before it could even submit a legal filing, layers in the web hierarchy: through ISPs, the DNS, which is itself yet another time-consuming process or web and content hosting providers. [10]. Traditionally, authorities have relied on takedowns Likewise, before police authorities can seize or dis- at the ISP or DNS level. For example, in the late rupt criminal infrastructure, they must present ev- 2000s, the security community cooperated with ISPs idence before a judge and secure a warrant. Even to dismantle the Russian Business Network (RBN)[1] when bulletproof hosts operate in states with robust and McColo Corp [14], large-scale bulletproof host- legal institutions and strong law enforcement capabil- ing providers involved in the distribution of spam, ities, takedown orders and enforcement action take malware, botnet C&C and child pornography. To time. Even successful operations have noted that do so, security researchers shared information with difficulty of meeting legal requirements, whether be- the entities’ upstream ISPs proving that abnormally cause they are not clearly drafted, or because they high rates of abusive content were connected with pose significant legal obstacles[11]. the RBN and McColo Corp autonomous systems. At Unfortunately, these ideal conditions rarely obtain that point, the ISPs agreed to sever access to the two when it comes to bulletproof hosting. Bulletproof companies’ networks. While the result was a desir- hosts deliberately exploit jurisdictional loopholes and able one, there are implications. While this highlights enforcement gaps in order to avoid sanction (as cov- the role that different participants in the ecosystem ered later). Some bulletproof hosts even have so- can play, without the need for legal compulsion or phisticated customer service departments that advise new legal regimes, there still remains the possibility clients on where to apportion content so as to be lo- for over-reliance on private entities. If a private en- cated in the most lenient jurisdictions. In addition, tity fails to cooperate, other measures will need to bulletproof hosts often set up their operations in ju- be sought - and this is exactly what BPHs do. This risdictions that either turn a blind eye to their activ- then necessitates seeking the cooperation of a signif- ities or have weak political institutions. A number icant number of other entities in the ecosystem. The of bulletproof hosts have even moved into war-torn challenges surrounding this are explored in Section 4. areas of Syria and Ukraine, ostensibly, to avoid inter-

5 2.3.2 Law Enforcement ample of both facilitative legislation and lawful hacking, amendments were made in 2016 to the Federal While this paper argues that traditional methods are Rules of Criminal Procedure to enable magistrates insufficient, there have been several instances of effec- to issue search warrants when “where the media or tive law enforcement, especially against highly toxic information is located has been concealed through BPHs. The most recent example was in Germany, on technological means”3. September 27 2019[15]. Based on available English sources, the basis for legal action in German law is unclear, but seems targeted at whether information Jurisdictional Loopholes As mentioned above, is transmitted for illegal purposes. One possible basis bulletproof hosts operate in countries that do not im- for some law enforcement actions is Germany’s Data pose strict requirements on the provision of internet services. Recent news have been dominated by bullet- Retention Law, which was recently amended to em- 4 power security agencies to secure access to metadata proof hosts in Russia , or areas with weaker political of phone calls and internet connections of persons and and legal institutions. In the former case,hosts hew organizations who are under investigations for severe closely to the letter of Russian law, and adhere to crimes in German law. Russian legal requirements for companies textite.g. Other States have similarly relied on existing leg- occasional reporting. Beyond that, bulletproof host- islation or similar laws. In the United States, the ing is fair game for the Russian-registered company. Federal Trade Commission relied on a broad read- Looking further afield, there is even further evi- ing of violations of Section 5(a) in order to file for dence of widespread jurisdictional loopholes. The injunctions against such hosts under Section13(b), availability of some other territory in which to exploit 15 USC 45(a)1. In November 2016, American and jurisdictional loopholes means that tackling BPHs European law enforcement agencies, international or- will remain a game of jurisdictional whack-a-mole[16] ganisations, and private companies collaborated and until an international regime or consensus is in place. relied on domestic legal bases to take down the On the global front, the United Nations Conference Avalanche botnet infrastructure - the same basis on Trade and Development has stated that as of 2019 would ostensibly apply to botnets. there are 138 countries with cybercrime legislation Other strategies have also been taken elsewhere. In (this represents 72%); 9% with draft legislation on Singapore, ISPs are regulated by the Infocomm and cybercrime; 18% with no legislation on cybercrime; and 1% with no data at all showing the existence or Media Regulatory Authority, under the Broadcasting 5 (Class Licence) Notification, which relies on licensing otherwise of cybercrime legislation. Further, accord- of ISPs (each is issued with a Class Licence) and have ing to the Global Cybersecurity Index 2018, which is to comply with the IMDA’s Internet Code of Practice an initiative under the auspices of the International to ensure that they prevent access to harmful sites or Telecommunication Union, as many as 87 countries certain types of content. 3Susan Hennesssey, Rule 41: Resolving Procedural De- States sometimes also conduct (apparently) lawful bates to Face the Tough Questions on Government Hacking, ‘hack-back’ operations. For example, GCHQ under- Lawfare, 1 December 2016, https://www.lawfareblog.com/ rule-41-resolving-procedural-debates-face-tough-questions-government-hacking. took operations against Anonymous. China possesses - this was catalysed by the warrants that were given under the Great Cannon capability which injects code into the good faith exception to violations of the old Rule 41, in users’ browsers, that re-directs foreign traffic meant relation to the Playpen child pornography case. 4 for Chinese websites to target servers2. In another ex- Brian Krebs, “Meet the World’s Largest Bul- letproof Hoster,” Krebs on Security, 19 July 1Section 5(a) provides that engaging in unfair 2019, https://krebsonsecurity.com/2019/07/ acts and practices is illegal. For a 2009 example, meet-the-worlds-biggest-bulletproof-hoster/. 5 see https://arstechnica.com/tech-policy/2009/06/ UNCTAD, “Cybercrime Legis- ftc-forces-hive-of-scum-and-villainy-isp-offline/; lation Worldwide”, available at 2 Assuming that the legality of governmental action is https://unctad.org/en/Pages/DTL/STI_and_ICTs/ICT4D- undisputed. Legislation/eCom-Cybercrime-Laws.aspx.

6 were ranked ‘Low’ in terms of their commitment to- Further, encouraging harmful, active steps to inter- wards cybersecurity6. Evidently, BPHs have plenty fere in another State’s digital infrastructure is also of space to operate within, and are able to relocate several degrees removed from a failure to regulate. to suitably lax regimes to avoid law enforcement. International law also does not justify States taking Taking a continen-wide perspective, Kshetri notes action against BPHs that are based beyond their ter- that there is an increasing amount of cyber-attacks on ritorial borders, even though there have been pockets the African continent that is attributable to vulnera- of international efforts towards such end. This is evi- ble systems, lax cybersecurity practices, weak legisla- dent in in the EU Convention on Cybercrime, and the tion and law enforcement7. A November 2016 report 2014 African Union Cyber Security Convention (’AU of the African Union Commission and Symantec in- Cybersecurity Convention’). However, even such ef- dicated that, out of the 54 countries in Africa, 30 did forts can be patchy or incomplete. For example, the not have specific legal provisions to fight cybercrime AU Cyber Security Convention has only been ratified, in general or deal with electronic evidence.8 that is implemented in domestic legislation, by 5 of That is not to say that having legislation solves the the 55 AU states[17]. In the absence of international issue, the next step of enforcement is equally tricky. consensus, the spectre of potential violations of an- Promulgating appropriate legislation is a necessary other State’s sovereignty always looms when action first step to reduce the space within BPHs can op- is taken beyond a State’s territorial borders. The erate. States are likely to be more proactive when cumulative effect of this is that countries can take attention is formally drawn to their lack of regula- as much or as little action as they wish10. Our rec- tory efforts, and the Registry might be one effective ommendation to create an international Registry will way of institutionalising this process. go some way in ameliorating this problem. We also recommend taking measures at the international and 2.3.3 International law and cooperation domestic levels, as well as measures to increase the synergies at the interface of domestic and interna- International law, as it currently stands, offers even tional law. These will be covered in Section 4. less legal tools than domestic law for solving the problem posed by BPHs. International law currently does Traditional modes of international coopera- not prohibit bulletproof hosting, or accessing of bul- tion Despite these obstacles, international cooper- letproof hosts. In addition, failing to regulate BPHs ation still takes place in several traditional modes. It falls short of breaching any traditional notion of state can take the form of cooperation agreements, extra- responsibility9. For instance, omitting to regulate dition arrangements, mutual legal assistance (MLA), BPHs may bear some similarity to state-backed hack- mutual recognition of foreign judgments, informal ing, when States encourage or deliberately fail to take police-to-police cooperation and partnership, etc. For action against hackers. In such a case, the norms instance, the Budapest Convention on Cybercrime11, surrounding use of force and state attribution would provides for a regime for extradition, mutual legal seem to be the first ports of call. However, BPHs are assistance, as well as exceptions for protecting state private enterprises and not State-owned or related. interests, to facilitate quick investigation and prose- 6In contrast to 54 countries ranked high, and 53 cution amongst EU Member States (Articles 25-35). countries ranked ‘Medium’: Global Cybersecurity Index, In addition, there are also several multilateral instru- International Telecommunications Union, available at https://www.itu.int/en/ITU-D/Cybersecurity/Pages/global- 10 Both in a legal and political sense. As a matter of inter- cybersecurity-index.aspx. national law, see S.S. Lotus (France v. Turkey), 1927 P.C.I.J. 7 Nir Kshetri, “Cybercrime and Cybersecurity in Africa”, (ser. A) No. 10 (Sept. 7). Journal of Global Information Technology Management, avail- 11 Convention on Cybercrime, ETS No. 185; opened for able at: https://www.tandfonline.com/doi/full/10.1080/ signature in Budapest on 23/11/2001; entry into force on 1097198X.2019.1603527. 01/07/2004 (under the auspices of the European Council), 8 Ibid., Nir Kshetri. Available at: coe.int. 9Political attribution is a different matter.

7 ments12, that criminalize illegal access to a computer tem within which BPHs operate, to avoid taking a system, illegal interception, illegal computer data and whack-a-mole approach. This is difficult. The same system interference, and misuse of devices.13 Cou- network effects that have enabled huge benefits for pled with several successful examples of takedowns14, millions of internet users also mean that BPHs, with it is clear that an international consensus on the ne- the right blend of opportunism, weak legal regimes, cessity of effective cybercrime prevention and prose- and slower enforcement, can operate with impunity. cution exists15. While there is some political force be- However, we argue that taking such an approach al- hind the making of model norms, international norm- lows for the identification of points of the ecosystem making has not kept up with the need for clear and which can be targeted to take action against BPHs facilitative legislation that clearly empowers law en- more effectively. forcement agencies to go beyond their borders to pursue BPHs. The traditional modes of international co- 3.1 New developments and emerging 16 operation therefore remain slow and bureaucratic - vulnerabilities a recipe for ineffectiveness compared to nimble BPHs. Recent developments in bulletproof hosting pattern render traditional takedown methods less effective. 3 Challenges and ecosystem ap- In an effort to avoid takedowns, most bulletproof proaches hosts no longer host content on their own network. Instead, bulletproof hosts have begun rent- The foregoing has highlighted that it is necessary to ing and reselling infrastructure from legitimate up- takes a an approach that considers the entire ecosys- stream providers, or even operating as marketplaces for hosting without directly engaging in the illicit ac- 12 e.g. African Union (AU), the Economic Community of West African States (ECOWAS), the Common Market for tivity[18]. Eastern and Southern Africa (COMESA), the League of Arab This new operational models poses two new chal- States (Arab League), the Commonwealth Secretariat (COM- lenges. First, it makes attribution more difficult. SEC), and the International Telecommunications Union (ITU), Whereas in the past the security community could “Combating Cybercrime: Tools and Capacity Building for Emerging Economies”; World Bank and United Nations Pub- study abuse rates at the level of an entire AS, bullet- lication, Available at: documents.worldbank.org. proof hosts now hide within sub-allocated networks 13 Ibid Combating Cybercrime: Tools and Capacity Building that are more difficult to spot. Second, as a re- for Emerging Economies. sult of these actions, cutting off access to entire au- 14 In Operation Blue Amber, police arrested 130 suspects who were connected with cyber fraud including fraudulent on- tonomous systems, or even sizeable sub-allocated net- line purchases of airline tickets. The suspects used stolen credit works, risks significant collateral damage to legiti- card data at 140 airports around the world. Through an in- mate hosting providers. ternational law enforcement operation, coordinated through Fortunately, mitigating the problem of bulletproof Europol in The Hague, the Netherlands, INTERPOL in Sin- gapore and Ameripol in Bogota, Columbia with support from hosting does not require relying on direct takedowns Canadian and United States law enforcement authorities, those alone. In a paper presented at the 2019 USENIX 130 suspects were tracked and arrested. Increased commitment Security Symposium, a group of researchers found from the private sector, law enforcement apparatus and inter- that bulletproof hosts are sensitive even to modest national organizations helped the operation to be executed at airports in 25 countries in Europe and 24 other countries in business disruption: Asia, Australia, America and Africa. BPH providers have few choke points in the supply 15 Ibid Combatting Cybercrime: Tools and Capacity Build- chain amenable to intervention, though profit mar- ing for Emerging Economies. 16 See, for instance, Vipual Kharbanda, “International Co- gins are very slim, so even a marginal increase in op- operation in Cybercrime: The Budapest Convention”, 29 April erating costs might already have repercussions that 2019, The Center for Internet and Society India Blog, avail- render the business unsustainable[18]. able at https://cis-india.org/internet-governance/blog/ In other words, even if takedown efforts fail, the . vipul-kharbanda-april-29-2019-international-cooperation-in-cybercrime-the-budapest-conventionthreat of takedown via greater pressure from the se-

8 curity community can be valuable in and of itself. By the initial platform takes down the BPH services. forcing bulletproof hosting business to take defensive For bulletproof hosts, relying on intermediary ser- measures and undercutting customer confidence by vices is a double-edged sword. On one hand, it shining a spotlight on offenders, regulators can un- provides them an additional layer obfuscating their dermine the profitability of bulletproof hosts. identity and physical location. Under the U.S. law, On the other hand, the new bulletproof hosting Internet intermediaries such as CDNs are protected model presented above introduces new vulnerabilities from having to proactively enforce content laws on for bulletproof hosts. Now that that illicit content their customers.[20] Low-toxicity bulletproof hosts hosts are reliant on access to legitimate providers, can therefore hide in the mass of services hosted by they are vulnerable to takedowns at a lower level cloud providers, and move from one provider to an- of the hierarchy: legitimate web content and ser- other when necessary. On the other hand, adding vices providers. Accordingly, the security community intermediaries to the technical stack expands the sur- should devise methods that empower these entities face attack of bulletproof hosts, who become vulner- and mitigate the existing challenges they face. able to any change of policy of their cloud provider. The relative centralization of the market for cloud 3.2 Cloud services and content deliv- services provides a reduced number of points of fail- ery networks ure that law enforcement agencies can target. The Electronic Frontier Foundation notes that content The technology behind online services hosting has takedown requests, for example from industry groups evolved since the time when each service was physi- such as the Motion Picture Association of America, cally located on a single server plugged to the Inter- focus increasingly on CDNs rather than on the web- net. Large-scale static content distribution increas- sites providing the original content.17 CloudFlare, ingly relies on content delivery networks (CDN), that one of the main CDNs, has so far refused to comply store and deliver the content through a network of to most of these requests but decided to drop support replicas located closer to the customers. CDNs in- for controversial customers such as 8chan in 2019[21]. crease the speed of content delivery, offer protection against DDoS attacks and guarantee a higher reliabil- 3.3 Filtering and modern networking ity thanks to load balancing and fault-tolerant mechanisms. However CDNs make traditional IP blocking 3.3.1 Collateral damage and under-blocking techniques obsolete. Indeed, the same content can In modern networking, one IP address does not al- be served by different servers belonging to the same ways correspond to one machine: blocks of IP ad- content delivery network but having different IP addresses are often shared by multiple websites, and dresses. Conversely, a given IP controlled by a CDN more generally online services rely increasingly on server can be used to serve different and completely shared infrastructure. As a result, blocking a range unrelated websites, potentially mixing legitimate and of IP addresses in an attempt to take down a BHP nefarious traffic. CDNs can be actively optimized to can wreak havoc on innocent services. For example, achieve censorship-resistance[19]. in 2003 Pennsylvania issued blocking orders to tar- Besides the simple hosting of static content get about 400 alleged child pornography sites. Many through CDNs, the complete backend of bulletproof ISPs complied with the orders by using IP block- hosts can be moved to cloud services. Platforms such ing. However, more than 1 million non-targeted web- as DigitalOcean or Amazon Web Services provide sites were blocked as collateral damage, because they various services ranging from Virtual Private Servers shared some IP space with addresses located on the to storage infrastructure. With serverless computing, blacklist. In 2004, the District Court for the Eastern a bulletproof host does not have to run a physical District of Pennsylvania invalidated Pennsylvania’s server, and infrastructure-as-code makes it easier to redeploy nefarious services on a different platform if 17 Ibid.

9 law and ruled that it violated the First Amendment South Korea has been implementing SNI-based filter- [22]. ing in 2019 to block sites served on HTTPS[23]. The dual of collateral damage is the problem of un- In addition to that, in order to resolve the name derblocking. It is extremely difficult to block exactly of a TLS-enabled server, a client has to do unen- the right set of IP addresses used by a BPH. Simi- crypted DNS queries, that can also be analyzed and larly, some domain name extensions or subdomains intercepted. However, encryption is being deployed might go through DNS-level filtering. Content filter- to increase privacy and security of DNS resolutions, ing based on URL also suffers of underblocking, es- with DNS-over-HTTPS (DoH). With DoH, the DNS pecially when several strings of ASCII characters can request between the client and the DoH resolver is be used to request the same resource. Indeed, dif- encrypted using HTTPS. Mozilla has been deploying ferent types of encoding and obfuscation techniques DoH with Cloudflare DNS at the application level provide URL evasion tools[7]. within Firefox in 2018[24] and in November 2019 Mi- crosoft announced their intent to support DoH at Legal implications When taking action against the operating system level [25]. Traditional DNS- BPHs, the possibility of collateral damage being based content filtering techniques are not applicable caused to innocent parties is always salient. One to DoH-encrypted traffic, and this is one of the rea- prominent example was in 2004, when Pennsylvania sons why American ISPs have been arguing against issued approximately 500 blocking orders, under a the widespread deployment of DoH[26]. Pennsylvanian law targeting child pornography, that targeted fewer than 400 alleged child pornography 3.3.3 Machine learning over encrypted traffic sites, but ended up blocking more than 1 million inno- No matter the extent to which traffic is encrypted, cent websites. This was because targeting was based some metadata features can be leveraged for analy- on IPs, and internet service providers “filtered” these sis and content filtering. For example, it is possible IPs from users’ access. Unfortunately, several web- to use machine learning to perform network analy- sites sharing the same server share the same IP. A sis on encrypted traffic and classify service providers district court later found that the law was in violation without directly exposing the privacy of users. Typi- of the First Amendment, and Commerce Clause[22]. cal features include the number of packets, their size, duration, TCP flags or TLS features. Other privacy- 3.3.2 Traffic filtering and encryption preserving classification techniques such as private The filtering techniques described above rely on the decision trees or secure naive Bayes classifiers are di- detection of predefined patterns in the traffic going rectly performed on encrypted data instead of clear- through ISPs. Encrypted connections make it impos- text metadata [27], but as of today their interest is sible for a third-party to examine the content of the mainly theoretical. communications between a client and a server. For Machine learning classification over encrypted traf- example, some techniques such as keyword filtering fic can focus on different granularity levels. The and URL filtering do not work on HTTPS traffic, i.e. coarsest classification tasks simply aim at separating HTTP traffic encrypted using Transport Layer Secu- cleartext traffic from encrypted traffic, for example rity. with simple entropy-based methods [28]. On a higher However, even if the content of the communication level, it is possible to perform traffic characterization is encrypted with TLS, some parts of the traffic are to identify broad traffic categories such as FTP, P2P left in clear. For example, to initiate a TLS hand- or VoIP, while application identification aims at clas- shake with a server, the client sends an initial un- sifying which particular application a client is run- encrypted request using the Server Name Indication ning [29]. Finally, fine-grained analysis can classify (SNI) TLS extension. It is possible for a third-party traffic coming from a given application into precise to examine and intercept such requests: for instance, buckets, such as keywords or application-specific user

10 actions. For example, [30] applies classification tech- did not disclose the technical details of how they took niques on the size of TLS-encrypted Google Images down this peer-to-peer system. More generally, peer- requests in order to detect when some particular key- to-peer file sharing is vulnerable to mass network words are searched. surveillance [33]. However, this type of classification method requires important amounts of training data closely tied to a 3.4 Empowering web content and web specific application that can evolve over time. A first services providers approach to quickly address protocol-specific patterns is to train classifiers on synthetic datasets gen- Three major issues will be key to tackling this is- erated from expected distributions of behaviors. For sue given the new importance of web content and example, [31] shows how to classify user actions on services providers: attribution, verification, and self- the encrypted messaging application Signal – for ex- regulation. ample, tell whether a user sends a regular message, Bulletproof hosts have repeatedly adapted to avoid an image or a location – from an artificially-generated law enforcement or security oversight. The new oper- dataset. Real-world training data for traffic analysis ational model described above complicate this chal- can otherwise be obtained from ISPs, corporate net- lenge. However, the security community has proven works or governmental agencies, such as the Cana- itself capable of keeping pace with the criminal un- dian Institute for Cybersecurity [28]. derground. One 2017 study on bulletproof hosting was able to build classifier method capable of detect- 3.3.4 Censorship-resistant architectures ing malicious bulletproof hosting sub-allocations with 98% recall and a 1.5% false discovery rate[34]. While Some advanced bulletproof hosts are not using the this approach assumes that upstream providers accu- protocol stack presented above, and are hence invul- rately update their WHOIS records when they dele- nerable to traditional blocking techniques. For ex- gate network blocks and that the bulletproof host- ample, some BPHs provide support to setup Tor hiding operator functions as a reseller of the upstream den services, that are using a censorship-resistant and provider, it shows significant promise [34]. In short, anonymity-preserving onion routing technique. It is if there is no silver bullet, neither is the problem in- possible to tackle these advanced techniques with dif- soluble. The security and technical communities will ferent strategies[32]: several Tor marketplaces have simply need to work hard to stay one step ahead of been taken down after complex investigations allow- malicious hosts. ing authorities to raid the servers and their opera- Second, a method that relies on compliance with tors. However, it is worth noting that in spite of some takedown requests at the level of the provider will technical similarities, Tor hidden services and peer- place considerable strain on providers. Most well- to-peer sharing networks do not correspond to BPHs intentioned hosting providers do not have the re- per se, but are rather censorship-resistant techniques sources to staff a robust abuse department that can that can be used by a certain class of BPHs. respond quickly to complaints coming from outside Peer-to-peer communication protocols and dis- the company[35]. For example, a 2018 study by tributed architectures do not have a single point abuse.ch found that there are between 4,000-5,000 of failure nor easily-filtered bottlenecks. Some net- active malware distribution sites each day, yet take- works C&C use peer-to-peer channels to communi- downs take an average of more than eight days[36]. cate orders instead of using fixed domain names that Providers face two obstacles. First, they do not are easy to filter at the DNS level. For example, have a strong economic incentive to adapt. The more Gameover Zeus master servers were sending orders content they host on their infrastructure, the more through a layer of infected machines acting as prox- revenue that comes in. Second, even well-resourced ies, that were themselves communicating to the rest abuse departments face significant challenges in es- of the botnet in a peer-to-peer fashion [9]. The FBI tablishing the legitimacy of the requests they receive.

11 The private sector has nonetheless shown more po- 3.6 Leveraging international law tential for self-regulation than is often recognized. In conjunction with the above technical measures, we For example, a qualitative study commissioned by the argue that legal tools need to be leveraged to enlarge UK Home Office in 2016 found law enforcement in the the policy space for governments to take legitimate United Kingdom to be far less efficient at implement- action, and to shrink the space within which BPHs ing content takedowns than commercial firms[35]. have to operate. That general legal principles apply While the study is limited in scope, it yields sugges- to cyberspace is no surprise18. The same is true in tive results that likely apply to other jurisdictions. international law19. In the absence of legally bind- Often, law enforcement simply lacks the skills and ing international norms that restrict BPHs, we ar- knowledge necessary to handle takedown requests, gue for a jus cogens and erga omnes norm - in other which require a degree of specialization and technical words, a peremptory norm - that would apply to know-how better suited to the private sector. BPHs of ‘Very High’, ‘High’, and more controver- sially, ‘Medium’ toxicity levels. One of the crucial 3.5 Targeting customers: transaction reasons why BPHs attract a constant patronage is tracing and traffic analysis that they market their lack of concern towards hosting such toxic content, that are almost certainly ille- Instead of blocking the bulletproof host, one solu- gal in the jurisdictions in which customers reside[38]. tion is to track the requests to identify who are the It is rare that such content is not prohibited, or pro- customers of bulletproof services. Then, it is pos- hibited without a compelling reason. Given these sible to target these customers according to their factors, consensus is much more likely to crystallise national regulation or disrupt their operations. Un- around existing practices and understandings. der this cross-border approach, an Australian send- How would such a norm form? Of the various ing spam to American companies through the ser- sources of international law, as listed in Section 38(1) vices of a Russian BPH could be prosecuted under of the Statute of the Permanent Court of Inter- the Australian regime, and the nefarious activities national Justice20, we focus on customary interna- could be curbed without having to take down the tional law (CIL). Peremptory norms were implied to Russian provider. This approach shifts the techni- be a species of customary international law, in the cal challenge from blocking to traceability. Efficient Barcelona Traction case21. CILs form when a major- traceability in cyberspace is difficult in general, but ity of States have shown evidence of believing them- some technologies such as cryptocurrencies and peer- selves bound by such a norm 22. Evidence can take to-peer protocols—even though they are associated two forms: state practice, and opinio juris23. State with privacy and anonymity—are more easily trace- able. 18Easterbrook, Frank H. (1996), Cyberspace and Transaction tracing, such as bitcoin transactions, the Law of the Horse, University of Chicago Le- gal Forum: Vol. 1996 , Article 7. Available at: can be employed to track down the service providers https://chicagounbound.uchicago.edu/uclf/vol1996/iss1/7 and their customers. Tracing and analyzing the 19“International Law and State Behaviour in Cyberspace blockchain transactions can assist in mapping out the Series- Compendium of Regional Seminars,” United Nations contributors and users of the website. More gener- Institute for Disarmament Research (UNIDIR) Resources. 20Section 38(1) Statute of the Permanent Court of Interna- ally, traffic analysis can be used to impose sanctions tional Justice, available at https://www.icj-cij.org/en/statute. against content users and the parents of the content 21In the course of discussing erga omnes norms. providers. For example, the French HADOPI law re- 22North Sea Continental Shelf (Federal Republic of Ger- quired ISP participation to monitor and track peer- many/Denmark; Federal Republic of Germany/ Netherlands), ICJ Reports (1969) 3; Arrest Warrant of 11 April 2000 (Demo- to-peer activities, and ultimately imposed sanctions cratic Republic of the Congo v. Belgium), ICJ Reports (2002) to individuals in case of repeated infringements[37]. 3; Jurisdictional Immunities of the State (Germany v. Italy: Greece intervening), ICJ Reports (2012) 99. 23Ibid.

12 practice refers to the actions that states take, while these instruments, against BPHs. Since derogation opinio juris refers to "a subjective obligation, a sense by States is illegal (jus cogens), and all States have on behalf of a state that it is bound to the law in ques- an interest in their enforcement (erga omnes), tak- tion", as expressed through its representatives[39]. ing extraterritorial and collaborative action would be The upshot is that a State with a history of persis- justifiable. Coupled with the fact that such bullet- tent objection as evidenced by these two forms may proof hosts now reside in a relatively small minority not be bound by such a norm; this is not the case for of countries27, we argue that States could thus jus- peremptory norms24, that bind states regardless of tify effective retorsionary28 measures or even some whether they have evinced any objection. One the- countermeasures29, with a view to taking down or ory of formation of CIL posits that there is a sliding removing access to BPHs. The difference between scale of evidence: the more state practice there is, the two types of measures lies in the former involving the lesser the amount of opinio juris that would be actions that do not entail interference with the tar- required to be demonstrative of such belief, and vice get State’s rights, while the latter involves interfer- versa25. In other words, the more states say they ence. While such an invocation is without precedent, are bound by a supposed norm, the less they have the transnational and privatised nature of the inter- to do to demonstrate that they consider themselves net is also unprecedented. We therefore argue that bound. The converse is also true. These norms are there is sufficient and justifiable space for a State to usually norms that States can agree upon in spite of take more muscular action, especially where a BPH their distinct and often diverging national interests, is affecting several States. That is not to say action because of how universally abhorrent the conduct in should be unilateral, or undertaken without notify- question is considered to be. ing the concerned states. It would take a deft piece The prohibition against torture and genocide are of international diplomacy to build international rap- examples of peremptory norms that have received in- port amongst states, but also establish global leader- ternational consensus, and are therefore binding on ship as a result. Such a State would be careful to all States, regardless of whether they have evinced point out that the high threshold posed by ’High’, an existing objection to the norm26. It would be dif- and ’Very High’ toxicity levels had been met, avoid- ficult to imagine a situation where countries evince ing any debate about the different levels of comfort an objection to genocide - the same could be argued that different cultures might have about the same for child pornography and ‘High-Very High’ toxicity. content. This has the added advantage of allowing At ‘medium’ toxicity and below, such harms would states with legal regimes that allow more muscular probably never reach the same status, for lack of enforcement action to do what they can, while other international consensus. Arguably, States have an states are in the process of implementing measures. interest in changing the status quo which is char- This is especially important were a multilateral con- acterised by slow resolution of the problems posed vention to be pursued. States would need time to by BPHs. There is some evidence of such consen- negotiate, sign, and finally ratify (or in other words) sus forming in the Paris Call and Articles 9-11 of the implement such a convention. This would take years, Budapest Convention. These international legal in- while the process we argue for would facilitate action struments meet the definition of opinio juris; it would 27Bobbie Johnson, “Internet pirates find ’bulletproof’ be interesting to monitor the first steps taken under havens for illegal file sharing”, 5 January 2010, The Guardian, available at: https://www.theguardian.com/ 24De Wet, Erika, Jus Cogens and Obligations Erga Omnes technology/2010/jan/05/internet-piracy-bulletproof; (January 15, 2013). Dinah Shelton (Ed), The Oxford Hand- ChunWei,‘‘ClusteringSpamDomainsandHosts: book on Human Rights (OUP 2013), available at SSRN: Anti-SpamForensicswithDataMining’’,2010, https://ssrn.com/abstract=2279563. UniversityofAlabamaatBirminghamDissertation. 25Frederic L. Kirgis, “Custom on a Sliding Scale”, 81 Am. J. 28Thomas Glegerich, Entry on Retorsion, March 2011, Max Int’l L. 146 (1987). Planck Encyclopedias of International Law. 26Ibid, at n. 54. 29Ibid.

13 while domestic legislation and enforcement across the push for a multilateral convention that would broadly world plays catch-up to the reality of the problem. adopt the recommendations of the 2013 UNODC Re- We also argue that breach of such a norm would port34. render States responsible e.g. under Article 12 of the ILC Draft Articles on State Responsibility 30. For ex- Censorship and Content Regulation Issues ample, it has been argued by the Council on Foreign Another particularly difficult concern associated with Relations, a prominent US think-tank, that policy tackling BPHs is possible censorship. In the US makers ought to establish a principle that States are context, two pertinent rights are the First and responsible for the harm botnets situated within their Fourth Amendments. Another crucial issue would be territories cause on others.[40]31 The same reasoning whether the persons which law enforcement pursue would apply to BPHs within their jurisdiction - espe- action against are American citizens or not. The for- cially if States have been notified. As put bluntly by mer would clearly be protected, but the latter may Healey and Knake: “When governments are unable or only be if they are on US territory. More broadly, unwilling to be responsible, other states may be jus- other jurisdictions would have concerns about free- tified in taking action, in or out of the cyber domain, doms of information and speech. This stems from to thwart cross-border effects.”32 From a retorsion- the fact BPHs may also host legitimate content, or ary perspective, States can and should do everything that the lines between prohibited content and politi- within their power to block other parties or entities cally unwelcome content is often more blur at lower from carrying out those activities from within their toxicity levels. However, we argue that this is a bal- territories, especially where a target state’s right are ancing exercise that has been carried out effectively not interfered with.33 in reality, and is much less of a concern at higher Regulating BPHs offers an opportunity to recog- toxicity levels. nise that States owe international obligations towards From the First Amendment perspective, it has regulation of the internet to regulate the egregious been argued that US courts already undertake some harms that could be caused to their citizens, such as form of “free-speech consequentialism”: a “weighing those mediated by BPHs. First, this is not without whether a particular kind of speech causes harms that precedent. Supra-nationally, this is evident in the ju- outweigh its benefits, or asking whether the govern- risprudence of the European Court of Human Rights. ment has especially strong reasons for regulating par- In K.U. v. Finland, an individual right to protec- ticular kinds of speech”35. In the paper, the author tion against cybercrime was recognised. Second, it 34 can seed state practice necessary for the international United Nations Office on Drugs and Crime, “Compre- hensive Study on Cybercrime- February 2013,” pp. ix-xv norm to crystallise. available at https://www.unodc.org/documents/organized- We also argue that States should take creative ad- crime/UNODC_CCPCJ_EG.4_2013/CYBERCRIME_STUDY_210213.pdf. vantage of existing mutual legal assistance provisions Recommendations include: i) clarifying international stan- in the UNODC and UNCAC. On top of encouraging dards of investigations involving extra-territorial data, so that they do not impinge on sovereignty; ii) formulating States to do so, we also recommend continuing the international model provisions for powers of investigations

30 and criminalisation of key acts (evident in the Paris Call); With the caveat that they are only indicative of interna- iii) capacity building to familiarise law enforcement agencies tional law. 31 in developing countries to utilise existing domestic laws to Jason Healey and Robert K. Knake, Council Special Re- undertake more effective action while safeguarding human port No. 83, November 2018 titled “Zero Botnets- Building a rights and privacy; and v) establishing consensual and effec- Global Effort to Clean Up the Internet”, Council on Foreign tive bases for taking jurisdiction in criminal matters, basing Relations (CFR), at Foreword by Richard N. Haass, President, it, for instance, on the objective territoriality principle or CFR, 2018, available at: https://www.cfr.org/report/zero- the substantial effects doctrine. This is especially important botnets. 32 where different states may concurrently exercise jurisdiction. textitIbid, Introduction by Jason Healey and Robert K. 35 Erica Goldberg, “Free Speech Consequentialism”, Knake. 33 Columbia Law Review, Vol. 116, 2016, August 2015, avail- Ibid, Foreword by Richard N. Haass. able at SSRN: https://ssrn.com/abstract=2645869.

14 situates the discussion in more controversial forms tions. Rather, complaints are lodged by the public, of speech such as campus hate speech, trigger warn- including ISPs and content hosts. This presents an ings, revenge pornography, etc. We submit that at opportunity for collaboration between the public and ‘Medium’ to ‘Very High’ levels, the harms clearly out- private sectors. weigh any potential benefits, and that the concomi- With regard to geolocation, the Australian regime tant balancing exercise would be effectively carried takes varied actions depending on the location of the out by courts and government officials. This is not to negative content in issue. For example, if the content say that the First Amendment case-law is clear and is found to be hosted in Australia, a take-down no- settled, free-speech issues as engaged by BPHs could tice can be sent to the host - failure to comply with be clearly be vexing ones, especially at lower toxic- such a notice constitutes a breach of the law and al- ity levels, or where BPHs support ’good’ uses such lows the regulator to impose fines. With regard to as whistle-blowing activity, or civil society activities content hosted outside the jurisdiction, the regulator that may not be kindly looked upon by certain gov- will notify ISPs, who are to take action based on the ernments. For reasons of brevity, this paper does not relevant code of practice within their territory. For- delve further into the potential free-speech issues in eign law enforcement agencies can also be notified such cases. when appropriate. These arrangements are entirely Secondly, Courts already utilise existing First voluntary. Amendment exceptions, such as the “speech integral Another effective aspect of the regime is the duty to illegal conduct” exception36. The Roberts court imposed on ISPs and content hosts to police one an- has used it to “justify prohibitions on distributing and other for violations and report to the regulator ac- possessing child pornography, on soliciting crime, and cordingly - or ‘peer-to-peer monitoring’. Notably, all on announcing discriminatory policies”. While there decisions taken by the regulator against violations are is rightly a debate to be had whether the exception ultimately subject to judicial review. We submit that is “well-defined” and “narrowly limited”, as the US such a regulatory regime strikes a good balance be- Supreme Court asserts, we submit that where ade- tween safeguarding rights and freedoms of expression quate technical measures are taken to avoid collat- and information, and responding to the harms caused eral damage, this exception provides a well-grounded by BPHs. basis for more assertive legislation against BPHs. Elsewhere, codes and restrictions instituted by From a Fourth Amendment perspective, our brief national regulators are also instructive. These re- review of the literature suggests that law enforce- quire ISPs, content hosts and carriers to educate ment agencies have been very scrupulous with re- users about content. Some regulators have a ‘fam- specting Fourth Amendment protections when under- ily friendly’ labelling scheme if they comply with cer- taking search and seizure operations, especially where tain content regulation requirements. This includes American citizens are concerned. prior age verification mechanisms and warnings that Looking farther afield, we found that other States aim to shield children from undesirable content. The have been successful in carrying out the balancing result is an indirect “blacklisting” should ISPs and exercise. We argue that the best features of these content hosts fail to comply. The downside is that regimes could be harnessed in any domestic solu- service providers, including content distribution net- tion. For example, the Australian regulatory regime works like Cloudflare, or large scale webhosts like is complaints-based. This means that the regulator Amazon Web Services would be affected, because does not set out to search for internet content viola- many of their customers might be blocked. We ar- 36Giboney v. Empire Storage & Ice Co. (1949); Volokh, gue that this is a boon - service providers would have Eugene, The ’Speech Integral to Criminal Conduct’ Excep- an added impetus to respond to complaints from the tion (November 15, 2015). Cornell Law Review, Forthcoming; public or government agencies to avoid the risk of UCLA School of Law Research Paper No. 15-50, available at blocking. This would not be excessive given the con- SSRN: https://ssrn.com/abstract=2696856. tinued pervasiveness of the problem, the egregious

15 harms caused, and the hitherto patchy approaches. the CLOUD Act38. One such agreement has been fi- Service providers would also have an added impetus nalised between the US and UK, and will form the ba- to implement search-and-filtering technologies - these sis of expedited investigations especially when com- are already prevalent in avoiding copyright infringe- pared to current timelines under the current MLA ment. It is arguable that the harms caused by BPHs regime. Negotiations are also underway with the Eu- are more egregious in some respects than the com- ropean Union and Australia. Using such a mode of mercial harms caused by copyright infringement. cooperation can therefore be effective, but with the That these measures are all voluntary underscores risk that there may be a multitude of bilateral agree- the fact that effective action can still be taken where ments that criss-cross and complicate the regulatory legislation is unlikely to be passed anytime soon. environment, increasing administrative costs39. We also recommend another solution that takes advantage of the prominence of BPHs, especially if the Registry provides a list of BPHs. ISPs should include 4 Registry-based solutions new Terms and Conditions that restrict accessing of Having explored the various issues, we now study ex- known bulletproof hosts by default. Customers would isting registry models before proposing a model of have to take a proactive step to turn off the default. our own. These registries have been successful in Where possible, regulators should also encourage and contributing to the effort to tackle such varied issues compel ISPs to include such terms. Such ‘nudges’ as spam mail, and malware. We believe the same have been effective in other areas of regulation37. strategies and lessons learnt from such initiatives can States should therefore be compelled, whether po- be applied in the realm of BPHs. litically, or legally, to take the maximum level of intervention that would be consistent with their current regulatory posture. In the US, this would be un- 4.1 Nonprofit blacklists likely to engage issues of First Amendment or Fourth We begin by studying the following nonprofit black- Amendment concerns, given that this is a default op- lists: i) StopBadware; and ii) Spamhaus. tion that subscribers can opt-out of, and ultimately protects subscribers from malware etc(which is typi- cally also hosted on such sites). StopBadware. StopBadware is a non-profit organization fighting against badware websites, a broad category of (intentionally or not) malicious websites Domestic-international interface Supplement- that distribute malware or unwanted software. The ing the above, domestic regimes can also facilitate main mission of StopBadware is to provide data more comprehensive forms of international coopera- about badware websites[41]. tion. Such regimes could capitalise on the Registry. For instance, in the US, the CLOUD Act provides 38Jennifer Daskal and Peter Swire, “UK-US for Executive Agreements with other States that pro- CLOUD Act Agreement is Finally Here”, Law- fare, 8 October 2019, https://www.lawfareblog.com/ vide a “mechanism for law enforcement in either the uk-us-cloud-act-agreement-finally-here-containing-new-safeguards; U.S.[other country]...to request data from a service Notably, the Executive Agreement with the UK contains safe- provider in the other country without having to go guards over and above the CLOUD Act. through the laborious mutual legal assistance process 39The spaghetti bowl effect caused by the criss-crossing and multiplying of bilateral trade agreements was said to do so”, once Congress is assured that the part- to have counterproductively increased the administrative ner country has at least the same protections as in costs of harnessing preferential tariffs, when compared to the multilateral WTO regime: see, Jagdish N. Bhag- 37 Thaler, Richard H.,Sunstein, Cass R. (2008) Nudge: im- wati, “US Trade Policy: The Infatuation with FTAs”, proving decisions about health, wealth, and happiness, New Discussion Paper Series No. 726, April 1995, available at: Haven: Yale University Press https://academiccommons.columbia.edu/doi/10.7916/D8CN7BFM.

16 For example, StopBadware’s Clearinghouse pro- private initiative does not aim at specifically deal- vides a binary classification (badware or not bad- ing with bulletproof hosts or law enforcement issues. ware) for URLs, IP adresses and AS numbers [42]. Webmasters who manage a legitimate website that In order to update this database in real time, Stop- has been classified as “unsafe” have to find the source Badware does not conduct their own scanning but of the problem, fix it and request a review by Google rely on data provided by three industrial partners: [46]. This process can be lengthy even if the site was Google, ThreatTrack Security, and NSFOCUS. incorrectly labelled, and offers no accountability from Another mission of StopBadware is to encourage Google. While the service fills a much needed gap, information-sharing efforts across industry, through critics may criticise the long arm of what is essen- data exchange programs. This third-party position tially a private company with more opaque processes is especially interesting when it comes to appealing underlying the review process. blacklisting decisions. Indeed, StopBadware provides The Safe Browsing blacklist is also accessible to independent reviews of the data provided by their third-parties, thanks to an API provided by Google. partners such as Google [43], for example when a Major web browsers such as Safari or Firefox of- webmaster thinks that their website has been mis- fer built-in protection that relies on Google’s Safe classified. Browsing [47][48]. These browsers also rely on other sources such as Tencent Safe Browsing or StopBad- Spamhaus. The Spamhaus Project is an interna- ware. Such a dependence of the ecosystem on a few tional nonprofit organization that mainly provides private actors poses privacy concerns: even if URLs DNS blocking lists for spam and related threats. are usually hashed and truncated before being sent to They also provide data and solutions to fight phish- the blacklist server through the API to check for a po- ing, malware and botnets. The lists maintained by tential collision, it is not sufficient to achieve high pri- Spamhaus are widely used by anti-spam services and vacy guarantees. For example, [49] describes URL re- Internet companies.[44] identification techniques and browsing history track- Spamhaus DNS blocking database is maintained by ing through API calls to Yandex and Google Safe automatic traffic analysis and is manually curated by Browsing lists. Spamhaus researchers. Like other reputation systems such as Google’s ranking system, the criteria used to 4.3 Proposal for an international reg- compute the reputation of a given email sender are istry not public, to avoid issues similar to fraudulent search engine optimization. To solve the attribution, verification, and incentives problems facing legitimate providers, as well as the 4.2 “Safe browsing” lists legal and political challenges posed by bulletproof hosts in general, we recommend developing an inter- Next, we analyse the converse of black lists: safe- national registry to collect, aggregate and publish in- browsing lists. Google maintains a service called Safe formation on bulletproof hosting services. In suggest- Browsing, that aims at listing unsafe websites. This ing such a model, we use the successful model of the list is embedded into other Google services such as International Atomic Energy Agency. It was formed Android or Chrome, so that a notification is displayed in 1955 despite objections from the then USSR to in- to users to warn them when they try to access a po- ternational custody of nuclear fissile material if the tentially harmful resource. This on-device warning US did not first agree to disarm itself of nuclear system–instead of direct content blocking–is imple- weapons. We may observe parallels to the Cold War mented on approximately 2 billion devices [45]. in cyberspace: states may want to ensure equality The list of unsafe websites contains two main cate- of superiority of arms as it relates to cyberweapons, gories: phishing and malware. This security-focused or at least preservation of adequate space to conduct cyber-operations. The way the deadlock was resolved

17 was to envision the IAEA as a clearinghouse for inter- 265 security researchers [36]. In addition, Spamhaus national transactions. We argue that an international has been able to earn the trust of ISPs, security ven- registry handling such BPHs could do the same. dors, internet companies, and the global technical It would be given a broad-based legitimacy to ac- bodies in terms of its ability to identify malicious web cess and aggregate several streams of data, including domains. Other entities, like the Cyber Peace Insti- existing ones such as Spamhaus, or even governmen- tute, the Global Commission on the Stability of Cy- tal ones - subject of course to the appropriate privacy berspace, and the United Nations, are working hard and confidentiality agreements. The registry would to develop norms to regulate cyberspace. collect data from the private and non-profit sector, Crucial to such a registry will be the estabishment partner governments, international technical bodies, of proper governance and procedures to elect its exec- and an in-house technical staff in order to shine a light utive offices. In this respect, we believe the approach on the most abusive hosting providers and network taken by agencies in the United Nations system or blocks. Such cooperation between an international other international civil service agencies have struck organisation and both public and private sectors are a practical balance that facilitate the achievement of not unheard of: international organisations such as their various objectives. While this merits another in- the IAEA and UNICEF play similar roles, and bridge depth study, we argue that emulating the governance efforts between various stakeholders in their respec- model of one or an amalgamation of these interna- tive fields. tional organisations would be effective for the setting Using the aggregated data, this agency, like the up of the international registry. IAEA would serve as an unbiased body with the authority to investigate claims brought by constituents. Envisioned effects Creating an international en- Its determinations about the complicity of compa- tity responsible for monitoring hosting providers will nies, persons, and local and federal government in address some of the major problems identified above. malicious content hosting could be used as a basis It will help the international community, the private for blacklisting, takedown action, legal or economic sector, and local and international law enforcement sanctions, and even offensive cyber action. agencies identify abusive hosts and verify malicious Any supervisory body that operates on a global content. In so doing, it will also supply content scale will face significant challenges resulting from providers with an incentive to do so: Those providers the need to forge uniform standards of transparency that fail to address concerns quickly will suffer ad- and content regulation, while being capable of cut- verse reputational consequences. Finally, an interna- ting across the differing legal regimes, cultural prefer- tional monitoring body can draw attention to nations ences, and political prerogatives of constituent states. and sub-national jurisdictions that repeatedly fail to Hence, the proposed international registry should monitor or contain abusive hosting. When it comes only spotlight providers that enable the propaga- to the world’s most abusive hosts, the registry can tion of “high” and “very high” toxicity content: child provide a basis for cooperative action - whether it be pornography, malware, botnet C&C, and exploit kits. legal sanction or a forcible police action. While countries evince a significant degree of dis- A focus on information sharing and transparency agreement when it comes to the acceptability of con- has two advantages over traditional methods. First, tent at the lower end of the toxicity scale, such as it avoids the problems of a whack-a-mole approach, spam or various forms of free speech, the interna- where lengthy legal processes and costly enforcement tional community will have an easier time if it focuses actions targeting individual companies merely incen- on these four content types. tivize others in the hosting ecosystem to fill a gap left Precedents for this model have already shown im- by takedowns. Second, it enables well-intentioned mense promise. For example, abuse.ch’s URLhaus companies and states to do a better job of self- project reported taking down 100,000 malware distri- regulating, while putting more pressure on those who bution sites over a period of 10 months with as few as

18 regularly fail to respond to takedown requests. 4.4 The problem of identification and The registry is not without drawbacks. Primarily, prescription we anticipate that the agency will cause unease in nations that have consistently taken a position against Prior to any attempt at blocking or taking down bul- any violation of state sovereignty, no matter how con- letproof hosts, one must solve the problem of identifi- sensual or insignificant. A few difficulties are appar- cation and prescription: how to create and maintain ent: what would such countries gain by contribut- a consistent list of BPHs that represents as accurately ing to such efforts? How genuine would their efforts as possible the scope of publicly available malicious be? How would the agency strike a compromise that services? Even with a broad support from public and does not end up rendering the agency ineffectual? private sector actors to curate a list of hosts, and le- We argue that even states like Russia and China gal tools to enforce takedowns, a Register without an will have a vested interest in supporting the interna- efficient prescription method is doomed to fail. For tional registry. First, doing so will give those coun- example, trivial IP blacklisting is easily defeated by tries more control over this process than they cur- IP rotation, proxies and other escaping methods de- rently have over entities like abuse.ch and Spamhaus, scribed above. whose work is already used by Western companies In addition to IP, domain name and keyword-based and governments to justify takedown action or black- blacklists, state-of-the-art content filtering software listing. Second, participation in such an organisation aggregate various heuristics to curate and update will give these countries an additional lever of ’soft- their lists in real time. For example, SmartFilter is a power’. Countries often pursue strategies both over- private company offering services to states and cor- and under-the-table, and having a hand in the work of porations to block specific categories of websites such such a ’clearinghouse’ would arguably be better than as gambling or pornography[50]. However, the meth- having no say at all. Ostensibly, countries like Russia ods used to create the list remain confidential, and and China are more concerned with internal informa- the list of blocked content is not available, even for tion flows. Counter-intuitively, participation in such companies or governments purchasing the services of an agency would give them greater insight into data SmartFilter. flows coming into and out of the country, with re- Good heuristics to identify and register bulletproof gard to at least a few types of information. In this hosts must be dynamic: classification tools that rely respect, regulating BPHs might be a good trade-off. only on static descriptors are not efficient to identify These countries would therefore have some interest new content as it is uploaded, or follow the changes in genuinely engaging such a registry. Some measure performed by bulletproof hosts that attempt to es- of politicization in the registry is unavoidable, and to cape blocking. Content analysis can provide high ac- some extent unpredictable as it depends on the polit- curacy in identifying malicious content, but at the ical acumen and the decision-making abilities of the price of reaction speed. A study about pornography agency’s executive officers. This is a typical prob- detection systems[51] gives some heuristic guidelines lem at most agencies in the UN international system. such as analyzed data types, networked groups, mon- In this respect, recruitment of the right talent, a fo- itoring and tracking service, compound words, refer- cus on the mandate of the organisation, as well as ence links and proxy access log mining. This trade- a constant appraisal of its effectiveness compared to off between accuracy and speed of blocking must be private entities will be crucial. Despite these possible considered with regards to the toxicity level and the drawbacks, we argue that a registry will nonetheless scale of BPHs. For instance, very highly toxic hosts go a long way in ameliorating some of the issues in such as child exploitation content providers are not as the current state of play. widespread as general adult content providers. As a result, human curation techniques can be more easily applied to this type of hosts. Data analysis and machine learning techniques pro-

19 vide technical tools to extract useful elements from there are policy, technical, and legal challenges, we masses of information without having to choose a have outlined how these can be resolved more effec- limited number of clever heuristics or hand-picked tively by adopting a range and variety of measures indicators. For example, Internet traffic can be ac- that are greater than the sum of its parts. curately classified thanks to machine learning techniques such as SVMs[52]. In this 2008 study, researchers use high-dimensionality datasets from which they select a large number of optimized features (about 200 features for some algorithms) and obtain 97% accuracy on broad categories such as “mail”, “web”, “P2P” or “Attack”. The evolution of routing policies of a given Au- tonomous System (AS) is also a useful source of information to identify bulletproof hosting. ASes are groups of connected IP prefixes managed by a single entity with a given routing policy. Monte et al. have proposed ASwatch, a reputation system[53] to identify ASes directly controlled by cybercriminals: they can detect up to 93% of malicious ASes with a 5% false positive rate. Interestingly, this method relies on learning control-plane behavior, that is to say routing strategies (e.g. aggressive rewiring, BGP routing dynamics, partition and rotation of IP space) instead of data-plane behaviors (e.g. the type of traffic emitted by a given IP address). As a result, ASwatch can detect a malicious AS before any attack has been emitted, and it can tell the difference between a legitimate but abused AS and a malicious AS controlled by cybercriminals.

5 Conclusion

With the growth of BPH services for illicit content, it is evident that new solutions are needed against the most toxic levels of hosted content. These solutions can no longer take a whack-a-mole approach, and must address the problem from the perspective of the larger ecosystem in which BPHs operate. An international registry will allow for the tracking and monitoring of high and very high toxic content. The focus on information sharing and transparency across states and actors will incentivize quicker and more effective takedowns that will tackle existing BPHs at all stages, and prevent their future growth. Though

20 References [8] Hendrik Schulzeg Klaus Mochalski. Deep Packet Inspection: Technology, Applications & [1] Brian Krebs. “Shadowy Russian Firm Seen as Net Neutrality. url: https://web.archive. Conduit for Cybercrime”. In: The Washing- org / web / 20200211045228 / https : / / ton Post (Oct. 13, 2007). url: https : / / www . scribd . com / document / 32040079 / web . archive . org / web / 20200211043737 / Whitepaper - Deep - Packet - Inspection - https : / / www . washingtonpost . com / Technology-Applications-Net-Neutrality. wp - dyn / content / article / 2007 / 10 / Accessed: 11 February 2020. 12 / AR2007101202461 _ pf . html (visited on [9] Department of Justice. Declaration of special 02/11/2020). agent Elliott Peterson in support of applica- [2] Digital Shadows Blog. Criminal Services - Bul- tion an emergency temporary restraining order letproof Hosting. url: https://web.archive. and order to show cause re preliminary injunc- org / web / 20200203162852 / https : / / www . tion. url: https://web.archive.org/web/ digitalshadows.com/blog- and- research/ 20200211045900 / https : / / www . justice . criminal-services-bulletproof-hosting/. gov / sites / default / files / opa / legacy / 21 January 2016. Accessed: 03 February 2020. 2014/05/30/declaration.pdf. 30 May 2014. [3] Justin Paine Doug Kramer. Cloudﬂare’s Re- Accessed: 11 February 2020. sponse to CSAM Online. url: https://blog. [10] Tom Burt. New Steps to Protect Customers cloudflare . com / cloudflares - response - From Hacking. url: https://web.archive. to-csam-online/. 6 December 2019. Accessed org/web/20200211041816/https://blogs. 20 February 2020. microsoft.com/on- the- issues/2019/03/ [4] Max Goncharov. Criminal Hideouts for Lease: 27 / new - steps - to - protect - customers - Bulletproof Hosting Services. url: https : / / from-hacking/. 27 March 2019. Accessed: 11 web . archive . org / web / 20200211041133 / February 2020. https://www.trendmicro.no/media/wp/wp- [11] Security Service of Ukraine. SBU jointly with criminal-hideouts-for-lease-en.pdf. 15 foreign colleagues blocks activity of powerful July 2015. Accessed: 11 February 2020. hacker group. url: https : / / ssu . gov . ua / [5] David Fiﬁeld et al. “Blocking-resistant commu- en / news / 1 / category / 21 / view / 6281 # nication through domain fronting”. In: PoPETs .J1jZcicu.dpbs:. 16 July 2019. Accessed 21 2015.2 (2015), pp. 46–64. doi: 10 . 1515 / February 2020. popets-2015-0009. url: https://doi.org/ [12] Maija Palmer. “Rogue states play host to out- 10.1515/popets-2015-0009. law servers”. In: Financial Times (Mar. 15, [6] Hermes Center for Transparency and Dig- 2016). url: https : / / web . archive . org / ital Human Rights. Projects & Technolo- web / 20200211042322 / https : / / www . ft . gies. url: https : / / web . archive . org / com/content/c926b4ec- da25- 11e5- 98fd- web / 20200221151157 / https : / / www . 06d75973fe09 (visited on 02/11/2020). hermescenter . org / home / projects - [13] Malware Bytles Blog. No man’s land: How a technologies/project/. Accessed: 20 Febru- Magecart group is running a web skimming op- ary 2020. eration from a war zone. url: https://web. [7] Richard Clayton. “Anonymity and traceability archive.org/web/20200211042554/https: in cyberspace”. PhD thesis. University of Cam- / / blog . malwarebytes . com / cybercrime / bridge, UK, 2005. url: http://ethos.bl.uk/ 2019/07/no- mans- land- how- a- magecart- OrderDetails.do?uin=uk.bl.ethos.444742. group - is - running - a - web - skimming -

21 operation-from-a-war-zone/. 18 July 2019. [20] Electronic Frontier Foundation. Content Deliv- Accessed: 11 February 2020. ery Networks (CDNs). url: https : / / web . [14] Roger Grimes. “Putting the Kobosh on Spam- archive.org/web/20200216201824/https: Spewing McColo”. In: PC World (Nov. 28, //www.eff.org/free- speech- weak- link/ 2008). url: https://web.archive.org/web/ cdn. Accessed: February 2020. 20200211043928 / https : / / www . pcworld . [21] Matthew Prince. “Terminating Service for com / article / 154639 / mccolo _ takedown . 8Chan”. In: The Cloudﬂare Blog (Apr. 8, html (visited on 02/11/2020). 2019). url: https : / / web . archive . org / [15] Brian Krebs. German Cops Raid ’Cyberbunker web / 20200214051111 / https : / / blog . 2.0,’ Arrest 7 in Child Porn, Dark Web Market cloudflare . com / terminating - service - Sting. url: https://krebsonsecurity.com/ for-8chan/ (visited on 02/14/2020). 2019/09/german-cops-raid-cyberbunker- [22] Center For Democracy & Technology. Sum- 2 - 0 - arrest - 7 - in - child - porn - dark - mary and highlights of the Philadelphia Fed- web - market - sting/. 28 Sep 2019, Accessed eral District Court’s decision in Center For 21 February 2020. Democracy & Technology V. Pappert. Case [16] Jack Goldsmith and Tim Wu. Who Controls No. 03-5051 (E.D. Pa. Sept. 10 2004). url: the Internet? Illusions of a Borderless World. https : / / web . archive . org / web / USA: Oxford University Press, Inc., 2006. isbn: 20200211050059 / https : / / cdt . org / wp - 0195152662. content / uploads / speech / pennwebblock / 20040915highlights.pdf. 15 September 2004. [17] NATO Cooperative Cyber Defence Centre of Accessed: 11 February 2020. Excellence. African Union. url: https : / / ccdcoe.org/organisations/au/. Accessed 21 [23] Sergiu Gatlan. “South Korea is Censoring the February 2019. Internet by Snooping on SNI Traﬃc”. In: Bleep- ing Computer (Feb. 13, 2019). url: https : [18] Arman Noroozian et al. “Platforms in Ev- //web.archive.org/web/20200211050420/ erything: Analyzing Ground-Truth Data https://www.bleepingcomputer.com/news/ on the Anatomy and Economics of Bullet- security/south-korea-is-censoring-the- Proof Hosting”. In: 28th USENIX Security internet- by- snooping- on- sni- traffic/ Symposium, USENIX Security 2019, Santa (visited on 02/11/2020). Clara, CA, USA, August 14-16, 2019. Ed. by Nadia Heninger and Patrick Traynor. [24] Patrick McManus. “Improving DNS Privacy USENIX Association, 2019, pp. 1341– in Firefox”. In: Firefox Nightly News (June 1, 1356. url: https : / / www . usenix . 2018). url: https://web.archive.org/web/ org / conference / usenixsecurity19 / 20200211051026 / https : / / blog . nightly . presentation/noroozian. mozilla . org / 2018 / 06 / 01 / improving - dns - privacy - in - firefox/ (visited on [19] Hadi Zolfaghari and Amir Houmansadr. “Prac- 02/11/2020). tical Censorship Evasion Leveraging Content Delivery Networks”. In: Proceedings of the [25] Kurt Mackie. “Microsoft Talks up WIndows 2016 ACM SIGSAC Conference on Computer Support for Encrypted DNS”. In: MCPMag and Communications Security, Vienna, Aus- (Nov. 19, 2019). url: https://web.archive. tria, October 24-28, 2016. Ed. by Edgar R. org/web/20200211051244/https://mcpmag. Weippl et al. ACM, 2016, pp. 1715–1726. doi: com / articles / 2019 / 11 / 19 / windows - 10.1145/2976749.2978365. url: https:// support- for- encrypted- dns.aspx (visited doi.org/10.1145/2976749.2978365. on 02/11/2020).

22 [26] US Telecom CTIA NCTA. Letter to Congress [32] Lasse Øverlier and Paul F. Syverson. “Locat- about DoH, September 2019. url: https : / / ing Hidden Servers”. In: 2006 IEEE Sympo- web . archive . org / web / 20200211051542 / sium on Security and Privacy (S&P 2006), 21- https : / / www . ncta . com / sites / default / 24 May 2006, Berkeley, California, USA. IEEE files / 2019 - 09 / Final % 20DOH % 20LETTER % Computer Society, 2006, pp. 100–114. doi: 10. 209-19-19.pdf. 19 September 2019. Accessed: 1109/SP.2006.24. url: https://doi.org/ 11 February 2020. 10.1109/SP.2006.24. [27] Raphael Bost et al. “Machine Learning Clas- [33] Michael Piatek, Tadayoshi Kohno, and Arvind sification over Encrypted Data”. In: 22nd Krishnamurthy. “Challenges and Directions for Annual Network and Distributed System Se- Monitoring P2P File Sharing Networks - or curity Symposium, NDSS 2015, San Diego, - Why My Printer Received a DMCA Take- California, USA, February 8-11, 2015. The down Notice”. In: 3rd USENIX Workshop on Internet Society, 2015. url: https : / / Hot Topics in Security, HotSec’08, San Jose, www . ndss - symposium . org / ndss2015 / CA, USA, July 29, 2008, Proceedings. Ed. by machine- learning- classification- over- Niels Provos. USENIX Association, 2008. url: encrypted-data. http://www.usenix.org/events/hotsec08/ [28] Kun Zhou et al. “Practical evaluation of en- tech/full%5C_papers/piatek/piatek.pdf. crypted traffic classification based on a com- [34] Sumayah A. Alrwais et al. “Under the Shadow bined method of entropy estimation and neu- of Sunshine: Understanding and Detecting ral networks”. In: ETRI Journal (Jan. 2020). Bulletproof Hosting on Legitimate Service doi: 10.4218/etrij.2019-0190. url: https: Provider Networks”. In: 2017 IEEE Symposium //doi.org/10.4218%2Fetrij.2019-0190. on Security and Privacy, SP 2017, San Jose, [29] Mohammad Lotfollahi et al. “Deep packet: a CA, USA, May 22-26, 2017. IEEE Computer novel approach for encrypted traffic classifica- Society, 2017, pp. 805–823. doi: 10.1109/SP. tion using deep learning”. In: Soft Comput. 24.3 2017.32. url: https://doi.org/10.1109/ (2020), pp. 1999–2012. doi: 10.1007/s00500- SP.2017.32. 019- 04030- 2. url: https://doi.org/10. [35] Alice Hutchings, Richard Clayton, and Ross J. 1007/s00500-019-04030-2. Anderson. “Taking down websites to prevent [30] Pierre-Olivier Brissaud et al. “Passive Moni- crime”. In: 2016 APWG Symposium on Elec- toring of HTTPS Service Use”. In: 14th In- tronic Crime Research, eCrime 2016, Toronto, ternational Conference on Network and Ser- ON, Canada, June 1-3, 2016. IEEE, 2016, vice Management, CNSM 2018, Rome, Italy, pp. 102–111. doi: 10 . 1109 / ECRIME . 2016 . November 5-9, 2018. Ed. by Stefano Salsano 7487947. url: https://doi.org/10.1109/ et al. IEEE Computer Society, 2018, pp. 219– ECRIME.2016.7487947. 225. url: http : / / ieeexplore . ieee . org / [36] Abuse.ch Blog. How to takedown 100,000 mal- document/8584960. ware sites. url: https://web.archive.org/ [31] Tomas Babej. “Detecting user actions from en- web / 20200211044847 / https : / / abuse . ch / crypted traffic using machine learning”. PhD blog/how- to- takedown- 100000- malware- thesis. Masaryk University, 2017. url: https: sites/. 21 January 2019. Accessed: 11 Febru- //web.archive.org/web/20200216210944/ ary 2020. https : / / is . muni . cz / th / iv9d7 / thesis - [37] Sylvain Dejean and Raphaël Suire. Digital files final.pdf. dealers and prohibition in the context of the French 3 strikes (HADOPI) law. Economics Working Paper Archive (University of Rennes

23 1 & University of Caen) 201406. Center for Re- [46] Google Developers. Request a review. url: search in Economics, Management (CREM), https : / / web . archive . org / web / University of Rennes 1, University of Caen, 20200219042958 / https : / / developers . and CNRS, Apr. 2014. url: https://ideas. google.com/web/fundamentals/security/ repec.org/p/tut/cremwp/201406.html. hacked/request_review. Accessed: 18 Febru- [38] Brian Krebs. Meet the World’s Biggest ary 2020. Bulletproof Hoster. url: https : / / [47] Safari User Guide. Change Security prefer- krebsonsecurity.com/2019/07/meet-the- ences in Safari on Mac. url: https://web. worlds- biggest- bulletproof- hoster/. 16 archive.org/web/20200219045444/https: July 2019, Accessed 21 February 2020. / / support . apple . com / guide / safari / [39] Cornell University Legal Information Institute. security-ibrw1074/mac. Accessed: 18 Febru- Wex Legal Dictionary Definition of Opinio Ju- ary 2020. ris. url: https://www.law.cornell.edu/ [48] Mozilla Support. How does built-in Phishing wex / opinio _ juris _ (international _ law). and Malware Protection work? url: https : Accessed 21 February 2020. //web.archive.org/web/20200130232730/ [40] Richard A. Clarke and Robert Knake. Cyber https : / / support . mozilla . org / en - US / War: The Next Threat to National Security and kb / how - does - phishing - and - malware - What to Do About It. USA: HarperCollins Pub- protection - work. Accessed: 18 February lishers, 2010. isbn: 0061962236. 2020. [41] StopBadware. What we do. url: https://web. [49] Thomas Gerbet, Amrit Kumar, and Cédric archive.org/web/20200221161152/https: Lauradoux. “A Privacy Analysis of Google //www.stopbadware.org/what- we- do. Ac- and Yandex Safe Browsing”. In: 46th Annual cessed: 20 February 2020. IEEE/IFIP International Conference on De- pendable Systems and Networks, DSN 2016, [42] StopBadware. Clearinghouse search. Toulouse, France, June 28 - July 1, 2016. IEEE url: https : / / web . archive . org / Computer Society, 2016, pp. 347–358. doi: 10. web / 20200221161404 / https : / / www . 1109/DSN.2016.39. url: https://doi.org/ stopbadware . org / clearinghouse / search. 10.1109/DSN.2016.39. Accessed: 20 February 2020. [50] IETF. A Survey of Worldwide Censorship [43] StopBadware. Request a review. url: https: Techniques. url: https://web.archive.org/ //web.archive.org/web/20200221161317/ web/20200214050651/https://tools.ietf. https : / / www . stopbadware . org / request - org/html/draft-irtf-pearg-censorship- review. Accessed: 20 February 2020. 00. August 2019. Accessed: 14 February 2020. [44] Spamhaus. About Spamhaus. url: https : / / [51] Chen Ding et al. “Centralized content-based web . archive . org / web / 20200221165906 / Web filtering and blocking: how far can https://www.spamhaus.org/organization/. it go?” In: IEEE SMC’99 Conference Pro- Accessed 21 February 2020. ceedings. 1999 IEEE International Confer- [45] Google Safe Browsing. Safe Browsing: mal- ence on Systems, Man, and Cybernetics (Cat. ware and phishing. url: https : / / web . No.99CH37028). Vol. 2. Oct. 1999, 115–119 archive.org/web/20200219043236/https: vol.2. doi: 10.1109/ICSMC.1999.825218. //transparencyreport.google.com/safe- [52] Ruixi Yuan et al. “An SVM-based machine browsing / overview. Accessed: 18 February learning method for accurate internet traffic 2020. classification”. In: Information Systems Fron- tiers 12.2 (2010), pp. 149–156. doi: 10.1007/

24 s10796-008-9131-2. url: https://doi.org/ 10.1007/s10796-008-9131-2. [53] Maria Konte, Roberto Perdisci, and Nick Feam- ster. “ASwatch: An AS Reputation System to Expose Bulletproof Hosting ASes”. In: Proceed- ings of the 2015 ACM Conference on Special Interest Group on Data Communication, SIG- COMM 2015, London, United Kingdom, Au- gust 17-21, 2015. Ed. by Steve Uhlig et al. ACM, 2015, pp. 625–638. doi: 10 . 1145 / 2785956.2787494. url: https://doi.org/ 10.1145/2785956.2787494.