1 SoK: Towards Grounding Censorship Circumvention in Empiricism Michael Carl Tschantz⇤, Sadia Afroz⇤, Anonymous‡, and Vern Paxson⇤† ⇤International Computer Science Institute †University of California, Berkeley Abstract—E↵ective evaluations of approaches to circumventing approaches meeting their own evaluation criteria can succumb government Internet censorship require incorporating perspec- to vulnerabilities not considered by their evaluation (e.g., [1]). tives of how censors operate in practice. We undertake an extensive examination of real censors by surveying prior mea- Approach. To address this disconnect between evaluation and surement studies and analyzing field reports and bug tickets from the actual operating conditions of censorship circumvention practitioners. We assess both deployed circumvention approaches approaches, in this work we seek to ground the evaluation and research proposals to consider the criteria employed in their of circumvention approaches in empirical observations of real evaluations and compare these to the observed behaviors of real censors, identifying areas where evaluations could more faithfully censors. To do so, we systematically compare the behaviors of and e↵ectively incorporate the practices of modern censors. These real censors to the evaluation criteria used by circumvention- observations lead to an agenda realigning research with the approach designers. predominant problems of today. Our work systematizes the evaluation of approaches for censorship circumvention in four ways: I. Introduction 1) We collect data on real-world attacks to show the current Censorship circumvention research seeks to develop ap- state of censorship practice (Sections II and IV), proaches for facilitating access to banned Internet resources, a 2) We survey circumvention approaches and their eval- domain with a fundamentally adversarial nature arising from uations to illuminate the current state of evaluation the ongoing interactions between circumventors and censors. (Sections V and VI, respectively), Both parties find themselves locked in an arms race where each 3) We compare the evaluations designed to assess the ffi side must manage tradeo↵s between efficacy and expenditure. di culty of blocking an approach to the actual actions These tradeo↵s continually evolve in subtle ways as new of real censors (Section VII), technologies change the costs of various approaches. 4) We point to open research problems whose resolution Given this complexity, undertaking sound evaluation of will improve evaluation (Section VIII). potential circumvention approaches proves both crucial and Scope. We focus our discussion on censorship by governments difficult. Sound evaluation is crucial since, due to limited attempting to prevent subjects from accessing particular web resources, the developers of circumvention approaches cannot resources outside the government’s jurisdiction. The censor implement and deploy every prospective approach; they need seeks to detect and disrupt banned traffic by placing monitors criteria for selecting the most promising. It is difficult, on the at the edges of their network—just as customs inspectors other hand, because unidentified weaknesses in an approach intercept and examine physical goods at international borders. o↵er potential openings to censors, but worst-case analyses We also limit the types of circumvention we consider that presume censors will necessarily exploit such vulnerabil- to channel-based approaches that (1) bypass country-level ities ignore the realities of censors who aim to avoid blocking censors that monitor network traffic between two end points, profitable traffic while staying within their budget constraints. (2) communicate with resources outside the censors’ borders, Soundly incorporating these realities into evaluations requires and (3) enable low-latency connections (roughly, fast enough grounding in empirical observations of real censors. for web browsing). Our scope excludes concerns such as While the evaluation sections of research papers provide internal censorship of newspapers or disruption of entirely some insight into the promise of a given circumvention ap- domestic communication. Even so, the remaining space of proach, each paper employs its own evaluation methodology, censor activity and circumvention approaches is large, with 55 typically selected with the capabilities of the approach in mind approaches or evaluations of approaches [2–56] falling within but often not balanced against realistic models of censors. Fur- our scope. Figure 1 shows a generic model of censorship and thermore, such approach-oriented evaluations make it difficult circumvention under the scope we use. to compare across di↵erent approaches, or to determine how well an evaluation predicts real-world performance. Prototyped Overview. We first take a detailed look at the arms race between Tor and China as an illustration of the cat-and-mouse ‡ This coauthor chooses to forgo identification in protest of the IEEE nature of censorship and circumvention (Section II). After Security and Privacy Symposium’s acceptance of support from the US covering related work (Section III), we broaden our view by National Security Agency. While it pains us for his significant contributions to the work to go unrecognized here, we respect the heartfelt principles that examining censorship incidents involving other channel-based led him to his position. circumvention approaches (Section IV). Inside Outside used is too wide for any one list to cover all use cases, or Instructions for a single study to attempt to comprehensively rank them. Circumventor's website Rather, we hope to provide a systematic method of thinking 1 about evaluation that will guide evaluators of circumvention Identifiers 2 Circumvention service approaches in their selection of appropriate criteria on a case- by-case basis. Also, we do not intend for this work to discount the utility Monitor Forwarder Destination of forward-looking studies that anticipate the more advanced 3 censors of the future. Rather, we aim to make the tradeo↵s User clear: some approaches considered by research are likely years ahead of the point where their overhead is justified Fig. 1. An illustration of the type of censorship we study. First, censored by actual censors e↵ectively blocking less advanced methods; users within a censor’s jurisdiction gather information about how to use an meanwhile, censors block deployed approaches using simple approach, which may include a program download. Second, the users gain attacks. We hope this observation inspires the research com- various identifiers, such as IP addresses and passwords. Third, they run their traffic through a client-side program that applies various transformations to munity towards also providing tools for preventing simpler hide the true destination and content. In the case of a banned destination, the attacks, which would yield immediate benefits for many real approach sends the obscured traffic to some allowed destination that acts as users. a forwarder to the real destination. We make details and our database available at http://internet-freedom-science.org/circumvention-survey/ Next, we provide a survey of channel-based censorship circumvention (Section V) and its evaluation (Section VI). II. Illustrating the Problem Space: Unlike other surveys on circumvention [57–59], we do not Tor and the Great Firewall focus on comparing the approaches themselves, but rather on comparing their evaluations. We enumerate the criteria that The problem space we consider has both disparate aspects the developers of each approach used in their evaluations. We and a complicated arms-race-driven evolution. In this section find little commonality in the evaluation methods employed. we frame the space through the lens of the Tor anonymity While some diversity is to be expected given that approaches system and the “Great Firewall” (GFW), the primary national di↵er in goals and intended deployment environments, we censorship apparatus of China. The conflict between these two find no globally organizing principles guiding the selection actors is representative of the larger world of censorship and ↵ ↵ of evaluation criteria. circumvention, and o ers us a way of introducing di erent notions and the associated terminology we will use in our We focus on the criteria related to detecting circumventing discussion, as well as providing some of the grounding in traffic and compare them to the actions of real censors (Sec- real-life censorship that underlies many of our perspectives. tion VII). We observe that system designers tend to emphasize censor capabilities that may become important in the future, Tor provides a convenient focus due to the relatively ex- but not seen in practice today, with little assessment on actual tensive documentation of its censorship and circumvention detection techniques used by current censors. In particular, we counter-responses. We mined blog posts, bug reports, and the Tor Project’s public documentation to identify the censorship identify three disconnects between practice and research: events that underlie our narrative. A progression emerges: Tor, 1) Real censors attack how users discover and set up which was not originally designed for circumvention, is used channels, whereas research often centers on channel to evade the GFW. The censor blocks the Tor website and the usage, servers that make up the anonymity network; Tor responds 2) Real censors prefer cheap passive monitoring or more with mirrors and secret entry servers.