Proceedings on Privacy Enhancing Technologies ; 2020 (4):321–335 Benjamin VanderSloot*, Sergey Frolov, Jack Wampler, Sze Chuen Tan, Irv Simpson, Michalis Kallitsis, J. Alex Halderman, Nikita Borisov, and Eric Wustrow Running Refraction Networking for Real time, popular circumvention techniques, such as domain Abstract: Refraction networking is a next-generation fronting and VPNs, are becoming harder to deploy or censorship circumvention approach that locates proxy more frequently blocked [15]. There is an urgent need to functionality in the network itself, at participating field more advanced circumvention technologies in order ISPs or other network operators. Following years of re- to level the playing field. search and development and a brief pilot, we established One proposed new circumvention approach, Refrac- the world’s first production deployment of a Refrac- tion Networking, has the potential to fill this need, and tion Networking system. Our deployment uses a high- has been developed in the form of several proposed pro- performance implementation of the TapDance protocol tocols [2, 7, 9, 13, 16, 17, 20, 27, 28] and other re- and is enabled as a transport in the popular circum- search [3, 10, 14, 19, 25] over the past decade. It works vention app Psiphon. It uses TapDance stations at four by deploying technology at ISPs or other network oper- physical uplink locations of a mid-sized ISP, Merit Net- ators that observes connections in transit and provides work, with an aggregate bandwidth of 140 Gbps. By the censorship circumvention functionality. Though promis- end of 2019, our system was enabled as a transport op- ing in concept, deploying Refraction Networking in the tion in 559,000 installations of Psiphon, and it served real world has faced a number of obstacles, including the upwards of 33,000 unique users per month. This pa- complexity of the technology and the need to attract per reports on our experience building the deployment cooperation from ISPs. Other than a one-month pilot and operating it for the first year. We describe how we that our project conducted in 2017 [8], no Refraction overcame engineering challenges, present detailed per- implementation has ever served real users at ISP scale, formance metrics, and analyze how our system has re- leaving the approach’s practical feasibility unproven. sponded to dynamic censor behavior. Finally, we review In this paper, we describe lessons and results from lessons learned from operating this unique artifact and a real-world deployment of Refraction Networking that discuss prospects for further scaling Refraction Network- we have operated in production for over a year and that ing to meet the needs of censored users. is enabled as a transport in more than 559,000 installa- tions of the popular Psiphon circumvention tool for PC and mobile users. Building on our 2017 pilot, the deploy- ment is based on a high-performance implementation of 1 Introduction the TapDance protocol [8]. It operates from stations in- stalled at Merit Network, a mid-sized ISP, that observe National-governments are deploying increasingly sophis- an average of 70 Gbps of aggregate commodity traffic ticated systems for Internet censorship, which often from four network locations, each of which individually take the form of deep-packet inspection (DPI) middle- processes a peak of 10–40 Gbps. The system has served boxes located at network choke-points [5]. At the same up to 500 Mbps of proxied traffic to circumvention users. Building and running this deployment required solv- ing complex technical, operational, and logistical chal- *Corresponding Author: Benjamin VanderSloot: Uni- lenges and necessitated collaboration among researchers, versity of Michigan, E-mail: [email protected] network engineers, and circumvention tool developers. Sergey Frolov: University of Colorado Boulder, E-mail: This reflects a non-trivial challenge to Refraction Net- [email protected] Jack Wampler: University of Colorado Boulder, E-mail: working systems: Refraction Networking cannot func- [email protected] tion in isolation from the rest of the Internet, and so Sze Chuen Tan: University of Illinois, Urbana-Champaign, its success depends on close interactions between the E-mail: [email protected] Internet operator and Internet freedom communities. Irv Simpson: Psiphon, E-mail: [email protected] In order to serve users well and meet requirements Michalis Kallitsis: Merit Network, E-mail: [email protected] set by our ISP and circumvention tool partners, we J. Alex Halderman: University of Michigan, E-mail: jhal- worked within the following technical constraints: [email protected] Nikita Borisov: University of Illinois, Urbana-Champaign, E-mail: [email protected] Eric Wustrow: University of Colorado Boulder, E-mail: [email protected] Running Refraction Networking for Real 322 – Each TapDance station could use only 1U of physi- 2 Background cal rack space at one of the ISP’s uplink locations. – All stations at the ISP would need to coordinate to Refraction Networking (previously known as “decoy function as a single distributed system. routing”) is an anticensorship strategy that places cir- – The deployment had to operate continuously, de- cumvention technology at Internet service providers spite occasional downtime of individual stations. (ISPs) and other network operators, rather than at – We had to strictly avoid interfering with the ISP’s network endpoints. Clients access the service by mak- network operations or its customers systems. ing innocuous-looking encrypted connections to existing, – The deployment had to achieve acceptable network uncensored websites (“decoys”) that are selected so that performance to users in censored environments. the connection travels through a participating network. The client covertly requests proxy service by including In this paper we describe our experience meeting these a steganographic tag in the connection envelope that requirements and the implications this has for further is constructed so that it can only be detected using a deployment of Refraction Networking. private key. At certain points within the ISP’s network, In addition, we analyze data from four months of devices (“stations”) inspect passing traffic to identify operations to evaluate the system’s performance. This tagged connections, use data in the tag to decrypt the four-month period reflected typical behavior for our ISP request, proxy it to the desired destination, and return partners and concluded with a significant censorship the response as if it came from the decoy site. To the event that applied stress to infrastructure of our cir- censor, this connection looks like a normal connection to cumvention tool deployment partner. It shows that our an unblocked decoy site. If sufficiently many ISPs par- deployment’s load is affected by censorship practices, ticipate, censors will have a difficult time blocking all and that it was able to handle the spike in utilization available decoy sites without also blocking a prohibitive effectively. During the censorship event, we provided In- volume of legitimate traffic [24]. ternet access to more users than at any previous time, Refraction Networking was first proposed in 2011. and the system handled this load without measurable Three independent works that year—Telex [28], Curve- degradation in quality of service or generating excessive ball [16] and Cirripede [13]—all proposed the idea of load on decoy websites, as reflected by the opt-out rate. placing proxy “stations” at ISPs, with various propos- Our final contribution is a discussion of lessons we als for how clients would signal the ISP station. For learned from building and operating the deployment instance, Curveball used a pre-shared secret between that can inform future work on Refraction Networking the client and station, while Telex and Cirripede used and other circumvention technologies. We identify two public-key steganography to embed tags in either TLS particular areas—decoy site discovery and reducing sta- client-hello messages or TCP initial sequence numbers. tion complexity—where further research and develop- Without the correct private key, these tags are crypto- ment work would be greatly beneficial. graphically indistinguishable from the random protocol We conclude that Refraction Networking can be de- values they replace, so censors cannot detect them. ployed continuously to end-users with sufficient network However, all of these first-generation schemes re- operator buy-in. Although attracting ISP partnership quired inline blocking at the ISP; that is, the station remains the largest hurdle to the technology’s practical needed to be able to stop packets in individual tag- scalability, even with relatively limited scale, Refraction carrying TCP connections from reaching their destina- can meet a critical real-world need as a fall-back trans- tion. This lets the station pretend to be the decoy server port that can provide service when other, lighter-weight without the real decoy responding to the client’s pack- transports are disrupted by censors. ets. While this makes for a conceptually simpler design, The remainder of this paper is structured as follows. inline blocking is expensive to do in production ISP In Section 2, we discuss existing techniques for and use networks, where traffic can exceed 100s of Gbps. Inline of Refraction Networking. We then describe our deploy- blocking devices also carry a higher risk of failing closed, ment’s architecture in Section 3. In Section 4, we quan- which would disrupt other network traffic, making ISPs tify the performance of our deployment using data from leery of deploying the Telex-era protocols. the first four months of 2019. We end the discussion with To address this concern, researchers developed Tap- a comparison to existing decoy routing schemes. In Sec- Dance [27], which only requires a passive tap at the ISP tion 5, we draw lessons from our deployment experience. station, obviating the need for inline blocking. Instead, Finally, we conclude in Section 6. Running Refraction Networking for Real 323 TapDance clients mute the decoy server by sending an 3.1 Station Placement incomplete HTTP request inside the encrypted TLS con- nection.