Not-Quite-So-Broken TLS: Lessons in Re-Engineering a Security Protocol Specification and Implementation David Kaloper-Meršinjak, Hannes Mehnert, Anil Madhavapeddy, and Peter Sewell, University of Cambridge https://www.usenix.org/conference/usenixsecurity15/technical-sessions/presentation/kaloper-mersinjak This paper is included in the Proceedings of the 24th USENIX Security Symposium August 12–14, 2015 • Washington, D.C. ISBN 978-1-939133-11-3 Open access to the Proceedings of the 24th USENIX Security Symposium is sponsored by USENIX Not-quite-so-broken TLS: lessons in re-engineering a security protocol specification and implementation David Kaloper-Mersinjakˇ †, Hannes Mehnert†, Anil Madhavapeddy and Peter Sewell University of Cambridge Computer Laboratory [email protected] † These authors contributed equally to this work Abstract sensitive services, they are not providing the security we need. Transport Layer Security (TLS) is the most widely Transport Layer Security (TLS) implementations have a deployed security protocol on the Internet, used for au- history of security flaws. The immediate causes of these thentication and confidentiality, but a long history of ex- are often programming errors, e.g. in memory manage- ploits shows that its implementations have failed to guar- ment, but the root causes are more fundamental: the chal- antee either property. Analysis of these exploits typically lenges of interpreting the ambiguous prose specification, focusses on their immediate causes, e.g. errors in mem- the complexities inherent in large APIs and code bases, ory management or control flow, but we believe their root inherently unsafe programming choices, and the impos- causes are more fundamental: sibility of directly testing conformance between imple- mentations and the specification. Error-prone languages: historical choices of pro- We present nqsb-TLS, the result of our re-engineered gramming language and programming style that tend to approach to security protocol specification and imple- lead to such errors rather than protecting against them. mentation that addresses these root causes. The same Lack of separation: the complexities inherent in code serves two roles: it is both a specification of TLS, working with large code bases, exacerbated by lack of executable as a test oracle to check conformance of traces emphasis on clean separation of concerns and modular- from arbitrary implementations, and a usable implemen- ity, and by poor language support for those. tation of TLS; a modular and declarative programming style provides clean separation between its components. Ambiguous and untestable specifications: the chal- Many security flaws are thus excluded by construction. lenges of writing and interpreting the large and ambigu- nqsb-TLS can be used in standalone Unix applica- ous prose specifications, and the impossibility of di- tions, which we demonstrate with a messaging client, rectly testing conformance between implementations and and can also be compiled into Xen unikernels (spe- a prose specification. cialised virtual machine image) with a trusted comput- In this paper we report on an experiment in developing ing base (TCB) that is 4% of a standalone system run- a practical and usable TLS stack, nqsb-TLS, using a new ning a standard Linux/OpenSSL stack, with all network approach designed to address each of these root-cause traffic being handled in a memory-safe language; this problems. This re-engineering, of the development pro- supports applications including HTTPS, IMAP, Git, and cess and of our concrete stack, aims to build in improved Websocket clients and servers. Despite the dual-role de- security from the ground up. sign, the high-level implementation style, and the func- tional programming language we still achieve reasonable We demonstrate the practicality of the result in sev- performance, with the same handshake performance as eral ways: we show on-the-wire interoperability with ex- OpenSSL and 73% – 84% for bulk throughput. isting stacks; we show reasonable performance, in both bulk transfer and handshakes; we use it in a test oracle, validating recorded packet traces which contain TLS ses- 1 Introduction sions between other implementations; and we use it as part of a standalone instant-messaging client. In addition Current mainstream engineering practices for specifying to use in such traditional executables, nqsb-TLS is us- and implementing security protocols are not fit for pur- able in applications compiled into unikernels – type-safe, pose: as one can see from many recent compromises of single-address-space VMs with TCBs that run directly USENIX Association 24th USENIX Security Symposium 223 on a hypervisor [32]. This integration into a uniker- ASN.1 X.509 Trust nel stack lets us demonstrate a wide range of working Anchors systems, including HTTPS, IMAP, Git, and Websocket Serialise TLS clients and servers, while sidestepping a further diffi- Nocrypto culty with radical solutions in this area: the large body Parse CSPRNG of legacy code (in applications, operating systems, and libraries) that existing TLS stacks are intertwined with. Policy We assess the security of nqsb-TLS also in several Config Flow Entropy ways: for each of the root causes above, we discuss why our approach rules out certain classes of associated flaws, Figure 1: nqsb-TLS is broken down into strongly sep- with reference to an analysis of flaws found in previ- arated modules. The main part, in bold boxes, has ous TLS implementations; and we test our authentication pure value-passing interfaces and internals. The PRNG logic with a large corpus of certificate chains generated maintains internal state, while Nocrypto includes C code by using the Frankencert fuzzer [8], which found flaws but has a pure effect-free interface. Arrows indicate in several previous implementations. We have also made depends-on relationships. the system publically available for penetration testing, as a Bitcoin Pinata˜ , an example unikernel using nqsb-TLS. This has a TCB size roughly 4% of that of a similar sys- implementation behaviour conforms to the specification. tem using OpenSSL on Linux. TLS is not unique in this, of course, and many other We describe our overall approach in the remainder of specifications are expressed in the same traditional prose the introduction. We then briefly describe the TLS pro- style, but its disadvantages are especially serious for se- tocol ( 2), analyse flaws previously found in TLS im- § curity protocols. plementations ( 3), and the result of applying our ap- § For nqsb-TLS, we specify TLS as a collection of pure proach, dubbed nqsb-TLS ( 4). We demonstrate the du- § functions over abstract datatypes. By avoiding I/O and ality of nqsb-TLS next by using its specification to vali- shared mutable state, these functions can be considered date recorded sessions ( 5) and executing its implemen- § in isolation and each is deterministic, with errors re- tation to provide concrete services ( 6). We evaluate the § turned as explicit values. The top-level function takes an interoperability, performance, and security ( 7) of nqsb- § abstract protocol state and an incoming message, and cal- TLS, describe related work ( 8), and conclude ( 9). § § culates the next state and any response messages. To do nqsb-TLS is freely available under a BSD license so, it invokes subsidiary functions to parse the message, https://nqsb.io ( ), and the data used in this paper is drive the state machine, perform cryptographic opera- openly accessible [27]. tions, and construct the response. This top-level function can be executed as a trace-checker, on traces both from 1.1 Approach our implementation and from others, such as OpenSSL, to decide whether they are allowed by our specification A precise and testable specification for TLS In prin- or not. In building our specification, to resolve the RFC ciple, a protocol specification should unambiguously de- ambiguities, we read other implementations and tested fine the set of all implementation behaviour that it allows, interoperability with them; we thereby capture the prac- and hence also what it does not allow: it should be pre- tical de facto standard. cise. This should not be confused with the question of whether a specification is loose or tight: a precise specifi- Reuse between specification and implementation cation might well allow a wide range of implementation The same functions form the main part of our imple- behaviour. It is also highly desirable for specifications mentation, coupled with code for I/O and to provide en- to be executable as test oracles: given an implementa- tropy. Note that this is not an “executable specification” tion behaviour (perhaps a trace captured from a particu- in the conventional sense: our specification is necessar- lar execution), the specification should let one compute ily somewhat loose, as the server must have the freedom whether it is in the allowed set or not. to choose a protocol version and cipher suite, and the In practice, the TLS specification is neither, but rather trace checker must admit that, while our implementation a series of RFCs written in prose [13, 14, 15]. An ex- makes particular choices. plicit and precise description of the TLS state machine Each version of the implementation (Unix, unikernel) is lacking, as are some security-critical preconditions of has a top-level Flow module that repeatedly performs I/O its transitions, and there are ambiguities in various semi- and invokes the pure functional core; the trace-checker formal grammars. There is no way such prose documents has a top-level module of the same type that reads in a can be executed as a test oracle to directly test whether trace to be checked offline. 2 224 24th USENIX Security Symposium USENIX Association Separation and modular structure This focus on on the security protocol itself (as we recall in 3, some § pure functional descriptions also enables a decomposi- vulnerabilities have been found there). We are also not tion of the system (both implementation and specifica- attempting to advance the state of the art in side-channel tion) into strongly separated modules, with typed inter- defence, though we do follow current best practice.