Analysis of DTLS Implementations Using Protocol State Fuzzing Paul Fiterau-Brostean and Bengt Jonsson, Uppsala University; Robert Merget, Ruhr-University Bochum; Joeri de Ruiter, SIDN Labs; Konstantinos Sagonas, Uppsala University; Juraj Somorovsky, Paderborn University https://www.usenix.org/conference/usenixsecurity20/presentation/fiterau-brostean This paper is included in the Proceedings of the 29th USENIX Security Symposium. August 12–14, 2020 978-1-939133-17-5 Open access to the Proceedings of the 29th USENIX Security Symposium is sponsored by USENIX. Analysis of DTLS Implementations Using Protocol State Fuzzing Paul Fiterau-Bro¸stean˘ Bengt Jonsson Robert Merget Joeri de Ruiter Uppsala University Uppsala University Ruhr University Bochum SIDN Labs Konstantinos Sagonas Juraj Somorovsky Uppsala University Paderborn University Abstract reach 11.6 billion by 2021 [26]. This will constitute half of all devices connected to the Internet, with the percentage set to Recent years have witnessed an increasing number of proto- grow in subsequent years. Such trends also increase the need cols relying on UDP. Compared to TCP, UDP offers perfor- to ensure that software designed for these devices is properly mance advantages such as simplicity and lower latency. This scrutinized, particularly with regards to its security. has motivated its adoption in Voice over IP, tunneling techno- DTLS is also used as one of the two security protocols in logies, IoT, and novel Web protocols. To protect sensitive data WebRTC, a framework enabling real-time communication. exchange in these scenarios, the DTLS protocol has been de- WebRTC can be used, for example, to implement video con- veloped as a cryptographic variation of TLS. DTLS’s main ferencing in browsers without the need for a plugin. It is challenge is to support the stateless and unreliable transport of supported by all major browsers, including Mozilla Firefox, UDP. This has forced protocol designers to make choices that Google Chrome, Microsoft Edge, and Apple’s Safari. affect the complexity of DTLS, and to incorporate features that need not be addressed in the numerous TLS analyses. Whereas significant effort has been invested into ensuring We present the first comprehensive analysis of DTLS im- security of TLS implementations, those based on DTLS have plementations using protocol state fuzzing. To that end, we ex- so far received considerably less scrutiny. Our work fills this tend TLS-Attacker, an open source framework for analyzing gap by providing an extensible platform for testing and analy- TLS implementations, with support for DTLS tailored to the zing systems based on DTLS. We describe this framework, stateless and unreliable nature of the underlying UDP layer. and use it to analyze a number of existing DTLS implementa- We build a framework for applying protocol state fuzzing on tions, including the most commonly used ones. Our specific DTLS servers, and use it to learn state machine models for focus is on finding logical flaws, which can be exposed by thirteen DTLS implementations. Analysis of the learned state non-standard or unexpected sequences of messages, using a protocol state fuzzing state fuz- models reveals four serious security vulnerabilities, including technique known as (or simply zing a full client authentication bypass in the latest JSSE version, ). as well as several functional bugs and non-conformance is- As in TLS, each DTLS client and server effectively imple- sues. It also uncovers considerable differences between the ments a state machine which keeps track of how far protocol models, confirming the complexity of DTLS state machines. operation has progressed: which types of messages have been exchanged, whether the cryptographic materials have been agreed upon and/or computed, etc. Each DTLS implementa- 1 Introduction tion must correctly manage such a state machine for a number of configurations and key exchange mechanisms. Correspon- UDP is widely used as an unreliable transfer protocol for ding implementation flaws, so-called state machine bugs, may Voice over IP, tunneling technologies, and new Web protocols, be exploitable, e.g., to bypass authentication steps or establish and is one of the commonly used protocols in the Internet insecure connections [5]. To find such flaws, state fuzzing of Things (IoT). As UDP does not offer any security by it- has proven particularly effective not only for TLS [13], but self, Datagram Transport Layer Security (DTLS) [29,36] was also for SSH [19], TCP [18], MQTT [40], OpenVPN [12], introduced. DTLS is a variation on TLS, a widely used secu- QUIC [33], and the 802.11 4-Way Handshake [28], leading rity protocol responsible for securing communication over a to the discovery of several security vulnerabilities and non- reliable data transfer protocol. conformance issues in their implementations. DTLS is one of the primary protocols for securing IoT State fuzzing automatically infers state machine descripti- applications [38]. The number of IoT devices is projected to ons of protocol implementations using model learning [32,41]. USENIX Association 29th USENIX Security Symposium 2523 This is an automated black-box technique which sends se- separately, and when exploring settings involving client certi- lected sequences of messages to the implementation, observes ficate authentication. Such complexity in the models creates the corresponding outputs, and produces a Mealy machine problems both for the model learning algorithm and for the that abstractly describes how the implementation responds to interpretation of resulting models. We ameliorate and avoid message flows. The Mealy machine can then be analyzed to some of the complexity in two ways: 1) Our test harness does spot flaws in the implementation’s control logic or check com- not employ reordering and fragmentation, and hence this is pliance with its specification. State fuzzing works without any not part of our learned models. 2) We adapt the MAPPER so a priori knowledge of the protocol state machine, but relies as to enable handshakes to “restart”, which has the additional on a manually constructed protocol-specific test harness, a.k.a. side-effect of decreasing the size of the learned models, since a MAPPER, which translates symbols in the Mealy machine successful restarts typically show up as back-transitions to to protocol packets exchanged with the implementation. regular handshake states. Obtaining models for a wide range of implementations Challenges resulting from the DTLS design. DTLS is and configurations. We have applied our platform to thirteen more complex than other security protocols that have so far implementations of ten distinct vendors (Section6). Besides been subject to state fuzzing. Most of these [12, 18, 19] run covering a wide spectrum of DTLS implementations, ranging over TCP, relying on its support for reliable connections. In from mature, general-purpose libraries to implementations contrast, DTLS runs over UDP, which is connectionless. This designed for IoT or WebRTC, we mention that some of them implies that DTLS has to implement its own retransmission are DTLS libraries without a TLS component, on which state mechanism and provide support for message loss, reordering, fuzzing has never been applied before. and fragmentation. Moreover, an ongoing DTLS interaction For each implementation we examine many, often all, com- cannot be terminated by simply closing the connection, as is binations of supported key exchange and client certificate the case with TLS. As a result, most DTLS implementations authentication configurations. This ensures that state fuzzing allow interaction to continue even after reception of unex- does not miss bugs that are only present in certain configurati- pected messages —after all, these messages might have just ons. In fact, this proved important: several of the Java Secure arrived out of order— and may subsequently allow a hands- Socket Extension (JSSE) bugs reported in Section 7.4 could hake to “restart in the middle” and finish successfully. Finally, only have been discovered with a configuration requiring compared to TLS, DTLS includes an additional message ex- client certificate authentication. change used to prevent Denial-of-Service attacks. All this From models to bugs. Once models are obtained we proceed added complexity makes protocol state fuzzing more difficult to analyze them, looking for unexpected or superfluous states to apply for DTLS than for TLS. and transitions. Some of the main findings of our analysis are: Supporting mapper construction. DTLS’ support for (i) A complete client authentication bypass in JSSE, which is message loss, reordering, and fragmentation requires additio- the default TLS/DTLS library of the Java Standard Edition nal packet parameters compared to TLS, such as message se- Platform. The bug allows attackers to authenticate themselves quence numbers. DTLS parameters have to be correctly mana- to a JSSE server by sending special out-of-order DTLS mes- ged by the MAPPER. This requires special care when deviating sages without ever proving to the server that they know the from an expected handshake sequence (a.k.a. a happy flow), private key for the certificate they transmit. The bug is especi- since each particular parameter management strategy may ally devastating, since it also affects JSSE’s TLS library. This allow or prohibit a “restarting” handshake to be eventually greatly increases its impact, as JSSE’s TLS library is often completed. In order to facilitate MAPPER construction and used to