Masaryk University Faculty of Informatics Applied Kleptography in TLS Master’s Thesis Bc. Adam Janovský Brno, Spring 2018 Declaration I declare that this paper is my original authorial work, which I have worked out on my own. All sources, references, and literature used or excerpted during elaboration of this work are properly cited and listed in complete reference to the due source. Bc. Adam Janovský Advisor: Mgr. et Mgr. Jan Krhovják, Ph.D. i Abstract The goal of this thesis is to examine the possibilities of kleptography in TLS protocol. Kleptography is a study of stealing information securely and subliminally from black-box cryptographic devices. Information is exfiltrated from the system via backdoored algorithm inside asymmetri- caly encrypted subliminal channel. The thesis first introduces the field of kleptography and illustrates the problems on RSA and ELGamal cryptosystems, and on Diffie-Hellman protocol for key exchange. Next, the thesis revisits the TLS protocol and reviews existing proposals of kleptographic backdoors in TLS protocol. Further, several improvements in construction of such backdoor are proposed. The thesis argues that the novel proposal is a kleptographic backdoor relying on computational security. In addition, the proposal is implemented into the OpenSSL library as a proof-of-concept. Finally, the thesis studies possible ways of detection of the backdoor in the TLS protocol. ii Keywords Backdoor, cryptovirology, information security, kleptography, TLS iii Contents 1 Introduction 1 2 Preliminaries 3 2.1 Black-box cryptography ..................3 2.2 Covert and subliminal channels ..............5 3 Kleptography Overview 9 3.1 History of kleptography .................. 10 3.2 SETUP concept ....................... 11 3.3 SETUP examples ...................... 13 3.3.1 SETUP in RSA . 13 3.3.2 SETUP in ElGamal . 16 3.3.3 SETUP in Diffie-Hellman protocol . 17 3.4 Kleptography in the wild .................. 18 3.5 Defences ........................... 19 4 Kleptography in TLS 23 4.1 Transport Layer Security .................. 23 4.1.1 TLS Handshake . 23 4.1.2 Master secret derivation . 25 4.2 Proposed asymmetric backdoors for TLS ......... 27 4.2.1 SETUP by Gołębiewski et al. 28 4.2.2 SETUP by Yung and Young . 30 4.2.3 Comparison of proposed solutions . 34 5 Proposal of SETUP for TLS 35 5.1 SETUP design ........................ 35 5.1.1 Desired improvements . 35 5.1.2 Backdoor description . 37 5.1.3 Properties of SETUP proposal . 38 5.1.4 Additional key exchange methods . 40 5.2 Implementation ....................... 42 6 Heuristic Backdoor Detection 43 6.1 Tests of randomness .................... 43 6.2 Timing experiments ..................... 44 v 6.2.1 Experiment setup . 45 6.2.2 Average execution times . 46 7 Conclusions 49 Appendices 55 A Statistical Tests of Randomness Results 57 B Distribution of Execution Times 65 C Data Attachment 67 vi 1 Introduction Security of communication over a computer network is being ensured by various cryptographic protocols and primitives. Together with advances in information hiding, new threats emerge, as careful design and imple- mentation of cryptographic protocols is a complex task. Tamper-proof devices were proposed as a remedy for many security-related problems. Their advantage is undeniable since they are protected from physical attacks and to change the executed code is very difficult. However, they inherently introduce a trust into the manufacturer. It was shown that such devices are theoretically vulnerable to the presence of so-called subliminal channels. Such channels can be used to exfiltrate private information from the underlying system covertly, inside cryptographic primitives. Consequently, viruses can utilise subliminal channels to break the security of black-box devices. Since the topic of information security is ubiquitous in the era of computer networks, it is crucial to explore the potential of such cryptographic viruses in order to propose systematic defence. This thesis studies kleptography – the art of stealing information se- curely and subliminally. The field of kleptography was discovered in the 1990s by Yung and Young [1]. Kleptographic backdoors for many protocols and primitives were proposed ever since. As examples, the thesis mentions subverted RSA and ElGamal encryption schemes, and the Diffie-Hellman (DH) protocol for shared key exchange. Also, two concepts of kleptographic backdoors for Transport Layer Secure (TLS) protocol were proposed [2, 3]. However, the proposed concepts for TLS are rather theoretical, and it remained unknown whether such backdoor can pose a practical threat. The thesis aims to answer this question. In particular, this thesis presents and implements a kleptographic backdoor for TLS protocol. In Chapter 2, the black-box environment is outlined and subliminal channels are introduced. Chapter 3 presents kleptography thoroughly. First, a history of kleptography is covered. Then, the thesis presents an asymmetric backdoor as a supreme tool of kleptography. Further, examples of kleptographic backdoors for RSA, ElGamal and Diffie- 1 1. Introduction Hellman are presented. Lastly, the chapter comments on real-world exploitations of kleptography and introduces possible defences. Chapter 4 revisits the TLS protocol and critically reviews two proposals of an asymmetric backdoor for the TLS protocol. Chapter 5 builds on the proposed backdoors for TLS, improves them, and describes the process of implementing the backdoor as a proof-of-concept into the OpenSSL library. Finally, Chapter 6 discusses potential detection mechanisms of the implemented backdoor. 2 2 Preliminaries 2.1 Black-box cryptography This chapter aims to introduce the context of black-box cryptography and to outline information channels that can be exploited for malicious purposes. Only a minimum amount of theory on cryptography is given throughout the thesis. The reader is therefore assumed to have knowledge of cryptography and related mathematical disciplines, e.g., number- theory or algebra. To address the aspects of malware in black-box cryptographic devices, the thesis first revisits the definition of a black- box cryptosystem from [4]. Definition. A black-box cryptosystem is an efficient probabilistic algo- rithm that has readable and writeable non-volatile memory. In other words, it has access to a fair coin and can store variables across multiple invocations. Furthermore, the algorithm and memory are not externally accessible. Only the input and output of the cryptosystem is accessible. The definition puts strong assumption on black-box cryptographic de- vices, namely that the internals of those cannot be scrutinized. Such assumption does not hold for most devices in real settings. Even tamper- resistant devices cannot be considered perfectly secure against internals analysis. The thesis adopts rather subtle definition of a black-box cryp- tographic device. We say that a cryptographic device is considered a black-box device if its internals cannot be easily scrutinized. Often, such device is special purpose hardware – a smart card or SSL accelerator for instance. Yet, there exists proprietary software that has not been fully reverse engineered for several years despite widespread use. An example of such software can be Skype. Why is it that some software gets reverse engineered quickly after the release and some not? The difficulty of reverse engineering is regularly increased by obfuscation of binaries. This practice is known as ‘security by obscurity’ and aims at slowing down potential reverse engineering. Consider that there are also legal aspects of reverse engineering. Accordingly, when examining the 3 2. Preliminaries impact of malware in black-box devices, one should not be limited to tamper-proof hardware. Due to the reasons above, the thesis considers proprietary software solutions as black-box devices as well. One could also say that every cryptographic device is a black-box until it is properly scrutinized. When the device gets scrutinized depends on whether the binaries of the device can be easily accessed and to what extent they are obfuscated. Nevertheless, stealthy malware can go undetected for several years in proprietary software. Notice that to detect malware in a certain type of devices, it is not sufficient to inspect a single piece of device. The malware may not be present in all devices of the same type. Black-box cryptography certainly has some advantages; for instance, it is difficult to change the executed code and to attack from outside in black-box settings. Yet, black-box devices necessarily introduce trust into the manufacturer. Indeed, it does not suffice to verify the code authenticity by computing the checksum, but it is necessary to examine the code on assembly level. Observe that there is a conflict between securing the device and verifying its functionality. It is simple to confirm that the device fulfils the specifications. But as the saying goes — the secure program does what it is supposed to do, and nothing else. As stated in the definition of a black-box device, it is difficult or even impossible to prove that such device performs no additional computations. Cryptographic devices that utilise public-key cryptography often commu- nicate public cryptographic keys, random nonces, challenges etc. Those can serve as a malicious information channel. Such channel can be exploited by a malware to exfiltrate information from a device covertly. To remain undetected, the malware must utilise already existing infor- mation channels to leak the data. The thesis