Security Analysis of the Signal Protocol Student: Bc
Total Page:16
File Type:pdf, Size:1020Kb
ASSIGNMENT OF MASTER’S THESIS Title: Security Analysis of the Signal Protocol Student: Bc. Jan Rubín Supervisor: Ing. Josef Kokeš Study Programme: Informatics Study Branch: Computer Security Department: Department of Computer Systems Validity: Until the end of summer semester 2018/19 Instructions 1) Research the current instant messaging protocols, describe their properties, with a particular focus on security. 2) Describe the Signal protocol in detail, its usage, structure, and functionality. 3) Select parts of the protocol with a potential for security vulnerabilities. 4) Analyze these parts, particularly the adherence of their code to their documentation. 5) Discuss your findings. Formulate recommendations for the users. References Will be provided by the supervisor. prof. Ing. Róbert Lórencz, CSc. doc. RNDr. Ing. Marcel Jiřina, Ph.D. Head of Department Dean Prague January 27, 2018 Czech Technical University in Prague Faculty of Information Technology Department of Computer Systems Master’s thesis Security Analysis of the Signal Protocol Bc. Jan Rub´ın Supervisor: Ing. Josef Kokeˇs 1st May 2018 Acknowledgements First and foremost, I would like to express my sincere gratitude to my thesis supervisor, Ing. Josef Kokeˇs,for his guidance, engagement, extensive know- ledge, and willingness to meet at our countless consultations. I would also like to thank my brother, Tom´aˇsRub´ın,for proofreading my thesis. I cannot express enough gratitude towards my parents, Lenka and Jaroslav Rub´ınovi, who supported me both morally and financially through my whole studies. Last but not least, this thesis would not be possible without Anna who re- lentlessly supported me when I needed it most. Declaration I hereby declare that the presented thesis is my own work and that I have cited all sources of information in accordance with the Guideline for adhering to ethical principles when elaborating an academic final thesis. I acknowledge that my thesis is subject to the rights and obligations stip- ulated by the Act No. 121/2000 Coll., the Copyright Act, as amended. In accordance with Article 46(6) of the Act, I hereby grant a nonexclusive author- ization (license) to utilize this thesis, including any and all computer programs incorporated therein or attached thereto and all corresponding documentation (hereinafter collectively referred to as the “Work”), to any and all persons that wish to utilize the Work. Such persons are entitled to use the Work for non- profit purposes only, in any way that does not detract from its value. This authorization is not limited in terms of time, location and quantity. In Prague on 1st May 2018 . Czech Technical University in Prague Faculty of Information Technology c 2018 Jan Rub´ın.All rights reserved. This thesis is school work as defined by Copyright Act of the Czech Republic. It has been submitted at Czech Technical University in Prague, Faculty of Information Technology. The thesis is protected by the Copyright Act and its usage without author’s permission is prohibited (with exceptions defined by the Copyright Act). Citation of this thesis Rub´ın,Jan. Security Analysis of the Signal Protocol. Master’s thesis. Czech Technical University in Prague, Faculty of Information Technology, 2018. Abstrakt Tato diplomov´apr´acese zab´yv´astudiem protokolu Signal. Zamˇeˇrujese pˇre- devˇs´ımna pouˇzitoukryptografii, funkcionalitu a strukturu protokolu. Pr´ace d´aleobsahuje anal´yzuzdrojov´ych k´od˚uofici´aln´ıimplementace a porovn´av´a stav protokolu s jeho dokumentac´ı. Pr´acetak´ediskutuje potenci´aln´ıbezpeˇc- nostn´ıslabiny protokolu a formuluje jejich zm´ırnˇen´ıˇciodstranˇen´ı. Kl´ıˇcov´a slova protokol Signal, bezpeˇcnostn´ı anal´yza,instant messaging, bezpeˇcn´akomunikace, Double Ratchet, forward secrecy, v´ymˇenakl´ıˇc˚u Abstract This thesis provides a security analysis of the Signal Protocol. The protocol’s cryptography, functionality, and structure are discussed. The source codes of the official implementation are analyzed and the protocol’s state is compared with the documentation. Finally, the protocol’s potential security vulnerabil- ities are examined and their mitigation or removal is formulated. Keywords Signal protocol, security analysis, instant messaging, secure com- munication, Double Ratchet, forward secrecy, key exchange vii Contents Introduction 1 1 Security Status of IM Protocols 3 1.1 IRC . 3 1.2 OTR . 5 1.3 Jabber . 6 1.4 MTProto . 7 1.5 Signal Protocol . 9 2 Protocol Analysis 13 2.1 Protocol Overview . 13 2.2 Mathematical Notations . 14 2.3 XEdDSA and VXEdDSA Signature Schemes . 15 2.4 X3DH Key Agreement Protocol . 22 2.5 Double Ratchet . 28 3 Implementation Analysis 41 3.1 Code . 41 3.2 Keys . 42 3.3 Initialization . 44 3.4 Ratcheting . 45 3.5 Encryption and Decryption of Messages . 50 3.6 Header Encryption . 53 3.7 Cleanup . 53 4 Evaluation of Results 55 4.1 Security Considerations . 56 Conclusion 59 ix Bibliography 61 A Acronyms 67 B Contents of the enclosed CD 69 x List of Figures 2.1 The shared secret establishment using the Curve25519 . 24 2.2 The KDF chain containing two ratchet steps . 29 2.3 The symmetric-key ratchet . 30 2.4 The Diffie-Hellman ratchet . 31 2.5 The Diffie-Hellman ratchet with renamed labels . 32 2.6 A single Diffie-Hellman ratchet step . 32 2.7 Alice’s Double Ratchet initialization . 33 2.8 Alice sends the Double Ratchet message . 34 2.9 Alice receives the Double Ratchet message . 35 2.10 Alice communicates with Bob using the old DH ratchet public key 35 2.11 Alice communicates with Bob using the new DH ratchet public key 36 2.12 Alice’s Double Ratchet initialization with the header keys . 38 2.13 Alice sends the message with the encrypted header . 39 2.14 Alice performs the DH ratchet step and derives new header keys . 40 2.15 Alice performs the symmetric-ratchet with header keys . 40 3.1 The difference between the chain keys and the message keys . 49 xi List of Tables 2.1 Elliptic curve parameters notation . 15 2.2 Curve25519 parameters . 19 2.3 Curve448 parameters . 19 2.4 X3DH keys . 24 3.1 Initialization parameters of Alice and Bob . 44 xiii Introduction Instant messaging (IM) is a very popular form of online communication. It allows people from all over the world to connect with each other in real- time, drawing inspiration from the written word. It is an alternative to the well-known email communication which provides a more formal method of correspondence. The first IM applications started to emerge in the mid-1960s and were originally limited to the local area networks. When the Internet began to grow, IM applications expanded to the global scope and IM communication evolved significantly. However, with the spread of this trend, people started to realize that not every communication platform is secure. This privacy awareness star- ted to grow since many cases of mass surveillance and communication inter- ceptions from government organizations were published. Edward Snowden, a former NSA employee, exposed one of the biggest mass surveillance cases which caused an increase in privacy awareness among internet users. In this thesis, we provide a brief overview of the IM protocols which are frequently used or which were very popular during their era. We focus on both unsecured protocols and very robust security solutions, providing an insight into the commonly used cryptographic principles, such as the forward secrecy, the asymmetric cryptography, the Diffie-Hellman key exchange or both the client-server and the end-to-end encryption schemes. In the second chapter, we analyze the supposedly most secure IM protocol on the planet – the Signal Protocol – in detail. We provide an in-depth description of the protocol, the theoretical understanding of its functionality along with the mathematical background. We explain how these prelimi- naries improve the security of the cryptographic operations and we describe the ideas behind the protocol which can be actually used separately in other implementations as well. In the third chapter, we analyze the official implementation of the Signal Protocol libraries which are used in the Signal for Android application. The 1 Introduction analysis is based on the theoretical knowledge of the protocol, inspecting the particular implementation and its specifications. We compare the Signal Pro- tocol implementation to the official documentation and we discuss any present discrepancies and undocumented specifications. Finally, we formulate security considerations regarding the protocol’s de- sign and we describe how to mitigate the vulnerabilities or how to remove them completely. 2 Chapter 1 Security Status of IM Protocols During the growth of the Internet, many developers worked hard on new in- stant messaging (IM) protocols which would connect people over the network. Currently, countless IM protocols are used all over the world which can transmit data from one side to another over the network – either safely using the cryptography methods, or by unsecured communication mostly in a plain text form. To provide a general perspective, we decided to present a few diametrically different protocols from less secure ones (e.g. the IRC protocol in its original design) to the very robust security solutions (e.g. the Signal Protocol). Fur- thermore, we decided to select some frequently used protocols for the security inspection. The OTR was one of the first protocols which supported end-to-end en- cryption and it also popularized the concept of the forward secrecy, as opposed to the Pretty Good Privacy protocol (PGP). Its design allowed it to be used as a plugin to other IM clients as well. Jabber was very popular during the spread of the IM communication. It provided a free and open alternative to the proprietary IM services of the day, such as ICQ or AOL Instant Messenger [1]. On the other hand, the MTProto is a very robust protocol which is cur- rently used in the Telegram instant messenger.