Cryptographic Security of SSH Encryption Schemes Torben Brandt Hansen Thesis submitted to the University of London for the degree of Doctor of Philosophy Department of Information Security Royal Holloway, University of London 2020 Declaration These doctoral studies were conducted under the supervision of Prof. Kenneth G. Paterson. The work presented in this thesis is the result of original research carried out by myself, in collaboration with others, whilst enrolled in the Department of Information Security as a candidate for the degree of Doctor of Philosophy. This work has not been submitted for any other degree or award in any other university or educational establishment. Torben Brandt Hansen November, 2020 2 Acknowledgements A grateful thanks to Kenny Paterson for his great supervision and supporting my non- academic endeavours during my Ph.D. Even more for his mentorship and for pushing me to develop and improve my academic skills; it is impossible to leave Kenny's office without learning something new and exciting. And, not least, thank you for your dedicated attempts to cryptographically hides my silly grammar mistakes. Also a special thanks to my advisor Martin Albrecht for always providing support and great collaborations. Thank you to Royal Holloway and EPSRC for providing the funding, allowing me to pursue a Ph.D. and for given me many opportunities to explore the real world. Most importantly, I am in debt to Aeriel for supporting me through my Ph.D. and encouraging me to pursue my dreams. Mahal Kita 3 Abstract SSH is a Swiss Army Knife protocol for creating secure communication links between machines and an indispensable tool for IT professionals. However, its underlying symmetric encryption scheme constructions have not seen the same rigorous analysis as constructions in other popular secure communication protocols. This thesis aims to bridge this gap, providing SSH deployment statistics, new attacks against a number of SSH encryption schemes, a thorough security analysis of several SSH encryption schemes, and development of new SSH encryption schemes that provide better security properties than existing schemes. Firstly, we report on several scans performed targeting publicly accessible SSH servers on the Internet. From these scans, we compile longitudinal SSH statistics evaluating the evolution of preferred SSH encryption scheme, SSH software and SSH version trends. Secondly, we describe several new attacks on SSH encryption schemes in OpenSSH that utilise the CBC encryption mode of operation. These attacks are a result of both inherent weaknesses in CBC-mode and bugs in the OpenSSH implementation. Thirdly, we use the ciphertext fragmentation framework to analyse the concrete cryptographic security of a number of SSH encryption schemes as implemented in OpenSSH. Finally, we develop a practical version of the symmetric encryption scheme InterMAC, implement it and evaluate its security and performance. The implementation is then used to construct InterMAC-based SSH encryption schemes. We evaluate the performance of these new schemes against existing SSH encryption schemes in OpenSSH. 4 Contents 1 Introduction 13 1.1 Motivation . 13 1.2 Vulnerability Disclosure . 16 1.3 Organisation of Thesis . 16 1.4 Associated Publications . 17 2 Background 19 2.1 Notation . 19 2.2 Bits vs Bytes . 21 2.3 Modern Cryptography . 22 2.3.1 Concrete Security . 22 2.3.2 Proofs and Security Notions . 23 2.3.3 Pseudorandom Functions . 24 2.3.4 Probabilistic Symmetric Encryption . 25 2.3.5 IV-based and Nonce-based Symmetric Encryption . 33 2.3.6 Block Ciphers . 37 2.3.7 Message Authentication . 39 2.4 SSH . 43 2.4.1 SSH Implementations . 44 2.4.2 SSH Binary Packet Protool . 45 3 Ciphertext Fragmentation 51 3.1 On the Choice of Security Model . 51 3.2 Syntax . 53 3.3 Correctness . 55 3.3.1 Alternative Correctness Definitions . 57 3.4 Confidentiality . 57 3.5 Authenticity . 59 3.6 Boundary Hiding . 60 3.7 Denial-of-Service . 65 3.8 A Deficient Definition of Confidentiality . 66 3.9 A Critique of the Active Boundary Hiding Security Property . 68 3.10 Related Security Models . 69 3.10.1 Stateful Symmetric Encryption . 69 3.10.2 Formal Security Treatment of SSH Encryption Schemes in CTR-mode . 70 3.10.3 On-line Symmetric Encryption . 70 3.10.4 Stream-based Channel . 72 5 CONTENTS 4 The SSH Ecosystem on the Internet 74 4.1 Ethical Considerations . 74 4.2 Data Set . 74 4.3 SSH Deployment Statistics . 76 4.3.1 SSH Implementations and Versions Statistics in the 2019 Scan 76 4.3.2 Evolution in SSH Implementations and Versions . 77 4.4 SSH Encryption Scheme Statistics . 78 4.4.1 Default SSH Encryption Schemes Preferences . 78 4.4.2 SSH Encryption Scheme Diversity . 80 4.4.3 Evolution in SSH Encryption Scheme Preferences . 80 4.5 CBC-mode Vulnerabilities . 81 4.5.1 OpenSSH . 82 4.5.2 Dropbear . 83 4.6 Less Frequent SSH Software Identifiers . 84 4.6.1 Cisco . 84 4.6.2 ROSSSH . 86 4.6.3 XXXX . 86 4.6.4 AWS SFTP 1.0 . 87 4.7 Noticeable Findings . 87 4.7.1 Age of Actively Used SSH Software . 87 4.7.2 SSH Encryption Schemes . 89 4.7.3 Vulnerable Servers . 90 5 Attacks on SSH's CBC-mode 91 5.1 The Albrecht-Paterson-Watson Attack . 91 5.1.1 Applying the Albrecht-Paterson-Watson attack to Dropbear . 93 5.1.2 OpenSSH Patch against Albrecht-Paterson-Watson Attack . 94 5.2 Timeline of New Attacks . 97 5.3 First Attack on OpenSSH CBC-mode . 97 5.3.1 Intuition . 97 5.3.2 The Bug . 98 5.3.3 Exploiting the Bug . 100 5.3.4 OpenSSH Patch against Attack One . 100 5.4 Second Attack on OpenSSH CBC-mode . 101 5.4.1 Intuition . 101 5.4.2 Description . 101 5.4.3 OpenSSH Patch against Attack Two . 103 5.5 Third Attack on OpenSSH CBC-mode . 103 5.5.1 Intuition . 103 5.5.2 Description . 104 5.5.3 OpenSSH Patch against Attack Three . 105 5.6 Experimental Results . 105 5.7 Extensions and Variants of Attack Two and Three . 106 5.8 Practical Impact, Mitigations and Recommendations . 108 6 Concrete Security of SSH Encryption Schemes 109 6.1 Modelling the OpenSSH Code . 109 6 CONTENTS 6.2 Adversary Resources Considered . 111 6.3 SSH-ChaCha20-Poly1305 in OpenSSH . 112 6.4 SSH-Generic-EtM in OpenSSH . 121 6.4.1 SSH-Fixed-Generic-EtM and IV-based Encryption . 129 6.5 SSH-AES-GCM in OpenSSH . 131 6.6 Boundary Hiding and DoS . 136 7 An Improved SSH Encryption Scheme 142 7.1 Weaknesses in SSH Encryption Schemes . 142 7.2 InterMAC . 143 7.2.1 Original InterMAC . 143 7.2.2 Modified InterMAC . 146 7.2.3 Security Analysis of IM ..................... 148 7.3 libInterMAC ................................ 155 7.3.1 Design Principles . 156 7.3.2 State Management . 157 7.3.3 Internal Nonce Construction . 157 7.4 Supported Nonce-based Symmetric Encryption Schemes . 157 7.4.1 ChaCha20-Poly1305 ........................ 158 7.4.2 AES-GCM ............................. 159 7.4.3 Why ChaCha20-Poly1305 and AES-GCM?........... 159 7.5 libInterMAC Data Limits . 160 7.5.1 ChaCha20-Poly1305 Data Limit Analysis . 161 7.5.2 AES-GCM Data Limit Analysis . 162 7.6 Side-Channels . 162 7.6.1 Constant-time Padding Removal . 163 7.6.2 Memory Allocation for InterMAC Decryption . 165 7.7 Performance Evaluation . 166 7.8 IM-based SSH Encryption Scheme in OpenSSH . 170 7.8.1 Implementation Details . 171 7.9 Performance of IM for Secure File Transfers . 173 8 Concluding Remarks 179 Bibliography 182 7 List of Figures 2.1 Algorithms for defining IND-CPA..................... 28 2.2 Algorithms for defining IND-CCA..................... 29 2.3 Algorithms for defining IND$-CPA.................... 30 2.4 Algorithms for defining INT-PTXT.................... 30 2.5 Algorithms for defining INT-CTXT.................... 31 2.6 Algorithms for defining AE........................ 33 2.7 Algorithms for defining ivE........................ 37 2.8 Algorithms for defining nAE....................... 38 2.9 MAC verification algorithm. 40 2.10 Algorithms for defining UF-CMA..................... 41 2.11 Algorithms for defining SUF-CMA.................... 42 2.12 SSH BPP packet layout. 46 2.13 Scope of encryption and authentication for SSH BPP packets. 46 3.1 Fragment f1 consisting of ciphertexts c1 = (1), c2 = (2) and part of ciphertext c3 = (34)............................ 53 3.2 Fragments f1 and f2 consisting of ciphertext fragments of ciphertext c1 = (1234)................................. 53 3.3 Fragment f1 consisting of ciphertexts c1 = (12), c2 = (345) and c3 = (6). 56 3.4 Fragments f1, f2 and f3 consisting of ciphertexts c1 = (12), c2 = (345) and c3 = (6) (respectively), where ciphertext boundaries coincide with fragment boundaries. 56 3.5 Fragments f1 and f2 consisting of ciphertexts c1 = (12) and c2 = (3). 56 3.6 Fragments f1 and f2 consisting of ciphertext c1 = (123) and extra data (40).................................. 57 3.7 Algorithms for defining IND-sfCFA advantage. 58 3.8 Algorithms for defining INT-sfCTF advantage. 60 3.9 Algorithms for defining BH-CPA and BH-sfCFA advantage. 62 3.10 Algorithms for defining DOS-sfCFA advantage. 66 3.11 The decryption oracle used in the deficient IND-sfCFA notion in [36]. 67 6.1 Cryptographic processing of an SSH BPP packet using the SSH en- cryption scheme SSH-ChaCha20-Poly1305. k1 and k2 represent the two 32-byte keys extracted from the 64-byte key. The size ratios between the boxes do not represent the size ratios between fields in a SSH BPP packet in general. 113 8 LIST OF FIGURES 6.2 The OpenSSH encryption scheme SSH-Generic-EtM prior to version 7.3 did not implement a true Encrypt-then-MAC construction.