ARCHITECTURAL TECHNIQUES FOR ENABLING SECURE CRYPTOGRAPHIC PROCESSING JOHN PATRICK MCGREGOR, JR. A DISSERTATION PRESENTED TO THE FACULTY OF PRINCETON UNIVERSITY IN CANDIDACY FOR THE DEGREE OF DOCTOR OF PHILOSOPHY RECOMMENDED FOR ACCEPTANCE BY THE DEPARTMENT OF ELECTRICAL ENGINEERING JUNE 2005 Copyright °c 2005 by John Patrick McGregor, Jr. All rights reserved. I certify that I have read this thesis and that in my opinion it is fully adequate, in scope and in quality, as a dissertation for the degree of Doctor of Philosophy. Ruby B. Lee (Principal Advisor) I certify that I have read this thesis and that in my opinion it is fully adequate, in scope and in quality, as a dissertation for the degree of Doctor of Philosophy. Edward W. Felten I certify that I have read this thesis and that in my opinion it is fully adequate, in scope and in quality, as a dissertation for the degree of Doctor of Philosophy. Sun-Yuan Kung Approved by the Princeton University Graduate School: Dean of the Graduate School i Abstract Cryptographic processing is a principal enabler of many secure computing systems. Using cryptographic techniques such as encryption and secure hashing, we can satisfy several essential security requirements for networks, computers, and data against a diverse set of threats. This thesis proposes four architectural solutions to problems associated with enabling cryptographic processing in software and hardware. Two of the solutions involve protecting cryptographic keys, which are small secrets upon which cryptographic security critically depends. The other two solutions improve performance and reduce vulnerabilities in cryp- tographic software implementations. First, since the security provided by cryptographic processing depends on the secrecy and integrity of cryptographic keys, we describe a flexible system for shielding a user’s keys while in storage, transmission, and use on networked computing devices. Second, we present a new broadcast encryption system that enables the identification of users who contribute to piracy by divulging cryptographic keys that can be used to decode protected information. Third, since software rather than specialized hardware often supplies cryp- tographic functionality, we describe a method for alleviating performance problems suf- fered by cryptographic software implementations. In particular, we propose new processor instructions to improve the performance of bit-level mappings employed by several com- mon cryptographic operations. Fourth, we present a processor-based method for mitigating certain software vulnerabilities in both cryptographic and general software. The method provides built-in and dynamic protection against buffer overflow attacks, which compose ii one of the most common classes of software exploits. By applying these four contributions individually or in concert, we can achieve improved cryptographic security in existing and future systems. iii Acknowledgments I wish to offer many thanks to colleagues, friends, and family. First, I thank Professor Ruby Lee, my research advisor. Her boundless energy, keen mind, and insistence on excellence were vital to the development of this dissertation and to the completion of my graduate studies. She was always very generous in sharing her time, and she was actively involved in every aspect of my research, from the overarching ideas to the finest system details. I would like to thank Professors Ed Felten and S.-Y. Kung for reading my dissertation and for providing helpful and insightful feedback. Also, I learned much about building practical secure systems from Professor Felten’s classes and research projects. I thank Yiqun Lisa Yin for the many hours that she spent working with me on the security analysis of the traitor tracing scheme. I found our collaboration to be highly enjoyable and enlight- ening. I would also like to thank Raj Kumar, who gave me the opportunity to work in his group at Hewlett Packard Laboratories. It has been a pleasure to work with many talented engineering students at Princeton. My research has benefited from collaboration with Zhijie Shi, David Karig, Peter Kwan, Jeff Dwoskin, and Zhenghong Wang. Of particular note, David Karig and Ruby Lee were involved in the early development of the secure return address stack concept. I have also enjoyed working on projects with other members of PALMS and other Princeton EE stu- dents, including Murat Fiskiran, Xiao Yang, and Scott Craver. I also owe thanks to Matt White. In addition to being a great friend through the ups and downs of my graduate school experience, he has been rather tolerant of my unusual business schedule resulting from my all-night research spurts. iv Last but most important, I thank my family. I am forever grateful for the love, the inspiration, and the understanding that my wife Beth has given me since the first days of my journey at Princeton. And, I am very fortunate to have such caring parents who have provided unwavering support and encouragement. v Contents Abstract ii Acknowledgments iv 1 Introduction 1 1.1 Thesis Contributions . 3 1.2 Thesis Organization . 5 2 Cryptographic Processing 7 2.1 Defining Security . 8 2.1.1 Threats . 8 2.1.2 Security Goals . 9 2.2 Evaluating Security . 11 2.3 Cryptography . 14 2.3.1 Symmetric-key Encryption . 15 2.3.2 Asymmetric-key Encryption . 19 2.3.3 Cryptographic Hash Functions . 21 2.3.4 Other Primitives . 23 2.4 Applying Cryptography . 24 2.4.1 Security Protocols . 24 2.4.2 Secure Data Transmission Protocol Example . 25 2.4.3 Secure Data Storage Protocol Example . 29 2.5 Architectural Opportunities for Cryptographic Software Security and Performance . 34 vi 2.5.1 Protecting Cryptographic Keys . 34 2.5.2 Accelerating Cryptography . 35 2.5.3 Mitigating Common Software Vulnerabilities . 37 2.6 Summary . 38 3 Virtual Secure Coprocessing 40 3.1 Threats to Cryptographic Keys . 41 3.1.1 Threats to Keys in Storage . 42 3.1.2 Threats to Keys in Transport . 42 3.1.3 Threats to Keys Exercised by Software . 43 3.1.4 Threats to Keys Exercised by Hardware . 43 3.2 Past Work . 44 3.2.1 Software-based Techniques . 44 3.2.2 Cryptographic Coprocessors and Tokens . 45 3.2.3 Trusted Computing Platforms . 46 3.2.4 General-purpose Architecture for Secure Computation . 47 3.3 A New Approach to Key Protection . 48 3.3.1 Virtual Secure Coprocessing . 49 3.3.2 VSCoP Components . 52 3.3.3 VSCoP Benefits . 54 3.4 Architectural Implementation . 54 3.4.1 Key Ring Structure . 55 3.4.2 Platform Enhancements and Protected Paths . 58 3.4.3 Cryptographic Operations Library . 60 3.4.4 Processor Enhancements . 65 3.4.5 New Instructions . 70 3.4.6 Operating System Enhancements . 73 3.5 Applying VSCoP . 75 3.5.1 Device Initialization . 75 3.5.2 User Initialization . 76 3.5.3 Protected Operation . 76 vii 3.5.4 Application Example . 77 3.6 Security Analysis . 78 3.6.1 Protection for Keys in Storage . 78 3.6.2 Protection for Keys in Transit . 79 3.6.3 Protection for Keys Exercised by Software . 79 3.6.4 Protection for Keys Exercised by Hardware . 80 3.7 Performance Analysis . 81 3.8 Extensions and Alternatives . 86 3.9 Summary . 87 4 A Traceability Scheme for Broadcast Encryption 89 4.1 Broadcast Encryption . 90 4.1.1 Broadcast Encryption Model . 90 4.1.2 Example Operation . 92 4.1.3 Attacks and Defenses . 96 4.2 Past Work . 97 4.3 A New Traitor Tracing Scheme . 102 4.3.1 RSA Preliminaries . 103 4.3.2 Components and Parameters . 105 4.3.3 Provider Initialization . 106 4.3.4 User Initialization . 107 4.3.5 Encryption, Transmission, and Decryption . 108 4.4 Security Analysis . 108 4.4.1 Security against Unauthorized Users . 109 4.4.2 Security against Traitor Collusions . 111 4.4.3 Security against Attacks on RSA . 117 4.4.4 Choosing the Parameters . 118 4.5 Identifying Traitors . 119 4.5.1 A Clear-box Tracing Algorithm . 120 4.5.2 A Limited Black-box Tracing Algorithm . 121 4.6 Performance Analysis . 124 viii 4.6.1 Provider Initialization Costs . 125 4.6.2 User Initialization Costs . 125 4.6.3 Encryption and Transmission Costs . 127 4.6.4 Decryption Costs . 127 4.6.5 Tracing Algorithm Costs . 129 4.7 A Key Renewal and Revocation Protocol . 131 4.8 Summary . 133 5 Processor Support for Fast Subword Mappings 135 5.1 Subword Processing, Permutations, and Mappings . 136 5.2 Past Work . 137 5.3 New Instructions for Subword Mappings . 140 5.3.1 Preliminaries . 140 5.3.2 The swperm Instruction . 141 5.3.3 The sieve Instruction . 142 5.4 Applying the Instructions . 146 5.4.1 Mapping 1-bit and 2-bit Subwords . 146 5.4.2 Mapping 4-bit or Larger Subwords . 149 5.4.3 Generating the Configuration Information . 149 5.4.4 Mappings in Large Values . 152 5.5 Hardware Implementation . 155 5.6 Performance Analysis . 159 5.6.1 Impact on 64-bit Permutations and Mappings . 159 5.6.2 Impact on the Data Encryption Standard . 162 5.7 Summary . 169 6 A Hardware Defense against Buffer Overflows 171 6.1 Buffer Overflows and Return Address Corruption . ..