Fast Internet-Wide Scanning: a New Security Perspective
Total Page:16
File Type:pdf, Size:1020Kb
Fast Internet-Wide Scanning: A New Security Perspective by Zakir Durumeric A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy (Computer Science and Engineering) in the University of Michigan 2017 Doctoral Committee: Professor J. Alex Halderman, Chair Associate Professor Michael Bailey, University of Illinois Urbana-Champaign Research Professor Peter Honeyman Professor Vern Paxson, University of California, Berkeley Assistant Professor Florian Schaub Zakir Durumeric [email protected] ORCID iD: 0000-0002-9647-4192 © Zakir Durumeric 2017 ACKNOWLEDGMENTS I want to thank my advisor J. Alex Halderman, as well as Michael Bailey and Vern Paxson, for their support and guidance over the past six years. I have learned an immense amount both professionally and personally from each of you, and I would not be where I am today without you. I also thank my two other committee members—Peter Honeyman and Florian Schaub—as well as Prabal Dutta and Brian Noble for their time and advice. I would not have been able to complete this dissertation without the help of family and friends. I want to thank my parents Oguz Durumeric and Robin Paetzold, and my brother Aleksander Durumeric for their unwavering support. I am further grateful for the friends who have kept me sane: Brad Campbell, Noah Klugman, Kevin Lu, Corey Melnick, Pat Pannuto, Bethany Patten, Lane Powell, and Audrey Shelton. I thank my labmates and others in the department for their camaraderie and discussion: David Adrian, Jethro Beekman, Matthew Bernhard, Denis Bueno, Mike Chow, Meghan Clark, Rob Cohn, Jakub Czyz, David Devecsery, Chris Dzombak, Denis Foo Kune, Bran- den Ghena, Grant Ho, Will Huang, James Kasten, Deepak Kumar, Kyle Lady, Frank Li, Zane Ma, Bill Marczak, Allison McDonald, Ariana Mirian, Paul Pearce, Drew Springall, Benjamin VanderSloot, Eric Wustrow, and Jing Zhang. I am particularly grateful to David Adrian, who has helped immensely on a number of projects over the past three years. Last, I want to thank the exceptional staff at the University of Michigan and Merit Network for their continuing help. This dissertation would not have been possible without Joe Adams, Kyle Banas, Sol Bermann, Jack Bernard, Chris Brenner, Kevin Cheek, Kelly Cormier, Laura Fink, Paul Howell, Michalis Kallitsis, Dan Maletta, Charlie Mattison, Steve Reger, Will Rhee, Don Winsor, and many others from ITS, CAEN, and DCO. The work in this dissertation was supported in part by the National Science Founda- tion, by the Google Anti-abuse Team, and by a Google Ph.D. Fellowship. ii TABLE OF CONTENTS Acknowledgments ................................... ii List of Figures ..................................... viii List of Tables ...................................... xiii Abstract ......................................... xviii Chapter 1 Introduction ..................................... 1 1.1 ZMap: A Fast Internet Scanner.......................2 1.2 Censys: Facilitating Easy Access to Scan Data...............3 1.3 Applications of High-Speed Scanning...................4 1.3.1 Detecting Widespread Weak Keys in Network Devices......4 1.3.2 Analyzing the HTTPS Certificate Ecosystem...........5 1.3.3 Uncovering Attacks on Email Delivery..............6 1.3.4 Tracking Vulnerability Impact...................7 1.3.5 Dissertation Roadmap.......................7 2 Fast Internet-Wide Scanning ............................ 9 2.1 Introduction.................................9 2.2 ZMap: The Scanner............................. 11 2.2.1 Addressing Probes......................... 12 2.2.2 Packet Transmission and Receipt................. 13 2.2.3 Probe Modules........................... 14 2.2.4 Output Modules.......................... 15 2.3 Validation and Measurement........................ 15 2.3.1 Scan Rate: How Fast is Too Fast?................. 16 2.3.2 Coverage: Is One SYN Enough?.................. 16 2.3.3 Variation by Time of Day..................... 18 2.3.4 Comparison with Nmap...................... 19 2.3.5 Comparison with Previous Studies................. 22 2.4 Applications and Security Implications................... 22 2.4.1 Visibility into Distributed Systems................. 22 2.4.2 Tracking Protocol Adoption.................... 24 2.4.3 Enumerating Vulnerable Hosts................... 25 2.4.4 Discovering Unadvertised Services................ 26 iii 2.4.5 Monitoring Service Availability.................. 28 2.4.6 Privacy and Anonymous Communication............. 28 2.5 Scanning and Good Internet Citizenship.................. 29 2.5.1 User Responses........................... 31 2.6 Related Work................................ 32 2.7 Future Work................................. 33 2.8 Conclusion................................. 34 3 Facilitating Easy Access to Scan Data ....................... 35 3.1 Introduction................................. 35 3.2 Good Internet Citizenship.......................... 37 3.3 Collecting Data............................... 38 3.3.1 Internet-Wide Scanning...................... 40 3.3.2 ZGrab: Our Application Scanner.................. 41 3.3.3 Validation, Extraction, and Annotation.............. 41 3.3.4 Challenges in Aggregating Data.................. 45 3.3.5 Censys Data Flow......................... 46 3.4 Exposing Data................................ 48 3.4.1 Search Interface.......................... 48 3.4.2 Programmatic Access....................... 49 3.4.3 SQL Interface............................ 49 3.4.4 Raw Data.............................. 50 3.4.5 Protocol Dashboards........................ 50 3.5 Initial Deployment............................. 50 3.5.1 Scanning.............................. 50 3.5.2 Backend.............................. 51 3.5.3 Frontend.............................. 51 3.6 Applications................................. 51 3.6.1 Industrial Control Systems..................... 51 3.6.2 Heartbleed, Poodle, and SSLv3.................. 54 3.6.3 Institutional Attack Surface.................... 55 3.6.4 Deprecating SHA-1........................ 56 3.6.5 Cipher Suites............................ 56 3.7 Related Work................................ 57 3.7.1 Scan Driven Search Engines.................... 57 3.8 Conclusion................................. 59 4 Detecting Widespread Weak Keys in Network Devices .............. 60 4.1 Introduction and Roadmap......................... 60 4.2 Background................................. 62 4.2.1 RSA review............................. 62 4.2.2 DSA review............................. 63 4.2.3 Attack scenarios.......................... 63 4.3 Methodology................................ 64 4.3.1 Internet-wide scanning....................... 64 iv 4.3.2 Identifying vulnerable device models............... 66 4.3.3 Efficiently computing all-pairs GCDs............... 67 4.4 Vulnerabilities................................ 70 4.4.1 Repeated keys........................... 70 4.4.2 Factorable RSA keys........................ 74 4.4.3 DSA signature weaknesses..................... 75 4.5 Experimental Investigation......................... 77 4.5.1 Weak entropy and the Linux RNG................. 77 4.5.2 Factorable RSA keys and OpenSSL................ 81 4.5.3 DSA signature weaknesses and Dropbear............. 84 4.6 Discussion.................................. 86 4.6.1 RSA vs. DSA in the face of low entropy.............. 86 4.6.2 /dev/(u)random as a usability failure................ 87 4.6.3 Are we seeing only the tip of the iceberg?............. 88 4.6.4 Directions for future work..................... 88 4.7 Defenses and Lessons............................ 89 4.7.1 For OS developers:......................... 89 4.7.2 For library developers:....................... 89 4.7.3 For application developers:..................... 90 4.7.4 For device manufacturers:..................... 90 4.7.5 For certificate authorities:..................... 91 4.7.6 For end users:............................ 91 4.7.7 For security and crypto researchers:................ 91 4.8 Related Work................................ 92 4.8.1 HTTPS surveys........................... 92 4.8.2 Problems with random number generation............. 93 4.8.3 Weak entropy and cryptography.................. 93 4.9 Conclusion................................. 95 5 Analyzing the HTTPS Certificate Ecosystem ................... 96 5.1 Introduction................................. 96 5.2 Background................................. 97 5.3 Related Work................................ 98 5.4 Methodology................................ 100 5.4.1 Host Discovery........................... 100 5.4.2 Collecting TLS Certificates.................... 102 5.4.3 Reducing Scan Impact....................... 102 5.4.4 Data Collection Results...................... 103 5.4.5 Is Frequent Scanning Necessary?................. 104 5.4.6 Server Name Indication Deployment............... 105 5.5 Certificate Authorities............................ 105 5.5.1 Identifying Trusted Authorities.................. 107 5.5.2 Sources of Intermediates...................... 109 5.5.3 Distribution of Trust........................ 110 5.5.4 Browser Root Certificate Stores.................. 111 v 5.5.5 Name Constraints.......................... 112 5.5.6 Path Length Constraints...................... 113 5.5.7 Authority Key Usage........................ 113 5.6 Leaf Certificates and Hosting........................ 114 5.6.1 Keys and Signatures........................ 114 5.6.2 Incorrectly Hosted Trusted Certificates.............. 117 5.6.3 Invalid Authority Types.....................