Fuzzing Windows Applications and Network Protocols Bachelor Thesis
Total Page:16
File Type:pdf, Size:1020Kb
Fuzzing Windows Applications and Network Protocols Bachelor Thesis Department of Computer Science University of Applied Science Rapperswil Spring Term 2012 Authors: Kevin Lynn, Michael Fisler Advisor: Prof. Dr. Andreas Steffen Project Partner: Compass Security AG External Co-Examiner: Dr. Ralf Hauser Internal Co-Examiner: Prof. Dr. Luc Bläser Abstract Fuzzing is a technique for detecting software flaws by intentionally sending invalid input to a target of evaluation, generally involving a high degree of automation. For software engineers it is crucial to identify and eliminate such flaws since they might be exploitable by a remote attacker. As a consequence an attacker could compromise the application as well as the operating system the software is running on, gain unauthorized access and steal or modify confidential data. Fuzzing provides an instrument to find bugs fast with relatively low costs. While the basic idea behind fuzzing is simple, creating thorough, precise and performant fuzzers is a real challenge. With time and computing power being limiting factors, the success of fuzzing depends on the level of detail when modeling the protocol as well as the effectiveness of the data mutations performed. The goal of this paper is to present methods to effectively fuzz a network (i.e. server) application using an example protocol and a custom fuzzer. By choosing the standards based Extensible Messaging and Presence Protocol (XMPP) we not only selected a contemporary protocol that has a gaining popularity in real-time communication applications but that also incorporates many advanced concepts found in other common networking protocols. The scope of the XMPP protocol including all its extensions is vast and therefore presents a major challenge in terms of protocol modeling and target coverage. As our tool of choice we selected the Peach Fuzzing Platform for fuzzer modeling. Peach is a free, powerful framework including many automation features. Partly due to the documentation, which in some areas is very scarce, Peach has quite a steep learning curve. By documenting our experience, successes and mistakes we aim to reduce the initial effort required to become acquainted with Peach and its varied features. Further- more, by following an incremental approach to build-up expertise moving from simple to more challenging problems and techniques proved a viable project guideline. In addition to general fuzzing considerations such as the circumstances under which fuzzing makes sense and the limitations of fuzzing we also present methods to approach fuzzing extensive XML-based protocols and tuning Peach to achieve higher performance and point out strategies to a high level of coverage and precision. Contents I. Technical Report 9 1. Introduction 10 1.1. Types of Fuzzing . 10 1.2. Attack Surface . 11 1.3. Fuzzer Types . 11 1.4. Failure Analysis . 11 1.5. Fuzzing Approaches Of Major Companies . 11 1.6. State of the Art Fuzzing Tools . 12 1.7. Project Goals & Vision . 12 1.8. General Conditions, Environment, Definitions, Limitations . 12 1.9. Procedure & Setup . 12 1.10. Chapter Overview . 13 2. Evaluation 14 2.1. Protocol . 14 2.1.1. Criteria . 14 2.1.2. Assessment . 14 2.1.3. Conclusion . 15 2.2. Fuzzer . 16 2.2.1. Fuzzing Frameworks . 16 2.2.2. Criteria . 18 2.2.3. Assessment . 18 2.2.4. Conclusion . 19 2.3. Open Source XMPP Server . 20 2.3.1. Criteria . 20 2.3.2. Assessment . 20 2.3.3. Conclusion . 20 2.4. Closed Source XMPP Server . 21 2.4.1. Criteria . 21 2.4.2. Assessment . 21 2.4.3. Conclusion . 21 3. The XMPP Protocol 22 3.1. XMPP Core Technology . 22 3.2. XML Stream . 22 3 3.3. Roster . 22 3.4. Stanza . 22 3.5. Simple Authentication and Security Layer (SASL) . 23 3.6. Client Registration . 23 4. The Peach Fuzzing Framework 24 4.1. General Peach Concepts . 24 4.2. Peach Fuzzer . 24 4.3. Peach Agent . 26 4.4. Peach Fuzzer Configuration File (Pit File) . 26 4.5. Peach Mutators . 29 4.5.1. Mutation Strategies . 29 4.5.2. Peach’s Default Mutators . 29 4.6. Peach Fixup . 31 4.6.1. Creating Custom Fixups . 32 4.6.2. Including The Fixup in the DataModel . 33 4.7. Slurp Action . 33 4.8. Peach Publisher . 33 5. Realization 35 5.1. Proof of Concept: Savant Web Server . 35 5.2. Proof of Concept: Known-Exploit Fuzzing . 35 5.3. General Fuzzing Considerations . 35 5.4. Choice Of Database . 36 5.5. Authentication . 36 5.6. XMPP-specific Peach DataModels . 37 5.6.1. JID . 37 5.6.2. DateTime . 38 5.7. Peach Mutator Optimization . 38 5.7.1. General String Mutators . 39 5.7.2. Overflow Mutator . 40 5.7.3. Invalid Char Mutators . 40 5.8. XMPP Schema Coverage: Input Space Determination . 40 5.9. Server Feature Coverage . 44 5.10. Code Coverage, Fuzzer Tracking . 45 5.10.1. PaiMei . 46 5.10.2. White-box code coverage . 47 5.10.3. Grey-box code coverage . 47 5.10.4. Results . 47 5.11. Logger Considerations . 47 6. Results 49 6.1. Benefits of Fuzzing . 49 4 6.2. Limitations of Fuzzing . 49 6.2.1. Time-Detail-Precision Tradeoff . 49 6.2.2. How Secure Is A Fuzzed Software? . 49 6.2.3. Misconfiguration . 50 6.2.4. Automated Fuzzer Creation . 50 6.3. Strategies for Efficient and Effective Fuzzing . 50 6.3.1. Selection of Features . 50 6.3.2. Mutator Quality . 51 6.3.3. Modeling Detail . 51 6.3.4. Features of an Optimal Fuzzing Framework . 51 6.3.5. Combining mutation-based and generation-based fuzzing . 53 6.4. Future Work . 53 6.4.1. Mutators . 53 6.4.2. Fuzzer Tracking / Code Coverage . 53 6.5. Personal Reports . 54 6.5.1. Kevin Lynn . 54 6.5.2. Michael Fisler . 54 6.6. Acknowledgments . 55 II. Software Documentation 56 7. Overview of Tools 57 7.1. Fuzzer Assisting and Analysis Tools . 57 7.1.1. config.py . 57 7.1.2. xmpp_schema_analyzer.py . 58 7.1.3. jabberd_mysql_reset.py . 58 7.2. Pit Generation Scripts . 58 7.2.1. pit_creator.py . 58 7.2.2. xep_ripper.py . 59 7.2.3. configure.py . 60 7.2.4. tag_dupl.py . 60 7.2.5. xml2sm.py . 61 7.2.6. xml2pdm.py . 62 8. Software Decisions 64 8.1. Software Engineering Decisions . 64 8.1.1. Script Robustness . 64 8.1.2. Script Scope . ..