Open Dissertation-Wenhuihu.Pdf
Total Page:16
File Type:pdf, Size:1020Kb
The Pennsylvania State University The Graduate School THIRD-PARTY LIBRARY SECURITY MANAGEMENT FOR MOBILE APPLICATIONS A Dissertation Document in Computer Science and Engineering by Wenhui Hu c 2016 Wenhui Hu Submitted in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy December 2016 The dissertation document of Wenhui Hu was reviewed and approved∗ by the following: Patrick D. McDaniel Distinguished Professor of Computer Science and Engineering Dissertation Co-Advisor, Co-Chair of Committee Peng Liu Professor of Information Sciences and Technology Dissertation Co-Advisor, Co-Chair of Committee Trent R. Jaeger Professor of Computer Science and Engineering Constantino Lagoa Professor of Electrical Engineering Damien Octeau Postdoctoral Scholar, University of Wisconsin and Pennsylvania State University Special Member Chita R. Das Distinguished Professor of Computer Science and Engineering Interim Head of the Department of Computer Science and Engineering ∗Signatures are on file in the Graduate School. Abstract In recent years, the Android operating system has had an explosive growth in the number of applications containing third-party libraries for different purposes. In this thesis, we identify three library-centric threats in the real-world Android application markets: (i) the library modification threat, (ii) the masquerading threat and (iii) the aggressive library threat. These three threats cannot effectively be fully addressed by existing defense mechanisms such as software analysis, anti-virus software and anti-repackaging techniques. The limitation of existing mechanisms is that the existing mechanisms ignore the fact the library and applications coexist, but are from different stakeholder in the ecosystem. For existing libraries, related security problems in applications, we propose a li- brary integrity verification tool for Android applications at application stores. This is non-trivial because the Android application build process merges library code and application-specific logic into a single binary file. Our approach uses reverse-engineering to achieve integrity verification. The evaluation results also provide empirical insight into the library integrity situation in the wild. In addition, third-party libraries are detected to contain potentially malicious behav- iors. With the Android security model, the core logic and the third-party library share permissions. Solutions are provided to enforce library isolation. Unfortunately, libraries in the isolation still share some permissions with the core logic. In order to generate distinct policies for third-party libraries, we need to know which part of libraries works for the core logic. Existing analysis tools cannot differentiate libraries' behaviors assisting for the appli- cation's main functionality from libraries' behaviors for itself because of two limitations: 1) the analysis cannot do backwards analysis; 2) the analysis cannot distinguish different stakeholders within one application. This shortcoming leads to the difficulty of creating a precise policy for library isolation solutions. The missing part is the intra-app cross- stakeholder data-flow analysis. To address it, we propose a callsite-aware bidirectional static analysis tool. The tool enables to classify the Android framework APIs in libraries into six categories. The classification information can be used to generate polices. The dissertation also has an enforcement of the library isolation. iii Table of Contents List of Figures viii List of Tables ix Acknowledgments xi Chapter 1 Introduction1 1.1 Library-Centric Security Threats...................... 2 1.2 Library based Isolation............................ 3 1.3 Thesis Statement............................... 4 1.4 Threat Models ................................ 5 1.4.1 The Threat Model of the Library Integrity Verification...... 5 1.4.2 The Threat Model of the Policy Generation............ 6 1.5 Contributions................................. 7 Chapter 2 Preliminary Concepts9 2.1 Libraries in Android............................. 9 2.1.1 Android Application Compilation.................. 11 2.1.2 Library Post-Processing....................... 11 2.2 Limitation of Android Security Model ................... 12 2.2.1 Android Security Model....................... 12 2.2.2 Madware and application developers................ 12 2.3 Relationship between Library and Application Logic........... 13 Chapter 3 Related Work 14 3.1 Smartphone Platform Security ....................... 14 3.2 Application Behavior Modification..................... 15 3.3 Android Application Library ........................ 16 3.3.1 Library Detection .......................... 16 3.3.2 Advertising Library ......................... 16 iv 3.3.3 Library Removal........................... 17 3.3.3.1 Native Library....................... 17 3.3.3.2 Other Approaches..................... 18 3.4 Information Flow Analysis Tools ...................... 18 3.5 Isolation.................................... 19 3.6 Other Security Solutions to Android Applications............. 19 3.6.1 Repackaging ............................. 19 3.6.2 Code Clones ............................. 20 3.6.3 Inline reference monitors for Java applications .......... 20 3.7 Malware Detection.............................. 20 Chapter 4 Duet: Library Integrity Verification 21 4.1 Introduction.................................. 21 4.2 Security Model and Problem Statement .................. 24 4.2.1 Security Model............................ 24 4.2.2 Problem Statement.......................... 26 4.2.3 Use Cases............................... 26 4.3 System Overview............................... 27 4.3.1 Naive Solutions and Challenges for Store Side Verification . 27 4.3.2 Our Idea (Double Reverse-Engineering).............. 28 4.3.3 Architecture of Duet......................... 28 4.3.4 Why this solution works? ...................... 30 4.4 Design and Implementation......................... 30 4.4.1 Reverse-Engineering Requirements................. 30 4.4.1.1 Potential Attack Surface ................. 31 4.4.2 Building the Reference Database.................. 32 4.4.2.1 Canonical Library Digest................. 32 4.4.2.2 Canonical File Digest................... 35 4.4.2.3 Issues............................ 35 4.4.3 Library Integrity Verification in Applications........... 36 4.5 Evaluation................................... 36 4.5.1 Libraries in Android Applications.................. 37 4.5.2 In-lab testing of the correctness of Duet.............. 40 4.5.3 In-the-wild Library Integrity Verification Results......... 40 4.5.4 Implications of Passing Rates.................... 44 4.5.5 Performance ............................. 44 4.6 In-depth Analysis of Applications that do not Pass Verification . 46 Chapter 5 Policy Generation for Library Isolation 49 5.1 Introduction.................................. 49 5.2 Background and Potential Applications .................. 53 5.2.1 Libraries for Android applications ................. 53 5.2.2 Dilemma for application developers................. 53 v 5.2.3 Applications of Intra-App Cross-Stakeholder Analysis . 54 5.3 Reducing Intra-App Cross-Stakeholder Module Relationship Analysis to a Data-Flow Analysis Problem ....................... 56 5.3.1 Definition of Intra-Application Modules.............. 56 5.3.2 Definition of Operations....................... 56 5.3.3 Data-flow Analysis for Read and Write Operations . 57 5.3.4 Understanding the Data-flow Analysis Results .......... 58 5.3.5 API Classification and Policy Generation ............. 61 5.4 System Overview............................... 62 5.4.1 Limitations.............................. 64 5.5 Bidirectional Callsite-aware Static Taint Analysis............. 64 5.5.1 Precise Modelling of Lifecycle.................... 64 5.5.2 Precise Bidirectional Taint Analysis ................ 64 5.5.2.1 IFDS Extension to Support Unbalanced Return Flows . 65 5.5.2.2 On-demand Forward Alias Analysis........... 65 5.5.2.3 Transfer Functions of Backwards Taint Analysis . 65 5.5.3 Callsite-Aware Extension ...................... 66 5.6 Evaluation................................... 66 5.6.1 RQ1: How serious is the intra-app cross-stakeholder security threats? 67 5.6.2 RQ2: What precision is achieved by the tool?........... 68 5.6.3 RQ3: Do we need a bidirectional static taint analysis tool? . 68 5.6.4 RQ4: How can this tool to classify the Android framework APIs in libraries of real Android applications?.............. 69 5.6.4.1 Summary of Classification Results............ 70 5.6.4.2 Results for each library Category............. 72 5.6.4.3 Overall Results....................... 77 Chapter 6 LibGuardian: Library Isolation 79 6.1 Introduction.................................. 79 6.2 Background.................................. 81 6.2.1 Stakeholders within the Ecosystem................. 81 6.2.1.1 Case Study: Advertising Library and Madware . 83 6.2.2 Problem Statement.......................... 83 6.2.3 Use Cases............................... 83 6.3 System Overview............................... 84 6.3.1 Design Choices............................ 84 6.3.2 Our Idea ............................... 85 6.3.3 Architecture of LibGuardian .................... 85 6.4 Design and Implementation......................... 86 6.4.1 Library Usage Analysis ....................... 86 6.4.2 Application Retrofit......................... 88 6.4.2.1 Android framework API Interception .......... 88 vi 6.4.2.2 Intercepted Method Handler Generation......... 88 6.4.2.3 Other