UNIVERSITY OF CALIFORNIA, SAN DIEGO Language-based Security for Web Browsers A dissertation submitted in partial satisfaction of the requirements for the degree of Doctor of Philosophy in Computer Science by Dongseok Jang Committee in charge: Professor Sorin Lerner, Chair Professor Samuel Buss Professor Ranjit Jhala Professor Todd Millstein Professor Hovav Shacham 2014 Copyright Dongseok Jang, 2014 All rights reserved. The Dissertation of Dongseok Jang is approved and is acceptable in quality and form for publication on microfilm and electronically: Chair University of California, San Diego 2014 iii DEDICATION To my family. iv EPIGRAPH Meanwhile, the poor Babel fish, by effectively removing all barriers to communication between different races and cultures, has caused more and bloodier wars than anything else in the history of creation. Douglas Adams “The Hitchhiker’s Guide to the Galaxy” v TABLE OF CONTENTS Signature Page . iii Dedication . iv Epigraph . v Table of Contents . vi List of Figures . viii List of Tables . ix Abstract of the Dissertation . x Chapter 1 Introduction . 1 1.1 Outline of this work. 5 1.2 Organization . 7 Chapter 2 Securing JavaScript via Dynamic Information Flow Tracking . 8 2.1 Motivation . 9 2.2 Information Flow Policies . 12 2.2.1 Policy Language . 13 2.2.2 Policy Enforcement . 17 2.2.3 Indirect Flows . 23 2.3 Implementation and Performance Evaluation . 26 2.3.1 Policies . 27 2.3.2 Timing Measurements . 30 2.4 Empirical Study of History Sniffing . 33 2.5 Empirical Study of Behavior Tracking . 40 Chapter 3 Securing C++ Virtual Function Calls . 46 3.1 Motivation . 46 3.2 SAFEDISPATCH Overview . 50 3.2.1 Dynamic Dispatch in C++ . 50 3.2.2 vtable Hijacking . 53 3.2.3 SAFEDISPATCH vtable Protection . 56 3.3 The SAFEDISPATCH Compiler . 57 3.3.1 Class Hierarchy Analysis . 58 3.3.2 SAFEDISPATCH Method Checking Instrumentation . 62 3.3.3 SAFEDISPATCH Optimizations . 65 3.4 An Alternate Approach: Vtable Checking . 67 3.4.1 Pointer Offsets for Multiple Inheritance . 68 vi 3.4.2 vtable Checking . 69 3.4.3 Performance Implications . 71 3.5 A Hybrid Approach for Method Pointers . 71 3.5.1 Revisiting Previous Approaches . 73 3.5.2 Hybrid Approach . 74 3.6 Evaluation . 74 3.6.1 SAFEDISPATCH Overhead . 74 3.6.2 Development Effort . 80 3.6.3 Compatibility . 81 3.7 SAFEDISPATCH Security Analysis . 82 3.7.1 SAFEDISPATCH Guarantee . 82 3.7.2 SAFEDISPATCH Limitations . 83 3.7.3 Performance and Security Tradeoffs . 85 Chapter 4 Formal Verification for Kernel-based Browsers . 87 4.1 Motivation . 88 4.2 QUARK Architecture and Design . 91 4.2.1 Graphical User Interface . 93 4.2.2 Example of Message Exchanges. 95 4.2.3 Efficiency . 97 4.2.4 Socket Security Policy . 98 4.2.5 Cookies and Cookie Policy . 100 4.2.6 Security Properties of QUARK ............................ 101 4.3 Kernel Implementation in Coq . 102 4.4 Kernel Verification. 107 4.4.1 Actions and Traces . 107 4.4.2 Kernel Specification . 108 4.4.3 Monads in Ynot Revisited . 110 4.4.4 Back to the Kernel . 111 4.4.5 Security Properties . 112 4.5 Evaluation . 118 4.6 Discussion . 124 Chapter 5 Related Work . 127 5.1 Securing JavaScript via Dynamic Information Flow Tracking . 127 5.2 Securing C++ Virtual Function Calls . 129 5.3 Formal Verification for Kernel-based Browsers . 133 Chapter 6 Conclusions and Future Work . 137 Bibliography . 141 vii LIST OF FIGURES Figure 2.1. JavaScript code on a:com importing untrusted code from ad servers. 12 Figure 2.2. Rewritten code from a:com ................................. 19 Figure 2.3. Slowdown for JavaScript and Page Loading . 31 Figure 2.4. Attack code as found on youporn.com . 35 Figure 3.1. C++ Dynamic Dispatch . 51 Figure 3.2. Example vtable Hijacking . 54 Figure 3.3. SAFEDISPATCH Protection . 57 Figure 3.4. Example Class Hierarchy Analysis (CHA) . 59 Figure 3.5. Our CHA which constructs ValidM at Compile Time . 60 Figure 3.6. SAFEDISPATCH Instrumentation . 63 Figure 3.7. Low-level SAFEDISPATCH Optimization . 66 Figure 3.8. Alternate SAFEDISPATCH vtable Checking . 70 Figure 3.9. Method Pointer Example . 72 Figure 3.10. SAFEDISPATCH Overhead . 75 Figure 4.1. QUARK Architecture. 92 Figure 4.2. QUARK Screenshot . 94 Figure 4.3. Body for Main Kernel Loop . 103 Figure 4.4. Traces and Actions . ..