Functionality-Based Application Confinement: A Parameterised and Hierarchical Approach to Policy Abstraction for Rule-based Application-oriented Access Controls Z. Cliffe Schreuders BSc Computer Science (First Class Honours) and Internet Computing This thesis is presented for the degree of Doctor of Philosophy of Murdoch University, 2012 Declaration I declare that this thesis is my own account of my research and contains as its main content work that has not previously been submitted for a degree at any tertiary education institution. Z. Cliffe Schreuders. i Abstract Access controls are traditionally designed to protect resources from users, and consequently make access decisions based on the identity of the user, treating all processes as if they are acting on behalf of the user that runs them. However, this user-oriented approach is insufficient at protecting against contemporary threats, where security compromises are often due to applications running malicious code, either due to software vulnerabilities or malware. Application-oriented ac- cess controls can mitigate this threat by managing the authority of individual ap- plications. Rule-based application-oriented access controls can restrict applica- tions to only allow access to the specific finely-grained resources required for them to carry out their tasks, and thus can significantly limit the damage that can be caused by malicious code. Unfortunately existing application-oriented access controls have policy complexity and usability problems that have limited their use. This thesis proposes a new access control model, known as functionality-based application confinement (FBAC). The FBAC model has a number of unique fea- tures designed to overcome problems with previous approaches. Policy abstrac- tions, known as functionalities , are used to assign authority to applications based on the features they provide. Functionalities authorise elaborate sets of finely grained privileges based on high-level security goals, and adapt to the needs of specific applications through parameterisation. FBAC is hierarchical, which en- ables it to provide layers of abstraction and encapsulation in policy. It also sim- ultaneously enforces the security goals of both users and administrators by providing discretionary and mandatory controls. An LSM-based (Linux security module) prototype implementation, known as FBAC-LSM, was developed as a proof-of-concept and was used to evaluate the new model and associated techniques. The policy requirements of over one hun- dred applications were analysed, and policy abstractions and application policies were developed. Analysis showed that the FBAC model is capable of represent- ing the privilege needs of applications. The model is also well suited to automa- ii tion techniques that can in many cases create complete application policies a priori , that is, without first running the applications. This is an improvement over previous approaches that typically rely on learning modes to generate poli- cies. A usability study was conducted, which showed that compared to two widely-deployed alternatives (SELinux and AppArmor), FBAC-LSM had signif- icantly higher perceived usability and resulted in significantly more protective policies. Qualitative analysis was performed and gave further insight into the is- sues surrounding the usability of application-oriented access controls, and con- firmed the success of the FBAC model. iii Contents Declaration ..................................................................................................................... i Abstract ......................................................................................................................... ii Contents ....................................................................................................................... iv Figures .......................................................................................................................... ix Tables ........................................................................................................................... xi Preface ......................................................................................................................... xii Acknowledgements .................................................................................................... xiv Publications and Conference Presentations ................................................................. xv Chapter 1 Introduction ................................................................................................ 1 1.1 Computer Insecurity ....................................................................................... 1 1.2 Insufficient Application Restrictions ............................................................. 1 1.3 Usability and Application Confinement......................................................... 3 1.4 Flexibility and Application Confinement....................................................... 3 1.5 Research Aims ............................................................................................... 4 1.6 Research Methodology................................................................................... 6 1.7 Thesis Structure .............................................................................................. 7 1.8 Chapter Conclusion ........................................................................................ 8 Chapter 2 Background and Literature Review ......................................................... 10 2.1 Chapter Introduction .................................................................................... 10 2.2 Limitations of User-oriented Access Controls ............................................. 11 2.3 Existing User-oriented Access Controls ...................................................... 13 2.3.1 Role-Based Access Control (RBAC) ................................................... 13 2.3.2 RBAC Standards .................................................................................. 15 2.3.3 Advantages of RBAC ........................................................................... 16 2.3.4 RBAC Implementations ....................................................................... 17 2.4 Application-oriented Access Control ........................................................... 17 2.4.1 Policy Providers.................................................................................... 18 2.4.2 Isolation Sandboxes and Virtualisation ................................................ 20 2.4.3 Disadvantages of Isolation-based Approaches ..................................... 23 2.4.4 Rule-based Sandboxes .......................................................................... 25 2.4.5 Rule-based System-wide Controls ....................................................... 27 2.4.6 Problems With Rule-based Application Restriction Schemes ............. 31 2.4.7 Policy Complexity of Rule-based Approaches..................................... 35 2.4.8 Application Restrictions and Usability ................................................. 37 2.4.9 Chapter Conclusion .............................................................................. 39 Chapter 3 Previous Work .......................................................................................... 40 3.1 Chapter Introduction .................................................................................... 40 3.2 Role-Based Execution Environment (RBEE) .............................................. 40 3.3 Adaptation of RBAC Elements .................................................................... 41 3.4 Adapting RBAC Functionality ..................................................................... 42 3.5 Execution and Process Ancestry .................................................................. 43 3.6 Functionality Parameterisation..................................................................... 44 3.7 Comparing RBAC and RBEE ...................................................................... 45 3.8 Proposed Advantages and the Limitations of RBEE ................................... 46 3.9 FBAC’s relation to RBEE ............................................................................ 47 3.10 Chapter Conclusion ...................................................................................... 48 Chapter 4 The Functionality-based Application Confinement Model...................... 49 iv 4.1 Chapter Introduction .................................................................................... 49 4.2 Access Control Aims ................................................................................... 49 4.3 Threats FBAC Mitigates .............................................................................. 50 4.4 Access Control Model Overview ................................................................. 51 4.5 Functionality-based Component .................................................................. 52 4.5.1 Discussion of the Functionality-based Component .............................. 56 4.6 Hierarchical FBAC Component ................................................................... 57 4.6.1 Discussion of the Hierarchical Component .......................................... 57 4.7 Parameterised FBAC Component ................................................................ 58 4.7.1 Discussion of the Parameterised Component ....................................... 60 4.8 FBAC User-Confinements