A Methodology for Trustworthy File Systems Sidney Amani Submitted in fulfilment of the requirements for the degree of Doctor of Philosophy School of Computer Science and Engineering Faculty of Engineering August 2016 PLEASE TYPE THE UNIVERSITY OF NEW SOUTH WALES Thesis/Dissertation Sheet Surname or Family name: Amani First name: Sidney Other name/s: Abbreviation for degree as given in the University calendar: PhD School: School of Computer Science and Engineering Faculty: Faculty of Engineering Title: A methodology for trustworthy file systems Abstract 350 words maximum: (PLEASE TYPE) The main contribution of this thesis is a methodology for designing, implementing and verifying realistic file systems with evidence of its effectiveness by application to a non-trivial flash file system. The goal of this research is to demonstrate that an implementation level machine-checked proof of correctness of a file system can be done at reasonable cost. Our approach leverages Cogent, a purely functional, memory and type-safe language we helped design, that bridges the gap between verifiable formal model and low-level code. Cogent takes a modular file system implementation as input and generates a C implementation and a formal proof that links it to its corresponding generated Cogent specification. Cogent specifications inherit the purely functional aspect of the input source code, and thus they proved much easier to reason about than the C code directly. In order to prove the correctness of complex file system components at reasonable cost, we decompose the functionality into a set of components such that the correctness of each can be proven in isolation. The component proofs are mechanically composed into a theorem that holds on the C implementation by refinement. To validate our approach, we designed and implemented BilbyFs, a modular flash file system. We formally specified BilbyFs’ file system operations in Isabelle/HOL, and proved the functional correctness of two key operations: sync() and iget(). BilbyFs’ design demonstrates the practicality of aggressive modular decomposition, and its Cogent implementation highlights the benefits and feasibility of using a linearly-typed language to implement a realistic file system. Our verification results show that we can exploit our modular design to reason about implementation components in isolation, and that overall our methodology drastically reduces the effort of verifying file system code. Declaration relating to disposition of project thesis/dissertation I hereby grant to the University of New South Wales or its agents the right to archive and to make available my thesis or dissertation in whole or in part in the University libraries in all forms of media, now or here after known, subject to the provisions of the Copyright Act 1968. I retain all property rights, such as patent rights. I also retain the right to use in future works (such as articles or books) all or part of this thesis or dissertation. I also authorise University Microfilms to use the 350 word abstract of my thesis in Dissertation Abstracts International (this is applicable to doctoral theses only). …………………………………………………………… ……………………………………..……………… Signature Witness Signature Date The University recognises that there may be exceptional circumstances requiring restrictions on copying or conditions on use. Requests for restriction for a period of up to 2 years must be made in writing. Requests for a longer period of restriction may be considered in exceptional circumstances and require the approval of the Dean of Graduate Research. FOR OFFICE USE ONLY Date of completion of requirements for Award: THIS SHEET IS TO BE GLUED TO THE INSIDE FRONT COVER OF THE THESIS Originality Statement ‘I hereby declare that this submission is my own work and to the best of my knowledge it contains no materials previously published or written by another person, or substantial proportions of material which have been accepted for the award of any other degree or diploma at UNSW or any other educational institution, except where due acknowledge- ment is made in the thesis. Any contribution made to the research by others, with whom I have worked at UNSW or elsewhere, is explicitly acknowledged in the thesis. I also declare that the intellectual content of this thesis is the product of my own work, except to the extent that assistance from others in the project’s design and conception or in style, presentation and linguistic expression is acknowledged.’ Signed ............................................................................. Date ............................................................................... ii Copyright Statement ‘I hereby grant the University of New South Wales or its agents the right to archive and to make available my thesis or dissertation in whole or part in the University libraries in all forms of media, now or here after known, subject to the provisions of the Copyright Act 1968. I retain all proprietary rights, such as patent rights. I also retain the right to use in future works (such as articles or books) all or part of this thesis or dissertation. I also authorise University Microfilms to use the 350 word abstract of my thesis in Dissertation Abstract International (this is applicable to doctoral theses only). I have either used no substantial portions of copyright material in my thesis or I have obtained permission to use copyright material; where permission has not been granted I have applied/will apply for a partial restriction of the digital copy of my thesis or dissertation.’ Signed ............................................................................. Date ............................................................................... Authenticity Statement ‘I certify that the Library deposit digital copy is a direct equivalent of the final officially approved version of my thesis. No emendation of content has occurred and if there are any minor variations in formatting, they are the result of the conversion to digital format.’ Signed ............................................................................. Date ............................................................................... iii iv Abstract The main contribution of this thesis is a methodology for designing, imple- menting and verifying realistic file systems with evidence of its effectiveness by application to a non-trivial flash file system. The goal of this research is to demonstrate that an implementation level machine-checked proof of correctness of a file system can be done at reasonable cost. Our approach leverages Cogent, a purely functional, memory- and type- safe language we helped design, that bridges the gap between verifiable formal model and low-level code. Cogent takes a modular file system implementation as input and generates a C implementation and a formal proof that links it to its corresponding generated Cogent specification. Cogent specifications inherit the purely functional aspect of the input source code, and thus they proved much easier to reason about than the C code directly. In order to prove the correctness of complex file system components at reasonable cost, we decompose the functionality into a set of components such that the correctness of each can be proven in isolation. The component proofs are mechanically composed into a theorem that holds on the C implementation by refinement. To validate our approach, we designed and implemented BilbyFs, a modular flash file system. We formally specified BilbyFs’ file system operations in Isabelle/HOL, and proved the functional correctness of two key operations: sync() and iget(). BilbyFs’ design demonstrates the practicality of aggressive modular de- composition, and its Cogent implementation highlights the benefits and feasibility of using a linearly-typed language to implement a realistic file system. Our verification results show that we can exploit our modular design to reason about implementation components in isolation, and that overall our methodology drastically reduces the effort of verifying file system code. v Acknowledgements This thesis would not have been possible without the help of Toby Murray, my main supervisor during my PhD program, whose supervision style was great. Toby, thank you for your patience over the years and for cheering me up when the future did not look bright. I am also indebted to Leonid Ryzhyk, who was my supervisor during the first year of my PhD. Leonid, you have been a model of diligence and integrity to me. I truly admire you as a colleague and, more importantly, as a friend. This adventure would not have been possible without the support of Gernot Heiser, Gabi Keller and Gerwin Klein who were always willing to help and never lost their trust in me. The rest of the Trustworthy File Systems team also played a significant role in this endeavour. It was a great pleasure to work with so many talented people. More specifically, I would like to thank Zilin Chen, Christine Rizkallah, Alex Hixon, Yutaka Nagashima, Liam O’Connor, Thomas Sewell, Joel Beeren, Japheth Lim and Keng Chai Ong. I am grateful that I had the support of my family: my parents, Bernardine and N’Goran Amani; my siblings, Steve, Johan, Gisele and their own families. I cannot enumerate all the many friends I am fortunate to have, but I would like to give a special thank to the members of Boyz In Da Hood. Thank you all for being supportive during that journey. Finally I would like to thank Lucy, whose love and support have been crucial. Lucy, without your support, I would certainly