DOCSLIB.ORG

A Diagnostic Tool for FUSE Modules Final Report

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
A Diagnostic Tool for FUSE Modules Final Report University of St Andrews Senior Honours Project A Diagnostic Tool for FUSE Modules Final Report Matthew Dooler April 2014 Supervisor: Dr. Graham Kirby 0.1 Abstract Filesystems provide the mechanisms for retrieval and storage of information on a computer, often organising files using a hierarchical naming scheme. Filesystems generally abstract over local physical media while others access information over the network. FUSE (Filesystem in Userspace) is a framework allowing filesystems to be implemented in user space, avoiding modification of the kernel by developers and users. This project presents a software diagnostic tool that facilitates de- velopment of FUSE filesystems. The tool provides three main facilities: a wizard that interactively guides developers through implementation of a FUSE filesystem from scratch, an automated test suite, and a debugger. i 0.2 Declaration I declare that the material submitted for assessment is my own work except where credit is explicitly given to others by citation or acknowledgement. This work was performed during the current academic year except where otherwise stated. The main text of this project report is 15,850 words long, including project specification and plan. In submitting this project report to the University of St Andrews, I give permission for it to be made available for use in accordance with the regulations of the University Library. I also give permission for the title and abstract to be published and for copies of the report to be made and supplied at cost to any bona fide library or research worker, and to be made available on the World Wide Web. I retain the copyright in this work. ii 0.3 Acknowledgements I wish to express my sincere appreciation to Dr. Graham Kirby for his guidance during practical work and write-up of this project. I am also grateful to Prof Alan Dearle for his advice during the initiation of this project. iii Contents 0.1 Abstract . .i 0.2 Declaration . ii 0.3 Acknowledgements . iii 1 Introduction 1 1.1 Context . .1 1.2 Problem definition . .2 1.2.1 Filesystem development from scratch . .2 1.2.2 Documentation . .3 1.2.3 Testing . .3 1.2.4 Platform independence . .4 1.2.5 Debugging . .5 1.3 Aim .............................................6 1.4 Success criteria . .6 1.5 Use cases . .7 1.5.1 Filesystem development from scratch . .7 1.5.2 Debugging on a new platform . .7 1.5.3 Quality assurance of a filesystem . .8 1.6 Report structure . .8 2 Objectives 9 3 Context Survey 11 3.1 User-space filesystems . 11 3.2 FUSE filesystems . 12 3.3 FUSE for Mac OS X . 13 3.4 General purpose debugging . 13 3.5 Filesystem monitoring . 14 3.6 Testing . 15 3.7 Summary . 15 iv 4 Requirements Specification 16 4.1 Preface . 16 4.2 Glossary . 16 4.3 User requirements . 17 4.3.1 General . 17 4.3.2 Wizard . 18 4.3.3 Test suite . 20 4.3.4 Debugger . 22 4.4 System requirements . 23 4.4.1 General . 23 4.4.2 Wizard . 24 4.4.3 Test suite . 24 4.4.4 Debugger . 25 4.5 Domain requirements . 26 5 Software Engineering Process 27 5.1 Methodology . 27 5.2 Version control . 29 5.3 Testing . 29 5.4 Programming conventions . 30 5.5 Debugging . 30 6 Ethics 31 7 Design 32 7.1 Overall architecture . 32 7.2 Application . 33 7.3 Instrumentation . 36 7.4 Simulated invocation . 37 7.5 Inter-process communication . 39 7.6 Controlling execution . 41 8 Implementation 42 8.1 Application . 42 8.2 GUI ............................................. 43 8.3 FUSE user-space library . 44 8.3.1 Compilation . 44 8.3.2 Dynamic linking . 45 8.4 Inter-process communication . 46 v 8.4.1 Pipes . 46 8.4.2 Environment variables . 46 8.4.3 Semaphores . 47 8.5 Wizard . 48 8.5.1 Test execution . 48 8.5.2 Frontend . 49 8.6 Debugger . 51 8.6.1 Call interception . 51 8.6.2 Frontend . 51 8.7 Test suite . 54 8.7.1 Logging test data . 54 8.7.2 Test execution . 55 8.7.3 Frontend . 56 8.8 Platform compatibility . 58 8.8.1 Unsupported functions . 58 9 Evaluation 59 9.1 Original objectives . 59 9.2 Emergent objectives . 60 9.3 Requirements . 60 9.3.1 User requirements . 60 9.3.2 System requirements . 62 9.3.3 Domain requirements . 63 9.4 Related work . 64 9.4.1 Filesystem debugging . 64 9.4.2 Filesystem monitoring . 64 9.4.3 Filesystem testing . 65 9.5 Summary . 65 10 Conclusion 66 10.1 Achievements . 66 10.1.1 C Programming & FUSE . 66 10.1.2 Extensible test suite . ..
Load more

Recommended publications
  • Cryptomator Documentation Release 1.5.0
    Cryptomator Documentation Release 1.5.0 Cryptobot Sep 15, 2021 Desktop 1 Setup 3 1.1 Windows...............................................3 1.2 macOS................................................3 1.3 Linux.................................................3 2 Getting Started 5 3 Adding Vaults 7 3.1 Create a New Vault..........................................8 3.2 Open an Existing Vault........................................ 13 4 Accessing Vaults 15 4.1 Unlocking a Vault.......................................... 16 4.2 Working with the Unlocked Vault.................................. 17 4.3 Locking a vault............................................ 18 5 Password And Recovery Key 21 5.1 Change Password........................................... 21 5.2 Show Recovery Key......................................... 22 5.3 Reset Password............................................ 23 6 Vault Mounting 27 6.1 General Adapter Selection...................................... 27 6.2 Options applicable to all Systems and Adapters........................... 27 6.3 WebDAV-specific options...................................... 28 6.4 Dokany-specific options....................................... 28 6.5 FUSE-specific options........................................ 28 7 Vault Management 29 7.1 Remove Vaults............................................ 29 7.2 Reorder Vaults............................................ 29 7.3 Vault Options............................................. 29 8 Setup 33 8.1 Google PlayStore..........................................
    [Show full text]
  • File System Design Approaches
    File System Design Approaches Dr. Brijender Kahanwal Department of Computer Science & Engineering Galaxy Global Group of Institutions Dinarpur, Ambala, Haryana, INDIA [email protected] Abstract—In this article, the file system development design The experience with file system development is limited approaches are discussed. The selection of the file system so the research served to identify the different techniques design approach is done according to the needs of the that can be used. The variety of file systems encountered developers what are the needed requirements and show what an active area of research file system specifications for the new design. It allowed us to identify development is. The file systems researched fell into one of where our proposal fitted in with relation to current and past file system development. Our experience with file system the following four categories: development is limited so the research served to identify the 1. The file system is developed in user space and runs as a different techniques that can be used. The variety of file user process. systems encountered show what an active area of research file 2. The file system is developed in the user space using system development is. The file systems may be from one of the FUSE (File system in USEr space) kernel module and two fundamental categories. In one category, the file system is runs as a user process. developed in user space and runs as a user process. Another 3. The file system is developed in the kernel and runs as a file system may be developed in the kernel space and runs as a privileged process.
    [Show full text]
  • High Velocity Kernel File Systems with Bento
    High Velocity Kernel File Systems with Bento Samantha Miller, Kaiyuan Zhang, Mengqi Chen, and Ryan Jennings, University of Washington; Ang Chen, Rice University; Danyang Zhuo, Duke University; Thomas Anderson, University of Washington https://www.usenix.org/conference/fast21/presentation/miller This paper is included in the Proceedings of the 19th USENIX Conference on File and Storage Technologies. February 23–25, 2021 978-1-939133-20-5 Open access to the Proceedings of the 19th USENIX Conference on File and Storage Technologies is sponsored by USENIX. High Velocity Kernel File Systems with Bento Samantha Miller Kaiyuan Zhang Mengqi Chen Ryan Jennings Ang Chen‡ Danyang Zhuo† Thomas Anderson University of Washington †Duke University ‡Rice University Abstract kernel-level debuggers and kernel testing frameworks makes this worse. The restricted and different kernel programming High development velocity is critical for modern systems. environment also limits the number of trained developers. This is especially true for Linux file systems which are seeing Finally, upgrading a kernel module requires either rebooting increased pressure from new storage devices and new demands the machine or restarting the relevant module, either way on storage systems. However, high velocity Linux kernel rendering the machine unavailable during the upgrade. In the development is challenging due to the ease of introducing cloud setting, this forces kernel upgrades to be batched to meet bugs, the difficulty of testing and debugging, and the lack of cloud-level availability goals. support for redeployment without service disruption. Existing Slow development cycles are a particular problem for file approaches to high-velocity development of file systems for systems.
    [Show full text]
  • Alpha Release of the Data Service
    This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 777533. PROviding Computing solutions for ExaScale ChallengeS D5.2 Alpha release of the Data service Start / 01 November 2017 Project: PROCESS H2020 – 777533 Duration: 36 Months Dissemination1: Public Nature2: R Due Date: 31 January 2019 Work Package: WP 5 Filename3 PROCESS_D5.2_Alpha_release_of_the_Data_service_v1.0.docx ABSTRACT During the first 15 months of its implementation, PROCESS has progressed from architecture design based on use cases’ requirements (D4.1) and through architecture validation again based on use cases (D4.2) towards initial implementations of computing services (D6.1) and data services - effort presented here in the deliverable D5.2. In D5.2 we provide an initial demonstrator of the data services, which works in cooperation with the computation services demonstrated in D6.1. The demonstrator is based on the design of the PROCESS data infrastructure described in D5.1. The implementation and initial integration of the infrastructure are based on use case requirements, formulated here as custom application-specific services which are part of the infrastructure. The central, connecting component of the data infrastructure is LOBCDER. It implements a micro-infrastructure of data services, based on dynamically provisioned Docker containers. Additionally to LOBCDER and use case-specific services, the data infrastructure contains generic data and metadata- handling services (DISPEL, DataNet). Finally, Cloudify integrates the micro-infrastructure and the orchestration components of WP7. 1 PU = Public; CO = Confidential, only for members of the Consortium (including the EC services). 2 R = Report; R+O = Report plus Other.
    [Show full text]
  • To FUSE Or Not to FUSE? Analysis and Performance Characterization of the FUSE User-Space File System Framework
    To FUSE or not to FUSE? Analysis and Performance Characterization of the FUSE User-Space File System Framework A Thesis Presented by Bharath Kumar Reddy Vangoor to The Graduate School in Partial Fulfillment of the Requirements for the Degree of Master of Science in Computer Science Stony Brook University Technical Report FSL-16-02 December 2016 Copyright by Bharath Kumar Reddy Vangoor 2016 Stony Brook University The Graduate School Bharath Kumar Reddy Vangoor We, the thesis committee for the above candidate for the Master of Science degree, hereby recommend acceptance of this thesis. Signature: Dr. Erez Zadok, Thesis Advisor Professor, Computer Science Signature: Dr. Mike Ferdman, Thesis Committee Chair Assistant Professor, Computer Science Signature: Dr. Vasily Tarasov IBM Research – Almaden This thesis is accepted by the Graduate School Charles Taber Dean of the Graduate School ii Abstract of the Thesis To FUSE or not to FUSE? Analysis and Performance Characterization of the FUSE User-Space File System Framework by Bharath Kumar Reddy Vangoor Master of Science in Computer Science Stony Brook University December 2016 Traditionally, file systems were implemented as part of operating systems kernels, which provide a limited set of tools and facilities to a programmer. As complexity of file systems grew, many new file systems began being developed in user space. Low performance is considered the main disadvan- tage of user-space file systems but the extent of this problem has never been explored systematically. As a result, the topic of user-space file systems remains rather controversial: while some consider user-space file systems a “toy” not to be used in production, others develop full-fledged production file systems in user space.
    [Show full text]
  • Comparison of Kernel and User Space File Systems
    Comparison of kernel and user space file systems — Bachelor Thesis — Arbeitsbereich Wissenschaftliches Rechnen Fachbereich Informatik Fakultät für Mathematik, Informatik und Naturwissenschaften Universität Hamburg Vorgelegt von: Kira Isabel Duwe E-Mail-Adresse: [email protected] Matrikelnummer: 6225091 Studiengang: Informatik Erstgutachter: Professor Dr. Thomas Ludwig Zweitgutachter: Professor Dr. Norbert Ritter Betreuer: Michael Kuhn Hamburg, den 28. August 2014 Abstract A file system is part of the operating system and defines an interface between OS and the computer’s storage devices. It is used to control how the computer names, stores and basically organises the files and directories. Due to many different requirements, such as efficient usage of the storage, a grand variety of approaches arose. The most important ones are running in the kernel as this has been the only way for a long time. In 1994, developers came up with an idea which would allow mounting a file system in the user space. The FUSE (Filesystem in Userspace) project was started in 2004 and implemented in the Linux kernel by 2005. This provides the opportunity for a user to write an own file system without editing the kernel code and therefore avoid licence problems. Additionally, FUSE offers a stable library interface. It is originally implemented as a loadable kernel module. Due to its design, all operations have to pass through the kernel multiple times. The additional data transfer and the context switches are causing some overhead which will be analysed in this thesis. So, there will be a basic overview about on how exactly a file system operation takes place and which mount options for a FUSE-based system result in a better performance.
    [Show full text]
  • Hands-On Linux Administration on Azure
    Hands-On Linux Administration on Azure Explore the essential Linux administration skills you need to deploy and manage Azure-based workloads Frederik Vos BIRMINGHAM - MUMBAI Hands-On Linux Administration on Azure Copyright © 2018 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information. Commissioning Editor: Vijin Boricha Acquisition Editor: Rahul Nair Content Development Editor: Nithin George Varghese Technical Editor: Komal Karne Copy Editor: Safis Editing Project Coordinator: Drashti Panchal Proofreader: Safis Editing Indexer: Mariammal Chettiyar Graphics: Tom Scaria Production Coordinator: Deepika Naik First published: August 2018 Production reference: 1310818 Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK. ISBN 978-1-78913-096-6 www.packtpub.com mapt.io Mapt is an online digital library that gives you full access to over 5,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career.
    [Show full text]
  • Tahoe-LAFS Documentation Release 1.X
    Tahoe-LAFS Documentation Release 1.x The Tahoe-LAFS Developers January 19, 2017 Contents 1 Welcome to Tahoe-LAFS! 3 1.1 What is Tahoe-LAFS?..........................................3 1.2 What is “provider-independent security”?................................3 1.3 Access Control..............................................4 1.4 Get Started................................................4 1.5 License..................................................4 2 Installing Tahoe-LAFS 5 2.1 First: In Case Of Trouble.........................................5 2.2 Pre-Packaged Versions..........................................5 2.3 Preliminaries...............................................5 2.4 Install the Latest Tahoe-LAFS Release.................................6 2.5 Running the tahoe executable.....................................8 2.6 Running the Self-Tests..........................................8 2.7 Common Problems............................................9 2.8 Using Tahoe-LAFS............................................9 3 How To Run Tahoe-LAFS 11 3.1 Introduction............................................... 11 3.2 Do Stuff With It............................................. 12 3.3 Socialize................................................. 13 3.4 Complain................................................. 13 4 Configuring a Tahoe-LAFS node 15 4.1 Node Types................................................ 16 4.2 Overall Node Configuration....................................... 16 4.3 Connection Management........................................
    [Show full text]
  • Porting FUSE to L4re
    Großer Beleg Porting FUSE to L4Re Florian Pester 23. Mai 2013 Technische Universit¨at Dresden Fakult¨at Informatik Institut fur¨ Systemarchitektur Professur Betriebssysteme Betreuender Hochschullehrer: Prof. Dr. rer. nat. Hermann H¨artig Betreuender Mitarbeiter: Dipl.-Inf. Carsten Weinhold Erkl¨arung Hiermit erkl¨are ich, dass ich diese Arbeit selbstst¨andig erstellt und keine anderen als die angegebenen Hilfsmittel benutzt habe. Declaration I hereby declare that this thesis is a work of my own, and that only cited sources have been used. Dresden, den 23. Mai 2013 Florian Pester Contents 1. Introduction 1 2. State of the Art 3 2.1. FUSE on Linux . .3 2.1.1. FUSE Internal Communication . .4 2.2. The L4Re Virtual File System . .5 2.3. Libfs . .5 2.4. Communication and Access Control in L4Re . .6 2.5. Related Work . .6 2.5.1. FUSE . .7 2.5.2. Pass-to-Userspace Framework Filesystem . .7 3. Design 9 3.1. FUSE Server parts . 11 4. Implementation 13 4.1. Example Request handling . 13 4.2. FUSE Server . 14 4.2.1. LibfsServer . 14 4.2.2. Translator . 14 4.2.3. Requests . 15 4.2.4. RequestProvider . 15 4.2.5. Node Caching . 15 4.3. Changes to the FUSE library . 16 4.4. Libfs . 16 4.5. Block Device Server . 17 4.6. File systems . 17 5. Evaluation 19 6. Conclusion and Further Work 25 A. FUSE operations 27 B. FUSE library changes 35 C. Glossary 37 V List of Figures 2.1. The architecture of FUSE on Linux . .3 2.2. The architecture of libfs .
    [Show full text]
  • Paratrac: a Fine-Grained Profiler for Data-Intensive Workflows
    ParaTrac: A Fine-Grained Profiler for Data-Intensive Workflows Nan Dun Kenjiro Taura Akinori Yonezawa Department of Computer Department of Information and Department of Computer Science Communication Engineering Science The University of Tokyo The University of Tokyo The University of Tokyo 7-3-1 Hongo, Bunkyo-Ku 7-3-1 Hongo, Bunkyo-Ku 7-3-1 Hongo, Bunkyo-Ku Tokyo, 113-5686 Japan Tokyo, 113-5686 Japan Tokyo, 113-5686 Japan [email protected] [email protected] [email protected] tokyo.ac.jp tokyo.ac.jp tokyo.ac.jp ABSTRACT 1. INTRODUCTION The realistic characteristics of data-intensive workflows are With the advance of high performance distributed com- critical to optimal workflow orchestration and profiling is puting, users are able to execute various data-intensive an effective approach to investigate the behaviors of such applications by harnessing massive computing resources [1]. complex applications. ParaTrac is a fine-grained profiler Though workflow management systems have been developed for data-intensive workflows by using user-level file system to alleviate the difficulties of planning, scheduling, and exe- and process tracing techniques. First, ParaTrac enables cuting complex workflows in distributed environments [2{5], users to quickly understand the I/O characteristics of from optimal workflow management still remains a challenge entire application to specific processes or files by examining because of the complexity of applications. Therefore, one low-level I/O profiles. Second, ParaTrac automatically of important and practical demands is to understand and exploits fine-grained data-processes interactions in workflow characterize the data-intensive applications to help workflow to help users intuitively and quantitatively investigate management systems (WMS) refine their orchestration for realistic execution of data-intensive workflows.
    [Show full text]
  • Software Defined Storage Overview
    Software Defined Storage Overview August 2019 Juan Jose Floristan Cloud Specialist Solution Architect 1 AGENDA 1. Why Red Hat Storage? 2. Red Hat Ceph Storage 3. Red Hat Gluster Storage 4. Red Hat Openshift Container Storage 2 Why Red Hat Storage? 3 Why Red Hat Storage? STORAGE IS EVOLVING TRADITIONAL STORAGE OPEN, SOFTWARE-DEFINED STORAGE Complex proprietary silos Standardized, unified, open platforms USER USER USER USER ADMIN Control Plane (API, GUI) ADMIN ADMIN ADMIN Software Ceph Gluster +++ Open Source Custom GUI Custom GUI Custom GUI Proprietary Hardware Proprietary Hardware Proprietary Hardware Standard Computers and Disks Proprietary Proprietary Proprietary Standard Hardware Software Software Software 4 Why Red Hat Storage? WHY THIS MATTERS PROPRIETARY Common, Lower cost, standardized supply chain HARDWARE off-the-shelf hardware SCALE-UP Scale-out Increased operational flexibility ARCHITECTURE architecture HARDWARE-BASED Software-based More programmability, agility, INTELLIGENCE intelligence and control CLOSED DEVELOPMENT Open development More flexible, well-integrated PROCESS process technology 5 Why Red Hat Storage? A RISING TIDE Software-Defined Storage is leading SDS-P MARKET SIZE BY SEGMENT $1,395M a shift in the global storage industry, Block Storage File Storage $1,195M with far-reaching effects. Object Storage $1,029M Hyperconverged “By 2020, between 70%-80% of unstructured $859M data will be held on lower-cost storage managed $705M $592M by SDS.” $475M Innovation Insight: Separating Hype From Hope for Software-Defined Storage “By 2019, 70% of existing storage array products will also be available as software-only versions.” 2013 2014 2015 2016 2017 2018 Innovation Insight: Separating Hype From Hope for Software-Defined Storage 2019 Source: IDC 6 Why Red Hat Storage? THE RED HAT STORAGE MISSION To offer a unified, open software-defined storage portfolio that delivers a range of data services for next generation workloads, thereby accelerating the transition to modern IT infrastructures.
    [Show full text]
  • The Android Platform Security Model∗
    The Android Platform Security Model∗ RENÉ MAYRHOFER, Google and Johannes Kepler University Linz JEFFREY VANDER STOEP, Google CHAD BRUBAKER, Google NICK KRALEVICH, Google Android is the most widely deployed end-user focused operating system. With its growing set of use cases encompassing communication, navigation, media consumption, entertainment, finance, health, and access to sensors, actuators, cameras, or microphones, its underlying security model needs to address a host of practical threats in a wide variety of scenarios while being useful to non-security experts. The model needs to strike a difficult balance between security, privacy, and usability for end users, assurances for app developers, and system performance under tight hardware constraints. While many of the underlying design principles have implicitly informed the overall system architecture, access control mechanisms, and mitigation techniques, the Android security model has previously not been formally published. This paper aims to both document the abstract model and discuss its implications. Based on a definition of the threat model and Android ecosystem context in which it operates, we analyze how the different security measures in past and current Android implementations work together to mitigate these threats. There are some special cases in applying the security model, and we discuss such deliberate deviations from the abstract model. CCS Concepts: • Security and privacy → Software and application security; Domain-specific security and privacy architectures; Operating systems security; • Human-centered computing → Ubiquitous and mobile devices. Additional Key Words and Phrases: Android, security, operating system, informal model 1 INTRODUCTION Android is, at the time of this writing, the most widely deployed end-user operating system.
    [Show full text]