DOCSLIB.ORG

Architectures for the High Availability of Stateful Firewalls

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Architectures for the High Availability of Stateful Firewalls ARCHITECTURES FOR THE HIGH AVAILABILITY OF STATEFUL FIREWALLS PABLO NEIRA AYUSO UNIVERSIDAD DE SEVILLA EUROPEAN DOCTORAL DISSERTATION ADVISED BY DR. RAFAEL MARTINEZ-GASCA´ AND DR. LAURENT LEFEVRE` Contents Acknowledgements 2 Abstract 3 Resumen 4 I Preface 5 1 Introduction 6 1.1 Research Context . .6 1.1.1 Firewalls . .7 1.1.2 Stateful firewalls . .8 1.1.3 Firewalls and their challenges . .9 1.1.4 Communicating user and kernel-space in Linux . 11 1.2 Contributions . 12 1.2.1 Summary of contributions . 12 1.2.2 Publications in reverse chronological order . 12 1.2.3 Books . 14 1.2.4 Research projects . 15 1.2.5 Research visits . 15 1.2.6 Software . 16 1.3 Structure of this dissertation . 16 II Selected research work 18 III Other relevant research work 51 IV Final remarks 97 1.4 Conclusions and future works . 98 1.4.1 Conclusions . 98 1.4.2 Future works . 98 Bibliography 99 1 Acknowledgements Many people have influenced this thesis in very different ways. I have plenty of kind words for all these people, without them this dissertation would not have been possible today. Firstly, I would like to thank to Dr. Laurent Lefevre` for introducing me to the world of the academic research during my study abroad in France. As a result of that work in collaboration, I won the Spanish Ministry of Education’s Arquimedes Award for young researchers in the IT chapter. Then, I was lucky enough to meet Dr. Rafael M. Gasca, head of the QUIVIR research group, who offered me the possibility to continue my Philosophae Doctor Thesis in cooperation with Dr. Laurent Lefevre` in the University of Sevilla. I am grateful to them for providing the framework of freedom and support that I required to finish this work. I am also thankful to the RESO/LIP team and the QUIVIR research group members that shared their research experience and impressions with me along this time. I have also enjoyed the collaboration with Dr. Leonardo Maccari and the LaRT team in the University of Firenze, who contributed with their high insight on wireless mesh networks to our research works. Dr. Alberto Escudero-Pascual also helped me significantly with very useful comments and orientations in the initial stage of the research, specifically on the definition of the Fault-Tolerant FireWall (FT-FW) architecture. Many people from the Free/Open Source community have also contributed to my research with their invaluable feedback. Some of them are Harald Welte, the former head of the Netfilter project; Patrick McHardy, current head of the Netfilter project and one of the maintainers of the Linux kernel networking subsystem; Jozsef Kadlecsik, Netfilter hacker fellow; and Jamal Hadi Salim, one of the relevant Netlink developers; among many others. Finally, I thank my mother, Teresa, and my father, Jose´ Antonio, for all their unconditional love and support; and my brothers Alvaro´ and Gonzalo. I would like to thank many others that are not enumerated here but that have also been crucial in other non-academic aspects. Abstract Nowadays, stateful firewalls are key parts of the critical infrastructure of the Internet. Basically, they help to protect network services and users against attackers by means of access control and protocol conformance checkings. However, stateful firewalls cover network security aspects at the cost of introducing more problems in terms of network performance, availability and com- plexity. Many research has been done with regards to firewalls during the last decades to appro- priately address these concerns. Specifically, these works have focused on improving network performance, through efficient packet classification and specialized hardware, and complexity, by means of model-based filtering policy representations and the detection of rule-set incon- sistencies. However, high availability of stateful firewalls have remained barely studied by the research community according to the existing academic works. This dissertation aims to fill the gap in the field of high availability and stateful firewalls. In several research articles that we have compiled in this thesis, we present the Fault-Tolerant stateful Firewall (FT-FW) architecture to provide high availability, we survey existing fault- tolerant firewall architectures and we provide experimental results that allow network arquitects to select what solution fulfills their requirements. We also provide a software implementation released as free software that the IT industry widely use these days. Moreover, we have ap- plied our research work in the context of wireless mesh networks. In this challenging scenario, we provide a distributed firewalling architecture that helps to improve network-resource man- agement. This architecture is based on Bloom filters and it considers aspects such as efficient filtering policy distribution and mobility. Resumen Hoy en d´ıa, los cortafuegos con estados son parte esencial de la infraestructura cr´ıtica de Inter- net. Basicamente,´ estos dispositivos protegen los servicios de red y a los usuarios de posibles ataques mediantes mecanismos de control de acceso y comprobaciones de conformidad sobre los protocolos. No obstante, los cortafuegos con estados tratan aspectos relacionados con la seguridad a cambio de introducir mas´ problemas en relacion´ con el rendimiento de la red, la disponibilidad y la complejidad. Es por ello que los cortafuegos han sido sujeto de estudio a lo largo de las ultimas´ decadas´ con el fin de abordar tales problemas. Concretamente, dichos trabajos de investigacion´ se han centrado en mejorar el rendimiento de la red, mediante tecnicas´ eficaces para la clasificacion´ de paquetes y el uso de hardware especializado; y la complejidad, mediante representaciones de las pol´ıticas de filtrado basadas en modelos y la deteccion´ de in- consistencias en el conjunto reglas de reglas de cortafuegos. Sin embargo, la alta disponibilidad de los cortafuegos con estados ha sido escasamente estudiada por la comunidad investigadora de acuerdo con los trabajos existentes. Esta tesis pretende completar el vac´ıo existente en el dominio de la alta disponibilidad y los cortafuegos con estados. En varios art´ıculos de investigacion´ que hemos seleccionado y que incluimos en este documento presentamos la arquitectura Fault-Tolerant stateful Firewall (FT-FW) que nos brinda la alta disponibilidad. Ademas,´ hemos realizado un estudio de las arquitecturas de cortafuegos tolerantes a fallos existentes y ofrecemos resultados experimen- tales que permiten a los arquitectos de red seleccionar la solucion´ que mejor se adecua´ a sus requisitos. Tambien´ ofrecemos una implementacion´ disponible bajo licencia de software libre que es usada por la industria TIC en la actualidad. Por otro lado, hemos aplicado nuestra inves- tigacion´ en el contexto de las redes inalambricas´ de malla. En este exigente entorno, ofrecemos una arquitectura distribuida de cortafuegos que ayuda a mejorar la gestion´ de los recursos de la red. Dicha arquitectura esta´ basada en filtros de Bloom y considera aspectos tales como la distribucion´ de la pol´ıtica de filtrado de manera eficiente y la movilidad. Part I Preface 5 Chapter 1 Introduction Failures are a permanent menace for the availability of Internet services. During the last decades, numerous fault-tolerant approaches have been proposed for the wide spectrum of Internet services to provide high availability. However, relatively few research work have ad- dressed the specific aspects of fault-tolerant stateful firewalls and distributed stateful firewalls in wireless mesh networks. Continued availability of firewalls has become a critical factor for the IT industry. Classic fault-tolerant solutions based on redundancy and health-checking mechanisms does not success to fulfill the requirements of stateful firewalls. On the other hand, emerging mobile technolo- gies, such as wireless mesh networks, proliferate in organizations more and more. However, existing network-resource management approaches in this environment remain rudimentary. In this scenario, a distributed firewall architecture that considers mobility and efficiency can help to administer network resources. This dissertation is presented as a compilation of research articles that aim to address these concerns. Basically, we provide several architectures that cover high availability of stateful firewalls and network-resource management through distributed firewalling in wireless mesh networks that supports mobility and efficient filtering policy distribution. 1.1 Research Context The research work that we performed in the frame of this thesis is composed of three parts: The research work in the context of cluster-based fault-tolerant firewalls in wired net- • works to survey existing solutions and to provide the Fault-Tolerant FireWall (FT-FW) architecture. The application of these research advances regarding FT-FW to distributed stateful fire- • walls in wireless mesh/ad-hoc networks. Our implementation released under free software license that uses Netlink sockets to • communicate the in-kernel firewalling subsystem with our user-space high availability state-proxy (as defined in the FT-FW architecture). Specifically, in wireless mesh/ad-hoc networks, we found no works that have focused on providing a distributed firewalling architecture for these environments. Thus, we initially de- 6 fined a stateless packet filtering solution that we extended to become stateful. Then, we applied the FT-FW architecture to enable high avalability and to support mobility. Moreover, with regards to the existing previous research of the high availability of stateful firewalls, most of the results were not established as laws or theories from an epistemological point of view. For that matter, this dissertation provides a wide empirical research as a valid approach to improve epistemological foundations in this topic. To summarize the connection between the different research works that have been compiled in this document, we provide the following concept map in Fig. 1.1. Figure 1.1: Concept map on this dissertation We continue with the basic concepts of this dissertation. In Section. 1.1.1, we provide an introduction to firewalls. Then, Section. 1.1.2 describe stateful firewalls. We overview firewalls and their problems in Section.
Load more

Recommended publications
  • Communicating Between the Kernel and User-Space in Linux Using Netlink Sockets
    SOFTWARE—PRACTICE AND EXPERIENCE Softw. Pract. Exper. 2010; 00:1–7 Prepared using speauth.cls [Version: 2002/09/23 v2.2] Communicating between the kernel and user-space in Linux using Netlink sockets Pablo Neira Ayuso∗,∗1, Rafael M. Gasca1 and Laurent Lefevre2 1 QUIVIR Research Group, Departament of Computer Languages and Systems, University of Seville, Spain. 2 RESO/LIP team, INRIA, University of Lyon, France. SUMMARY When developing Linux kernel features, it is a good practise to expose the necessary details to user-space to enable extensibility. This allows the development of new features and sophisticated configurations from user-space. Commonly, software developers have to face the task of looking for a good way to communicate between kernel and user-space in Linux. This tutorial introduces you to Netlink sockets, a flexible and extensible messaging system that provides communication between kernel and user-space. In this tutorial, we provide fundamental guidelines for practitioners who wish to develop Netlink-based interfaces. key words: kernel interfaces, netlink, linux 1. INTRODUCTION Portable open-source operating systems like Linux [1] provide a good environment to develop applications for the real-world since they can be used in very different platforms: from very small embedded devices, like smartphones and PDAs, to standalone computers and large scale clusters. Moreover, the availability of the source code also allows its study and modification, this renders Linux useful for both the industry and the academia. The core of Linux, like many modern operating systems, follows a monolithic † design for performance reasons. The main bricks that compose the operating system are implemented ∗Correspondence to: Pablo Neira Ayuso, ETS Ingenieria Informatica, Department of Computer Languages and Systems.
    [Show full text]
  • Mellanox Linux Switch †
    SOFTWARE PRODUCT BRIEF Mellanox Linux Switch † – Taking Open Ethernet to the next level, Mellanox Linux Switch provides users with the ability to deploy any standard Linux operating HIGHLIGHTS system on their switch devices – Mellanox Linux Switch enables users to natively install and use any standard Linux distribution as • 100% free open source software the switch operating system on the Open Ethernet Mellanox Spectrum® switch platforms. Mellanox Linux Switch is based on Switchdev, a Linux kernel driver model for Ethernet switches. Mellanox is • L2/L3 switch system as standard Linux the first switch vendor to embrace this model by implementing the switchdev driver for its Mellanox server Spectrum switches. This revolutionary concept breaks the dependency of using vendor-specific • Uniform Linux interfaces software development kits (SDK). Instead of proprietary APIs, fully standard Linux kernel interfaces are used to control the switch chip. This allows users to natively install and use any standard Linux • Fully supported by the Linux distribution as the switch operating system, while the switchdev driver ensures all network traffic is community seamlessly offloaded to the switch hardware. • Hardware independent abstraction layer Openness and Freedom • User choice of Linux operating The 100% free open-source Mellanox switchdev driver is developed and maintained as part of the systems and network applications Linux kernel and is promoted by the Linux community. There is no longer a need for closed code from switch vendors, no more binary blobs, and no need to sign a service level agreement (SLA). • Industry’s highest performance switch systems portfolio, including the high- Linux Switch provides users the flexibility to customize and optimize the switch to their exact needs density half-width Mellanox SN2100 without paying for expensive proprietary software that includes features they neither need nor will never use.
    [Show full text]
  • LPIC-2 Study Guide Second Edition
    LPIC-2 Study Guide Second Edition ffirs.indd 09/14/2016 Page i LPIC-2: Linux Professional Institute Certification Study Guide Exam 201 and Exam 202 Second Edition Christine Bresnahan Richard Blum ffirs.indd 09/14/2016 Page iii Senior Acquisitions Editor: Kenyon Brown Development Editor: Gary Schwartz Technical Editor: Kevin Ryan Production Editor: Christine O’Connor Copy Editor: Linda Rectenwald Editorial Manager: Mary Beth Wakefield Production Manager: Kathleen Wisor Executive Publisher: Jim Minatel Book Designers: Judy Fung and Bill Gibson Proofreader: Rebecca Rider Indexer: John Sleeva Project Coordinator, Cover: Brent Savage Cover Designer: Wiley Cover Image: Getty Images Inc./Jeremy Woodhouse Copyright © 2016 by John Wiley & Sons, Inc., Indianapolis, Indiana Published simultaneously in Canada ISBN: 978-1-119-15079-4 ISBN: 978-1-119-15081-7 (ebk.) ISBN: 978-1-119-15080-0 (ebk.) Manufactured in the United States of America No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as permitted under Sections 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 646-8600. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hobo- ken, NJ 07030, (201) 748-6011, fax (201) 748-6008, or online at http://www.wiley.com/go/permissions.
    [Show full text]
  • Vorlage Für Dokumente Bei AI
    OSS Disclosure Document Date: 16-October-2017 BEG-VS OSS Licenses used in Alpine AS1 Project Page 1 Open Source Software Attributions for Alpine AS1 0046_170818 AI_PRJ_ALPINE_LINUX_17.0F17 This document is provided as part of the fulfillment of OSS license conditions and does not require users to take any action before or while using the product. © Bosch Engineering GmbH. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights. OSS Disclosure Document Date: 16-October-2017 BEG-VS OSS Licenses used in Alpine AS1 Project Page 2 Table of content 1 Overview ............................................................................................................................................................. 12 2 OSS Licenses used in the project ......................................................................................................................... 12 3 Package details for OSS Licenses usage .............................................................................................................. 14 3.1 7 Zip - LZMA SDK ...................................................................................................................................... 14 3.2 ACL 2.2.51 ................................................................................................................................................... 14 3.3 Alsa Libraries 1.0.27.2 ................................................................................................................................
    [Show full text]
  • License Terms & Condition
    Date:7-June-2016 OSS Disclosure Document CM_CI1/EGM-P OSS Licenses used in GM-MY17 Project Page 1 1 Overview .................................................. 10 2 OSS Licenses used in the project .......................... 10 3 Package details for OSS Licenses usage .................... 11 3.1 7 Zip - LZMA SDK - 9.2/4.01 .................................. 11 3.2 ACL - 2.2.51 ................................................. 11 3.3 AES - Advanced Encryption Standard – 1.0 ..................... 11 3.4 Alsa Libraries - 1.0.27.2 .................................... 11 3.5 Alsa Plugins - 1.0.26 ........................................ 11 3.6 Alsa Utils - 1.0.27.2 ........................................ 12 3.7 APMD - 3.2.2 ................................................. 12 3.8 ATK - 2.8.0 .................................................. 12 3.9 Attr - 2.4.46 ................................................ 12 3.10 Audio File Library - 0.2.7 ................................. 12 3.11 Android - platform - bootable – recovery -4.2 .............. 12 3.12 Android - platform - system – core -4.2 .................... 12 3.13 Avahi - 0.6.31 ............................................. 13 3.14 Bash - 3.2.48 .............................................. 13 3.15 Bison with GPL 3.0 exception - 2.7.1 ....................... 13 3.16 Blktrace - 1.0.5 ........................................... 13 3.17 BlueZ - 4.101/5.15 ......................................... 13 3.18 Boost C++ Libraries- boost 1.50.0 .......................... 13 3.19 BPSTL .....................................................
    [Show full text]
  • Vorlage Für Dokumente Bei AI
    OSS Disclosure Document Date: 27-Apr-2018 CM-AI/PJ-CC OSS Licenses used in Suzuki Project Page 1 Table of content 1 Overview ..................................................................................................................... 11 2 OSS Licenses used in the project ................................................................................... 11 3 Package details for OSS Licenses usage ........................................................................ 12 3.1 7 Zip - LZMA SDK ............................................................................................................. 12 3.2 ACL 2.2.51 ...................................................................................................................... 12 3.3 Alsa Libraries 1.0.27.2 ................................................................................................... 12 3.4 AES - Advanced Encryption Standard 1.0 ......................................................................... 12 3.5 Alsa Plugins 1.0.26 ........................................................................................................ 13 3.6 Alsa Utils 1.0.27.2 ......................................................................................................... 13 3.7 APMD 3.2.2 ................................................................................................................... 13 3.8 Atomic_ops .................................................................................................................... 13 3.9 Attr 2.4.46 ...................................................................................................................
    [Show full text]
  • Software Fault Isolation with Api Integrity and Multi-Principal Modules
    Software Fault Isolation with Api Integrity and Multi-Principal Modules The MIT Faculty has made this article openly available. Please share how this access benefits you. Your story matters. Citation Yandong Mao, Haogang Chen, Dong Zhou, Xi Wang, Nickolai Zeldovich, and M. Frans Kaashoek. 2011. Software fault isolation with API integrity and multi-principal modules. In Proceedings of the Twenty-Third ACM Symposium on Operating Systems Principles (SOSP '11). ACM, New York, NY, USA, 115-128. As Published http://dx.doi.org/10.1145/2043556.2043568 Publisher Association for Computing Machinery (ACM) Version Author's final manuscript Citable link http://hdl.handle.net/1721.1/72580 Terms of Use Creative Commons Attribution-Noncommercial-Share Alike 3.0 Detailed Terms http://creativecommons.org/licenses/by-nc-sa/3.0/ Software fault isolation with API integrity and multi-principal modules Yandong Mao, Haogang Chen, Dong Zhou†, Xi Wang, Nickolai Zeldovich, and M. Frans Kaashoek MIT CSAIL, †Tsinghua University IIIS ABSTRACT complex, irregular kernel interfaces such as the ones found in the The security of many applications relies on the kernel being secure, Linux kernel and exploited by attackers. but history suggests that kernel vulnerabilities are routinely discov- Previous systems such as XFI [9] have used software isola- ered and exploited. In particular, exploitable vulnerabilities in kernel tion [26] to isolate kernel modules from the core kernel, thereby modules are common. This paper proposes LXFI, a system which protecting against a class of attacks on kernel modules. The chal- isolates kernel modules from the core kernel so that vulnerabilities lenge is that modules need to use support functions in the core in kernel modules cannot lead to a privilege escalation attack.
    [Show full text]
  • Istnanosat-1 Heart Processing and Digital Communications Unit
    ISTNanosat-1 Heart Processing and digital communications unit Joao˜ Andre´ Henriques Ferreira Dissertation submitted to obtain the Master Degree in Communication Networks Engineering Jury Chairman: Prof. Doutor Paulo Jorge Pires Ferreira Supervisor: Prof. Doutor Rui Manuel Rodrigues Rocha Co-Supervisor: Prof. Doutor Moises´ Simoes˜ Piedade Members: Prof. Doutor Mario´ Serafim dos Santos Nunes October 2012 Acknowledgments First of all, I would like to thank Professor Doutor Rui Manuel Rocha and Professor Doutor Moises´ Simoes˜ Piedade for giving me the opportunity to join the ISTNanosat project. The constant supervision from Professor Rui Rocha with his scientific background on embedded systems design and networking architectures allied with Professor Moises´ Piedade knowledge on electronics constraints and expected behaviours, were very useful during all project phases. I would also like to thank all the AMSAT-CT engineers for bringing me a lot of technical information on space related technology through very interesting discussions. I also want to highlight my colleagues on GEMS group, specially the Ph.D candidate Jose´ Catela for his useful knowledge and availability helping me on one of the implementation board programming and its details, sometimes until late evening. To all my course colleagues for the healthy acquaintanceship even during the long hard work nights, specially to Bruno Henriques for his valuable friendship since my first hour at IST. I’d like to thank my girlfriend Susana Rodrigues for her support and patience specially during the stress periods and to my grandparents for all the attention and affection. Finally, I want to express my deep thankfulness and gratitude to my parents for the unconditional support since the beginning.
    [Show full text]
  • Flood Prediction Model Simulation with Heterogeneous Trade-Offs in High Performance Computing Framework
    FLOOD PREDICTION MODEL SIMULATION WITH HETEROGENEOUS TRADE-OFFS IN HIGH PERFORMANCE COMPUTING FRAMEWORK Antoni Portero Simone Libutti Radim Vavrik Giuseppe Massari Stepan Kuchar William Fornaciari Martin Golasowski Politecnico di Milano Vit Vondrak Dipartimento di Elettronica, IT4Innovations Informazione e Bioengegneria (DEIB) VSB-Technical University of Ostrava Milano, Italy Ostrava, Czech Republic Email: [email protected] Email: [email protected] KEYWORDS that need on-demand computation are emerging. Looking into the future, we might imagine event-driven and data-driven Hydrology model simulation; High Performance Comput- HPC applications running on demand to predict any kind of ing systems; disaster management; reliability models; com- event, e.g. running air pollution dispersion models at real time puting resource management; multi and many-cores systems; to predict hazardous substance dispersion caused by some parallel systems accident. Of course, as we build confidence in these emerging computations, they will move from the scientists workbench ABSTRACT into critical decision-making paths, but HPC clusters will still In this paper, we propose a safety-critical system with be needed to execute and analyse these simulations in a short a run-time resource management that is used to operate an time frame and they will have to provide the results in a application for flood monitoring and prediction. This applica- reliable way based on the specified operational requirements. tion can run with different Quality of Service (QoS) levels The intent of this paper is to describe the characteristics depending on the current hydrometeorological situation. and constraints of disaster management (DM) applications for industrial environments. The system operation can follow two main scenarios - stan- dard or emergency operation.
    [Show full text]
  • Chapter 1 Starting with Linu Shells What Is Linux?
    - % >/J . >/ , I/# >/ , E# >/ . ! ,# >/ !"" C # . >/ !" ." , ; ! , # >/ / !"" . >/ ! > . C , >/ . " !. " .. . C" " ! >4 ?0 ,# >/ !,# !. % # # . >/ . . " ! !, /! ; ! , " . " . >/ " . ,# !"" ! = . " ", >/ "' ; >/ , ; ?0 1 #! , " 1! . I! . . !.! D >/ " I! . . C . ! #" . . # ! >/ " + 55 ; >/ " ; ! ! . " # . , # ! !" >/ " & ? ; ! . >/ " ; , ! . . !" " !# ! /!# . 7 . C .# >/ C " > ; > . !# . >/ , . 0 . :, : ! . 0/ " " # " " 1. # >/ , > !"" ! ## . "# ; " ! . !" # " & > !# ## ." . #"" ." 1# !# #""# ! , !" ! ; ". # > ! ! . "" ## " >C ! ! ## ! , ; " !! ! >/ , ! /! . D > !# , ! " . , , ; , " . " .!' & " "" "#" &. #" "#" : "#" = " "#" ; .# ! / ! . .! " # ) #) F . " .! . # " , "" "#" , "# ! "" ! ! "# "" "" ! / # ! , ! ##! ; , ! . "" ! !, . ." ! ! ! "" ; " , " "" ! / + - + 56 ; >/ " "" " ; "" ! # !, ! # ; , ! ! # . "" ! "" ! ; , " . "" # ! ! # ! "" ! # , ; , , !, . ! "" # "! ! "" # !! . . " ! +! ##- . C "" % #" !! "" # , " ", " . ! "" # .. "" # 7 # ." ! F ! , " ! # ! ; ! . # "" # . # ! ! . # >/ " # K ! ! . "" >/ " # ! . :C /" . " ' !"# $ %% & #' ()& *+' , +#% *+- ."# , +- ." *+- . , +- .# / +0 12 "% 12 !# 3)'
    [Show full text]
  • Resilient and Fast Persistent Container Storage Leveraging Linux’S Storage Functionalities Philipp Reisner, CEO LINBIT LINBIT - the Company Behind It
    Resilient and Fast Persistent Container Storage Leveraging Linux’s Storage Functionalities Philipp Reisner, CEO LINBIT LINBIT - the company behind it COMPANY OVERVIEW TECHNOLOGY OVERVIEW • Developer of DRBD • 100% founder owned • Offices in Europe and US • Team of 30 highly experienced Linux experts • Partner in Japan REFERENCES 31 Linux Storage Gems LVM, RAID, SSD cache tiers, deduplication, targets & initiators Linux's LVM logical volume snapshot logical volume Volume Group physical volume physical volume physical volume 31 Linux's LVM • based on device mapper • original objects • PVs, VGs, LVs, snapshots • LVs can scatter over PVs in multiple segments • thinlv • thinpools = LVs • thin LVs live in thinpools • multiple snapshots became efficient! 31 Linux's LVM thin-LV thin-LV thin-sLV LV snapshot thinpool VG PV PV PV 31 Linux's RAID RAID1 • original MD code • mdadm command A1 A1 • Raid Levels: 0,1,4,5,6,10 A2 A2 • Now available in LVM as well A3 A3 A4 A4 • device mapper interface for MD code • do not call it ‘dmraid’; that is software for hardware fake-raid • lvcreate --type raid6 --size 100G VG_name 31 SSD cache for HDD • dm-cache • device mapper module • accessible via LVM tools • bcache • generic Linux block device • slightly ahead in the performance game 31 Linux’s DeDupe • Virtual Data Optimizer (VDO) since RHEL 7.5 • Red hat acquired Permabit and is GPLing VDO • Linux upstreaming is in preparation • in-line data deduplication • kernel part is a device mapper module • indexing service runs in user-space • async or synchronous writeback • Recommended to be used below LVM 31 Linux’s targets & initiators • Open-ISCSI initiator IO-requests • Ietd, STGT, SCST Initiator Target data/completion • mostly historical • LIO • iSCSI, iSER, SRP, FC, FCoE • SCSI pass through, block IO, file IO, user-specific-IO • NVMe-OF • target & initiator 31 ZFS on Linux • Ubuntu eco-system only • has its own • logic volume manager (zVols) • thin provisioning • RAID (RAIDz) • caching for SSDs (ZIL, SLOG) • and a file system! 31 Put in simplest form DRBD – think of it as ..
    [Show full text]
  • The Sysfs Filesystem
    The sysfs Filesystem Patrick Mochel [email protected] Abstract sysfs is a core piece of kernel infrastructure, which means that it provides a relatively sim- ple interface to perform a simple task. Rarely sysfs is a feature of the Linux 2.6 kernel that al- is the code overly complicated, or the descrip- lows kernel code to export information to user tions obtuse. However, like many core pieces processes via an in-memory filesystem. The or- of infrastructure, it can get a bit too abstract ganization of the filesystem directory hierarchy and far removed to keep track of. To help al- is strict, and based the internal organization of leviate that, this paper takes a gradual approach kernel data structures. The files that are created to sysfs before getting to the nitty-gritty details. in the filesystem are (mostly) ASCII files with (usually) one value per file. These features en- First, a short but touching history describes its sure that the information exported is accurate origins. Then crucial information about mount- and easily accessible, making sysfs one of the ing and accessing sysfs is included. Next, most intuitive and useful features of the 2.6 ker- the directory organization and layout of sub- nel. systems in sysfs is described. This provides enough information for a user to understand the organization and content of the information that Introduction is exported through sysfs, though for reasons of time and space constraints, not every object and its attributes are described. sysfs is a mechanism for representing kernel objects, their attributes, and their relationships The primary goal of this paper is to pro- with each other.
    [Show full text]