Decentralized Authorization with Private Delegation by Michael P Andersen a Dissertation Submitted in Partial Satisfaction of Th

Total Page:16

File Type:pdf, Size:1020Kb

Decentralized Authorization with Private Delegation by Michael P Andersen a Dissertation Submitted in Partial Satisfaction of Th Decentralized Authorization with Private Delegation by Michael P Andersen A dissertation submitted in partial satisfaction of the requirements for the degree of Doctor of Philosophy in Computer Science in the Graduate Division of the University of California, Berkeley Committee in charge: Professor David E. Culler, Chair Associate Professor Deirdre Mulligan Assistant Professor Raluca Ada Popa Summer 2019 Decentralized Authorization with Private Delegation Copyright 2019 by Michael P Andersen 1 Abstract Decentralized Authorization with Private Delegation by Michael P Andersen Doctor of Philosophy in Computer Science University of California, Berkeley Professor David E. Culler, Chair Authentication and authorization systems can be found in almost every software system, and consequently affects every aspect of our lives. Despite the variety in the software that relies on authorization, the authorization subsystem itself is almost universally architected following a com- mon pattern with unfortunate characteristics. The first of these is that there usually exists a set of centralized servers that hosts the set of users and their permissions. This results in a number of security threats, such as permitting the operator of the authorization system to view or even change the permission data for all users. Secondly, these systems do not permit federation across administrative domains, as there is no safe choice of system operator: any operator would have visibility and control in all administrative domains, which is unacceptable. Thirdly, these systems do not offer transitive delegation: when a user grants permission to another user, the permissions of the recipient are not predicated upon the permissions of the granter. This makes it very difficult to reason about permissions as the complexity of the system grows, especially in the federation across domains case where no party can have absolute visibility into all permissions. Whilst several other systems, such as financial systems (e.g. blockchains) and communication systems (e.g. Signal / WhatsApp) have recently been reinvented to incorporate decentralization and privacy, there has been little attention paid to improving the authorization systems. This work aims to address that by asking the question “How can we construct an authorization system that supports first-class transitive delegation across administrative domains without trusting a central authority or compromising on privacy?” We survey several models for authorization and find that Graph Based Authorization, where principals are vertices in a graph and delegation between principals are edges in the graph, is capa- ble of capturing transitive delegation as a first class primitive, whilst also retaining compatibility with existing techniques such as Discretionary Access Control or Role Based Access Control. A proof of permission in the Graph Based Authorization model is represented by a path through the graph formed from the concatenation of individual edges. Whilst prior implementations of Graph Based Authorization do not meet the decentralization or privacy-preserving goals, we find that this is not intrinsic, and can be remedied by introducing two new techniques. The first is the con- 2 struction of a global storage tier that cryptographically proves its integrity, and the second is an encryption technique that preserves the privacy of attestations in global storage. The horizontally-scalable storage tier is based on a new data structure, the Unequivocable Log Derived Map, which is composed of three Merkle trees. Consistency proofs over these trees allow a server to prove that objects exist or do not exist within storage, as well as proving that the storage is append-only (no previously inserted objects have been removed). Our scheme advances prior work in this field by permitting efficient auditing that scales with the number of additions to the storage rather than scaling with the total number of stored objects. By utilizing cryptographic proofs of integrity, we force storage servers to either behave honestly, or become detected as compromised. Thus, even though the architecture is centralized for availability and performance, it is does not introduce any central authorities. The design of the storage does not ensure the privacy of the permission data stored within it. We address this through the introduction of Reverse Discoverable Encryption. This technique uses the objects representing grants of permission as a key dissemination channel, thus operating without communication between participants. By using Wildcard Key Derivation Identity Based Encryption in a non-standard way (with no central Private Key Generator) we allow for permission objects to be encrypted using the authorization policy as a key. Thus, RDE permits the recipient of some permissions to decrypt other compatible permissions granted to the grantee that could be concatenated together to form a valid proof. RDE therefore protects the privacy of permission objects in storage whilst still permitting decryption of those objects by authorized parties. We construct an implementation of these techniques, named WAVE, and evaluate its perfor- mance. We find that WAVE has similar performance to the widely used OAuth system and per- forms better than the equally widely used LDAP system, despite offering significantly better se- curity properties. We present an advancement to Graph Based Authorization which efficiently represents complex authorization proofs as a compact subgraph rather than a sequence of linear paths, and present a technique for efficient discovery of such proofs. To validate our techniques and ensure their efficacy in practice, we pose an additional question: “How can we leverage WAVE to improve the security of IoT communications?” We present a microservice architecture that abstracts the interfaces of IoT devices to permit a uniform security policy to be applied to heterogeneous devices of similar function. This is achieved by enforcing security policy at the communication bus and using hardware abstraction microservices to adapt the interfaces that devices expose on this communication bus. We construct and evaluate an instance of this communication bus, WAVEMQ and find that, with appropriate caching, its performance is comparable to that of prior publish/subscribe information busses. We discover that by enforcing WAVE’s security model in the core of the network, we gain a resistance to denial of service attacks. This is particularly valuable in the IoT context where devices are typically resource constrained or connected by a bandwidth-limited link. i Dedication To my soon-to-be wife, Soo Hyun Kim, who supported me throughout my PhD, and to my family, friends, and advisors, who made this possible. ii Contents Contents ii List of Figures v List of Tables vii 1 Introduction 1 1.1 Authentication and Authorization . 1 1.2 Centralization . 2 1.3 Federation Across Administrative Domains . 3 1.4 Transitive Delegation . 5 1.5 Motivation and Thesis Question . 6 1.6 Solution Overview . 7 1.7 Generalization . 10 1.8 Roadmap . 11 2 Background 13 2.1 Requirements for a Decentralized Authorization System . 13 2.2 Trust Management . 15 2.3 SDSI & SPKI . 17 3 Decentralized Graph Based Authorization 19 3.1 Concepts . 19 3.2 Entities . 20 3.3 Attestations . 21 3.4 Underlying System Requirements . 22 3.5 Revocation . 23 3.6 Proofs . 23 3.7 Common Authorization Modes . 24 3.8 Graph Based Authorization Evaluation . 28 3.9 Name Resolution . 31 3.10 Summary . 32 iii 4 Secure Storage 33 4.1 Operational and Security Requirements . 33 4.2 Abstract Implementation of Operational Requirements . 36 4.3 Blockchain Storage . 38 4.4 Blockchain Scalability Concerns . 51 4.5 Unequivocable Log Derived Map . 55 4.6 ULDM Storage Evaluation . 59 4.7 Storage Through DNS . 60 4.8 Storage Conclusions . 60 5 Private Delegation 63 5.1 Private Delegation Requirements . 64 5.2 Structural Reverse Discoverable Encryption . 64 5.3 Policy-Aware Reverse Discoverable Encryption . 65 5.4 Efficient Discoverability . 69 5.5 Reducing Leakage in Proofs . 70 5.6 Discovering an Attestation . 71 5.7 Extensions . 72 5.8 Privacy Micro Benchmarks . 73 5.9 Generalization to Other Policy Types . 74 5.10 Anonymous Proof Of Authorization . 75 5.11 Multicast End To End Encryption . 77 5.12 Protecting Name Declarations . 79 5.13 Summary . 79 6 System Design And Implementation 81 6.1 WAVE 3 System Overview . 81 6.2 System Integration Pattern . 82 6.3 Internal API Provider . 83 6.4 External API . 86 6.5 State Engine . 86 6.6 Indexed Local Storage . 87 6.7 Global Storage . 88 6.8 Serialization and Representation . 89 6.9 Proof Building Optimization . 91 6.10 Profiled Evaluation . 98 6.11 Summary . 101 7 Microservices in the Built Environment 103 7.1 Building Operating Systems Background . 103 7.2 An eXtensible Building Operating System . 106 7.3 A Secure Building Operating System . 108 iv 7.4 Summary . 111 8 Secure Syndication for IoT 112 8.1 Resource Design For Syndication Security . 113 8.2 Denial Of Service Resistance . 114 8.3 Intermittent Connectivity . 115 8.4 WAVEMQ System Overview . 116 8.5 WAVE in WAVEMQ . 118 8.6 Caching in WAVEMQ . 118 8.7 WAVEMQ Performance Evaluation . 119 8.8 Hiding Proofs From The Router . 120 8.9 Summary . 122 9 Concerns and Implications 123 9.1 Auditability . 123 9.2 Cryptographic Handoff . 124 9.3 Liberty . 127 9.4 Summary . 128 10 Related Work 129 10.1 Graph Based Authorization . ..
Recommended publications
  • NUMA-Aware Thread Migration for High Performance NVMM File Systems
    NUMA-Aware Thread Migration for High Performance NVMM File Systems Ying Wang, Dejun Jiang and Jin Xiong SKL Computer Architecture, ICT, CAS; University of Chinese Academy of Sciences fwangying01, jiangdejun, [email protected] Abstract—Emerging Non-Volatile Main Memories (NVMMs) out considering the NVMM usage on NUMA nodes. Besides, provide persistent storage and can be directly attached to the application threads accessing file system rely on the default memory bus, which allows building file systems on non-volatile operating system thread scheduler, which migrates thread only main memory (NVMM file systems). Since file systems are built on memory, NUMA architecture has a large impact on their considering CPU utilization. These bring remote memory performance due to the presence of remote memory access and access and resource contentions to application threads when imbalanced resource usage. Existing works migrate thread and reading and writing files, and thus reduce the performance thread data on DRAM to solve these problems. Unlike DRAM, of NVMM file systems. We observe that when performing NVMM introduces extra latency and lifetime limitations. This file reads/writes from 4 KB to 256 KB on a NVMM file results in expensive data migration for NVMM file systems on NUMA architecture. In this paper, we argue that NUMA- system (NOVA [47] on NVMM), the average latency of aware thread migration without migrating data is desirable accessing remote node increases by 65.5 % compared to for NVMM file systems. We propose NThread, a NUMA-aware accessing local node. The average bandwidth is reduced by thread migration module for NVMM file system.
    [Show full text]
  • Unravel Data Systems Version 4.5
    UNRAVEL DATA SYSTEMS VERSION 4.5 Component name Component version name License names jQuery 1.8.2 MIT License Apache Tomcat 5.5.23 Apache License 2.0 Tachyon Project POM 0.8.2 Apache License 2.0 Apache Directory LDAP API Model 1.0.0-M20 Apache License 2.0 apache/incubator-heron 0.16.5.1 Apache License 2.0 Maven Plugin API 3.0.4 Apache License 2.0 ApacheDS Authentication Interceptor 2.0.0-M15 Apache License 2.0 Apache Directory LDAP API Extras ACI 1.0.0-M20 Apache License 2.0 Apache HttpComponents Core 4.3.3 Apache License 2.0 Spark Project Tags 2.0.0-preview Apache License 2.0 Curator Testing 3.3.0 Apache License 2.0 Apache HttpComponents Core 4.4.5 Apache License 2.0 Apache Commons Daemon 1.0.15 Apache License 2.0 classworlds 2.4 Apache License 2.0 abego TreeLayout Core 1.0.1 BSD 3-clause "New" or "Revised" License jackson-core 2.8.6 Apache License 2.0 Lucene Join 6.6.1 Apache License 2.0 Apache Commons CLI 1.3-cloudera-pre-r1439998 Apache License 2.0 hive-apache 0.5 Apache License 2.0 scala-parser-combinators 1.0.4 BSD 3-clause "New" or "Revised" License com.springsource.javax.xml.bind 2.1.7 Common Development and Distribution License 1.0 SnakeYAML 1.15 Apache License 2.0 JUnit 4.12 Common Public License 1.0 ApacheDS Protocol Kerberos 2.0.0-M12 Apache License 2.0 Apache Groovy 2.4.6 Apache License 2.0 JGraphT - Core 1.2.0 (GNU Lesser General Public License v2.1 or later AND Eclipse Public License 1.0) chill-java 0.5.0 Apache License 2.0 Apache Commons Logging 1.2 Apache License 2.0 OpenCensus 0.12.3 Apache License 2.0 ApacheDS Protocol
    [Show full text]
  • Artificial Intelligence for Understanding Large and Complex
    Artificial Intelligence for Understanding Large and Complex Datacenters by Pengfei Zheng Department of Computer Science Duke University Date: Approved: Benjamin C. Lee, Advisor Bruce M. Maggs Jeffrey S. Chase Jun Yang Dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in the Department of Computer Science in the Graduate School of Duke University 2020 Abstract Artificial Intelligence for Understanding Large and Complex Datacenters by Pengfei Zheng Department of Computer Science Duke University Date: Approved: Benjamin C. Lee, Advisor Bruce M. Maggs Jeffrey S. Chase Jun Yang An abstract of a dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in the Department of Computer Science in the Graduate School of Duke University 2020 Copyright © 2020 by Pengfei Zheng All rights reserved except the rights granted by the Creative Commons Attribution-Noncommercial Licence Abstract As the democratization of global-scale web applications and cloud computing, under- standing the performance of a live production datacenter becomes a prerequisite for making strategic decisions related to datacenter design and optimization. Advances in monitoring, tracing, and profiling large, complex systems provide rich datasets and establish a rigorous foundation for performance understanding and reasoning. But the sheer volume and complexity of collected data challenges existing techniques, which rely heavily on human intervention, expert knowledge, and simple statistics. In this dissertation, we address this challenge using artificial intelligence and make the case for two important problems, datacenter performance diagnosis and datacenter workload characterization. The first thrust of this dissertation is the use of statistical causal inference and Bayesian probabilistic model for datacenter straggler diagnosis.
    [Show full text]
  • Characterizing, Modeling, and Benchmarking Rocksdb Key-Value
    Characterizing, Modeling, and Benchmarking RocksDB Key-Value Workloads at Facebook Zhichao Cao, University of Minnesota, Twin Cities, and Facebook; Siying Dong and Sagar Vemuri, Facebook; David H.C. Du, University of Minnesota, Twin Cities https://www.usenix.org/conference/fast20/presentation/cao-zhichao This paper is included in the Proceedings of the 18th USENIX Conference on File and Storage Technologies (FAST ’20) February 25–27, 2020 • Santa Clara, CA, USA 978-1-939133-12-0 Open access to the Proceedings of the 18th USENIX Conference on File and Storage Technologies (FAST ’20) is sponsored by Characterizing, Modeling, and Benchmarking RocksDB Key-Value Workloads at Facebook Zhichao Cao†‡ Siying Dong‡ Sagar Vemuri‡ David H.C. Du† †University of Minnesota, Twin Cities ‡Facebook Abstract stores is still challenging. First, there are very limited studies of real-world workload characterization and analysis for KV- Persistent key-value stores are widely used as building stores, and the performance of KV-stores is highly related blocks in today’s IT infrastructure for managing and storing to the workloads generated by applications. Second, the an- large amounts of data. However, studies of characterizing alytic methods for characterizing KV-store workloads are real-world workloads for key-value stores are limited due to different from the existing workload characterization stud- the lack of tracing/analyzing tools and the difficulty of collect- ies for block storage or file systems. KV-stores have simple ing traces in operational environments. In this paper, we first but very different interfaces and behaviors. A set of good present a detailed characterization of workloads from three workload collection, analysis, and characterization tools can typical RocksDB production use cases at Facebook: UDB (a benefit both developers and users of KV-stores by optimizing MySQL storage layer for social graph data), ZippyDB (a dis- performance and developing new functions.
    [Show full text]
  • Myrocks in Mariadb
    MyRocks in MariaDB Sergei Petrunia <[email protected]> MariaDB Shenzhen Meetup November 2017 2 What is MyRocks ● #include <Yoshinori’s talk> ● This talk is about MyRocks in MariaDB 3 MyRocks lives in Facebook’s MySQL branch ● github.com/facebook/mysql-5.6 – Will call this “FB/MySQL” ● MyRocks lives there in storage/rocksdb ● FB/MySQL is easy to use if you are Facebook ● Not so easy if you are not :-) 4 FB/mysql-5.6 – user perspective ● No binaries, no packages – Compile yourself from source ● Dependencies, etc. ● No releases – (Is the latest git revision ok?) ● Has extra features – e.g. extra counters “confuse” monitoring tools. 5 FB/mysql-5.6 – dev perspective ● Targets a CentOS-type OS – Compiler, cmake version, etc. – Others may or may not [periodically] work ● MariaDB/Percona file pull requests to fix ● Special command to compile – https://github.com/facebook/mysql-5.6/wiki/Build-Steps ● Special command to run tests – Test suite assumes a big machine ● Some tests even a release build 6 Putting MyRocks in MariaDB ● Goals – Wider adoption – Ease of use – Ease of development – Have MyRocks in MariaDB ● Use it with MariaDB features ● Means – Port MyRocks into MariaDB – Provide binaries and packages 7 Status of MyRocks in MariaDB 8 Status of MyRocks in MariaDB ● MariaDB 10.2 is GA (as of May, 2017) ● It includes an ALPHA version of MyRocks plugin – Working to improve maturity ● It’s a loadable plugin (ha_rocksdb.so) ● Packages – Bintar, deb, rpm, win64 zip + MSI – deb/rpm have MyRocks .so and tools in a separate package. 9 Packaging for MyRocks in MariaDB 10 MyRocks and RocksDB library ● MyRocks is tied RocksDB@revno MariaDB – RocksDB is a github submodule – No compatibility with other versions MyRocks ● RocksDB is always compiled with RocksDB MyRocks S Z n ● l i And linked-in statically a b p ● p Distros have a RocksDB package y – Not using it.
    [Show full text]
  • Dmon: Efficient Detection and Correction of Data Locality
    DMon: Efficient Detection and Correction of Data Locality Problems Using Selective Profiling Tanvir Ahmed Khan and Ian Neal, University of Michigan; Gilles Pokam, Intel Corporation; Barzan Mozafari and Baris Kasikci, University of Michigan https://www.usenix.org/conference/osdi21/presentation/khan This paper is included in the Proceedings of the 15th USENIX Symposium on Operating Systems Design and Implementation. July 14–16, 2021 978-1-939133-22-9 Open access to the Proceedings of the 15th USENIX Symposium on Operating Systems Design and Implementation is sponsored by USENIX. DMon: Efficient Detection and Correction of Data Locality Problems Using Selective Profiling Tanvir Ahmed Khan Ian Neal Gilles Pokam Barzan Mozafari University of Michigan University of Michigan Intel Corporation University of Michigan Baris Kasikci University of Michigan Abstract cally at run time. In fact, as we (§6.2) and others [2,15,20,27] Poor data locality hurts an application’s performance. While demonstrate, compiler-based techniques can sometimes even compiler-based techniques have been proposed to improve hurt performance when the assumptions made by those heuris- data locality, they depend on heuristics, which can sometimes tics do not hold in practice. hurt performance. Therefore, developers typically find data To overcome the limitations of static optimizations, the locality issues via dynamic profiling and repair them manually. systems community has invested substantial effort in devel- Alas, existing profiling techniques incur high overhead when oping dynamic profiling tools [28,38, 57,97, 102]. Dynamic used to identify data locality problems and cannot be deployed profilers are capable of gathering detailed and more accurate in production, where programs may exhibit previously-unseen execution information, which a developer can use to identify performance problems.
    [Show full text]
  • Real-Time LSM-Trees for HTAP Workloads
    Real-Time LSM-Trees for HTAP Workloads Hemant Saxena Lukasz Golab University of Waterloo University of Waterloo [email protected] [email protected] Stratos Idreos Ihab F. Ilyas Harvard University University of Waterloo [email protected] [email protected] ABSTRACT We observe that a Log-Structured Merge (LSM) Tree is a natu- Real-time data analytics systems such as SAP HANA, MemSQL, ral fit for a lifecycle-aware storage engine. LSM-Trees are widely and IBM Wildfire employ hybrid data layouts, in which dataare used in key-value stores (e.g., Google’s BigTable and LevelDB, Cas- stored in different formats throughout their lifecycle. Recent data sandra, Facebook’s RocksDB), RDBMSs (e.g., Facebook’s MyRocks, are stored in a row-oriented format to serve OLTP workloads and SQLite4), blockchains (e.g., Hyperledger uses LevelDB), and data support high data rates, while older data are transformed to a stream and time-series databases (e.g., InfluxDB). While Cassandra column-oriented format for OLAP access patterns. We observe that and RocksDB can simulate columnar storage via column families, a Log-Structured Merge (LSM) Tree is a natural fit for a lifecycle- we are not aware of any lifecycle-aware LSM-Trees in which the aware storage engine due to its high write throughput and level- storage layout can change throughout the lifetime of the data. We oriented structure, in which records propagate from one level to fill this gap in our work, by extending the capabilities ofLSM- the next over time. To build a lifecycle-aware storage engine using based systems to efficiently serve real-time analytics and HTAP an LSM-Tree, we make a crucial modification to allow different workloads.
    [Show full text]
  • As Focused on Software Tools That Support Software Engineering, Along with Data Structures and Algorithms Generally
    PETER C DILLINGER, Ph.D. 2110 N 89th St [email protected] Seattle WA 98103 http://www.peterd.org 404-509-4879 Overview My work in software has focused on software tools that support software engineering, along with data structures and algorithms generally. My core strength is seeing many paths to “success,” so I'm often the person consulted when others think they're stuck. Highlights ♦ Key developer and project lead in adapting and extending the legendary Coverity static analysis engine, for C/C++ bug finding, to find bugs with high accuracy in Java, C#, JavaScript, PHP, Python, Ruby, Swift, and VB. https://www.synopsys.com/blogs/software-security/author/pdillinger/ ♦ Inventor of a fast, scalable, and accurate method of detecting mistyped identifiers in dynamic languages such as JavaScript, PHP, Python, and Ruby without use of a natural language dictionary. Patent pending, app# 20170329697. Coverity feature: https://stackoverflow.com/a/34796105 ♦ Did the impossible with git: on wanting to “copy with history” as part of a refactoring, I quickly developed a way to do it despite the consensus wisdom. https://stackoverflow.com/a/44036771 ♦ Did the impossible with Bloom filters: made the data structure simultaneously fast and accurate with a simple hashing technique, now used in tools including LevelDB and RocksDB. https://en.wikipedia.org/wiki/Bloom_filter (Search "Dillinger") ♦ Early coder / Linux user: started BASIC in 1st grade; first game hack in 3rd grade; learned C in middle school; wrote Tetris in JavaScript in high school (1997); steady Linux user since 1998. Work Coverity, August 2009 to October 2017, acquired by Synopsys in 2014 Software developer, tech lead, and manager for static and dynamic program analysis projects.
    [Show full text]
  • What's Included in the Homeos
    What’s included in the HomeOS kit (and some tips on using it) 2/21/2011 HomeOS is an experimental operating system for the home which focuses on providing centralized control of connected devices in the home, useful programming abstractions for developers, and allows for the easy addition of new devices and application functionality to the home environment. This document explains what application and driver modules are included in the kit and how to use them in your setup. It is not a programming guide; that would be programming-howto.docx. Because it might be a little out of date, your most up-to-date source of information is the source code itself Contents Drivers ......................................................................................................................................................... 2 DriverAxisCamera: for IP cameras made by Axis Communications ..................................................... 2 DriverDlnaDmr: for DLNA media renderers ......................................................................................... 2 DriverDlnaDms: for DLNA media servers ............................................................................................. 2 DriverImgRec: virtual device for face recognition ............................................................................... 3 DriverNotifications: virtual device for sending notifications over email and SMS .............................. 3 DriverWebCam: for Webcams ............................................................................................................
    [Show full text]
  • Rocksdb…The Old Way…Log Structured Merge
    TRocksDB Remi Brasga Sr. Software Engineer | Memory Storage Strategy Division Toshiba Memory America, Inc. Hi! I’m Remi (Remington Brasga). As a Sr. Software Engineer for the Memory Storage Strategy Division at Toshiba Memory America, Inc., I am dedicated to development and collaboration on open source software. I earned my B.S. and M.S. from University of California, Irvine. Today, I am here fresh from having my 3rd son who has kept me very busy the past few weeks. (thanks for letting me come here and rest!) Applications are King • Software is eating the world • Storage is “Defined by Software” – i.e., applications define how storage is used • Yet, applications do not always use storage wisely For Example, RocksDB RocksDB is a popular data storage engine …used by a wide range of database applications ArangoDB Ceph™ Cassandra® MariaDB® Python® MyRocks Rockset Cassandra is a registered trademark of The Apache Software Foundation. Ceph is a trademark of Red Hat, Inc. or its subsidiaries in the United States and other countries. Python is a registered trademark of the Python Software Foundation. MariaDB is a registered trademark of MariaDB in the European Union and other regions. All other company names, product names and service names may be trademarks of their respective companies. The Challenge of Rocks While highly effective as a data storage engine, RocksDB has some challenges for SSDs: Write Amplification is very large 20-30X (or more) as a result of the compaction layers rewriting the same data This can result in impact to SSD endurance. How does RocksDB work? Keys and Values are stored Data Key together in pairs.
    [Show full text]
  • Reducing DRAM Footprint with NVM in Facebook (Eurosys'18)
    Reducing DRAM Footprint with NVM in Facebook (EuroSys’18) Assaf Eisenman1, Darryl Gardner2, Islam AbdelRahman2, Jens Axboe2, Siying Dong2, Kim Hazelwood2, Chris Petersen2, Asaf Cidon1, Sachin Katti1 1Stanford University 2Facebook, Inc. Extended Abstract first obvious step would be to simply reduce the DRAM Modern key-value stores like RocksDB and LevelDB are capacity on a MyRocks server configuration. Unfortunately, highly dependent on DRAM, even when using a persistent reducing the amount of DRAM degrades the performance storage medium (e.g. flash) for storing data. Since DRAM is of MyRocks. To demonstrate this point, Figure 1 shows that still about 1000× faster than flash, these systems typically when reducing the DRAM cache in MyRocks from 96 GB to leverage DRAM for caching hot indices and objects. 16 GB, the mean latency increases by 2×, and P99 latency MySQL databases are often implemented on top of these increases by 2.2×. key-value stores. For example, MyRocks, which is built on NVM offers the potential to reduce the dependence of top of RocksDB, is the primary MySQL database in Face- systems like MyRocks on DRAM. NVM comes in two book and is used extensively in data center environments. forms: a more expensive byte-addressable form, and a less MyRocks stores petabytes of data and is used to serve real- expensive block device. Since our goal is to reduce total cost time user activities across large-scale web applications [6]. of ownership, we use NVM as a block device, which has a In order to achieve high throughput and low latency, My- cost about an order of magnitude less per bit than DRAM.
    [Show full text]
  • Computational Textiles and Augmenting Space Through Emotion
    Softbuilt: Computational Textiles and Augmenting Space Through Emotion by Felecia A. Davis Bachelor of Science in Engineering, Tufts University (1983) Master of Architecture, Princeton University (1993) Submitted to the Department of Architecture In Partial Fulfilment of the Requirements for the Degree of Doctor of Philosophy in the field of Architecture: Design and Computation at the Massachusetts Institute of Technology September 2017 © 2017 Felecia A. Davis All rights reserved The author hereby grants to M.I.T. permission to reproduce and distribute publicly paper and electronic copies of this thesis document in whole or in part in any medium now known or hereafter created. Signature of Author Felecia Davis 11 August 2017 Department of Architecture Certified by Terry Knight Professor of Design and Computation Department of Architecture Thesis Supervisor Accepted by Sheila Kennedy Professor of Architecture Chair, Department Committee on Graduate Students 1 2 DISSERTATION COMMITTEE Dr. Terry Knight, Chair Professor of Design and Computation Massachusetts Institute of Technology Dr. Edith K. Ackermann Honorary Professor of Developmental Psychology University of Aix-Marseille 1, France Visiting Scientist Design and Computation Massachusetts Institute of Technology Dr. Leah Buechley Designer, Engineer, Educator Former Director High/ Low Tech Lab Massachusetts Institute of Technology Media Lab 3 4 Softbuilt: Computational Textiles and Augmenting Space Through Emotion By Felecia A. Davis Submitted to the Department of Architecture August 11 2017 in Partial Fulfilment of the Requirements for the Degree of Doctor of Philosophy in Architecture: In Design and Computation at the Massachusetts Institute of Technology ABSTRACT When we inhabit, wear, and make textiles we are in conversation with our pre-historical and historical past and in a sense already connected to what is to come by the structure of fabric that operates as a mode of understanding the world.
    [Show full text]