DOCSLIB.ORG

Bypassing the Limit of RAM Capacity in Distributed Le System Lizardfs

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Bypassing the Limit of RAM Capacity in Distributed Le System Lizardfs University of Warsaw Faculty of Mathematics, Informatics and Mechanics Piotr Sarna Student no. 321169 Bypassing the limit of RAM capacity in distributed le system LizardFS Master's thesis in COMPUTER SCIENCE Supervisor: dr Janina Mincer-Daszkiewicz Institute of Informatics June 2016 Supervisor's statement Hereby I conrm that the presented thesis was prepared under my supervision and that it fulls the requirements for the degree of Master of Computer Science. Date Supervisor's signature Author's statement Hereby I declare that the presented thesis was prepared by me and none of its contents was obtained by means that are against the law. The thesis has never before been a subject of any procedure of obtaining an aca- demic degree. Moreover, I declare that the present version of the thesis is identical to the attached electronic version. Date Author's signature Abstract Distributed le systems with a centralized metadata server are constrained by many bound- aries, one of which is limited RAM capacity. This particular issue can be potentially bypassed with the help of fast solid-state disks. My research will be based upon creating a library of al- ternative memory allocation methods, which would utilize underlying drive (e.g. SSD, ZRAM) for data storage, using RAM only as cache. The next step would be to implement the library as primary memory allocator in LizardFS an open-source, distributed le system. Results could then be veried and benchmarked on real-life installations. Keywords distributed, le system, storage, memory, optimization, SSD, ZRAM, LizardFS, library, C++ Thesis domain (Socrates-Erasmus subject area codes) 11.3 Computer Science Subject classication Software and its engineering Software organization and properties Contextual software domains Operating systems Memory management Allocation / deallocation strategies Information systems Information storage systems Storage architectures Distributed storage Tytuª pracy w j¦zyku polskim Obej±cie limitu pami¦ci RAM w rozproszonym systemie plików LizardFS Contents Introduction ........................................ 5 1. Background ....................................... 7 1.1. Introduction to LizardFS . .7 1.1.1. Architecture . .7 1.1.2. Resource consumption . .8 1.1.3. Memory of the master server . .9 1.1.4. Existent memory optimizations . 10 2. Inspiration ........................................ 15 2.1. Introduction to SSDAlloc . 15 2.1.1. Motivation . 15 2.1.2. Keywords . 15 2.1.3. Architecture . 16 2.2. Conclusions . 17 3. Design and implementation ............................. 19 3.1. Design . 19 3.2. Choice of programming language . 19 3.3. Used libraries and portability . 20 3.3.1. Linux-specic headers . 20 3.3.2. BerkeleyDB . 21 3.4. User-space memory management . 21 3.4.1. Details . 21 3.4.2. Performance of signal-based management . 23 3.5. Implementation details . 24 3.5.1. Application layer . 24 3.5.2. Runtime layer . 26 3.5.3. Storage layer . 28 4. Choice of storage ................................... 29 4.1. HDD . 29 4.1.1. Introduction . 29 4.1.2. Access time . 30 4.1.3. Prices . 30 4.1.4. Benchmarks . 30 4.2. SSD . 31 4.2.1. Introduction . 31 4.2.2. Access time . 32 3 4.2.3. Prices . 32 4.2.4. Benchmarks . 32 4.3. In-kernel compression . 32 4.3.1. Introduction . 32 4.3.2. Implementations . 33 4.3.3. Usage examples . 33 4.3.4. Compression algorithms . 34 4.3.5. Benchmarks . 34 4.4. Comparison . 35 4.4.1. Price-oriented choice . 36 4.4.2. Performance-oriented choice . 36 4.4.3. Summary . 36 5. Integration with LizardFS .............................. 39 5.1. Build conguration . 39 5.2. Code changes . 39 5.3. Summary . 41 6. Test results ....................................... 43 6.1. Parameters . 43 6.2. Congurations . 44 6.3. Scope . 44 6.4. Result sheets . 45 6.4.1. without fsalloc . 45 6.4.2. single stream zram . 45 6.4.3. multiple stream zram . 46 6.4.4. HDD . 47 6.4.5. SSD . 48 6.5. Combined results . 49 6.6. In-kernel compressed memory . 51 6.7. Proling . 51 6.7.1. Results . 51 6.7.2. Conclusions . 52 7. Conclusions ....................................... 57 7.1. Evaluation . 57 7.2. Perspectives . 58 7.3. Epilogue . 58 A. LizardFS graphs .................................... 59 B. Contents of the attached CD ............................ 65 Bibliography ........................................ 67 4 Introduction Global need of maintaining very large amounts of data is still increasing. The storage sector is currently enjoying the benets of having many independent distributed le systems designed to store petabytes of data safely and eciently. Unfortunately, these le systems have their own limitations CPU and RAM usage, power consumption, network latencies, keeping consistency of distributed data. One of the interesting architectures is a "le system with centralized metadata server". Its huge advantage is not having to worry about metadata consistency, which is a non-trivial problem to cope with. On the other hand, memory and CPU usage is centralized as well, which leads to very high hardware requirements for large installations. Some systems (e.g. LizardFS [13]) implement a policy of keeping all metadata in random- access memory. By that means, as long as there is enough memory available, metadata operations are performed quickly, without any need to use disk I/O. It is worth noting, that metadata operations are very frequent on a POSIX-compliant le system. The most excessively used one is a lookup call, which is used to extract information about a le (more specically, an inode) from its path. Ensuring that this operation is fast should be the primary goal of a distributed le system developer. This performance-oriented policy has one big aw, which is extremely high memory con- sumption. An attempt to reduce RAM constraint without rewriting the whole application is the main topic of this thesis. Princeton University researchers, Anirudh Badam and Vivek S. Pai presented an interest- ing idea on how to move the burden of dynamically allocated memory from RAM to solid-state hard drives [1, 3]. The project is called SSDAlloc and its goal is to intercept the allocation process and keep the data in designated SSD block device, using RAM only as temporary cache for frequently used pages. Solid-state drives are not the only reasonable way to store allocated data, though. Another noteworthy idea is to take advantage of compressed RAM devices, e.g. ZRAM module [15] available in Linux kernel. The thesis contains documented implementation of a memory management library fsal- loc, designed to overload traditional malloc/new calls with ones that keep allocated data in the database. The library was injected into distributed le system LizardFS in order to compare its performance and memory usage with and without fsalloc. 5 The thesis consists of 7 chapters: 1. Background introduction to open-source distributed le system LizardFS and its memory management. 2. Motivations introduction to SSDAlloc project and its summary. 3. Design and implementation implementation details of fsalloc memory management library. 4. Choice of storage review of storage back-ends that could be used by fsalloc. 5. Integration with LizardFS steps performed to introduce fsalloc as a memory manager for frequently used structures in LizardFS system. 6. Results performance results of experiments carried out on production-ready instal- lations. 7. Conclusions comments regarding performance results and future development oppor- tunities. Electronic version of this thesis, implementation of fsalloc library and detailed performance results are available on the attached CD. 6 Chapter 1 Background Allocation library is not a fully standalone product it is used along with an application that benets from its routines. In this particular case the application is a distributed le system cluster with high resource consumption fsalloc is to be integrated with LizardFS and tested in terms of both correctness and performance. In order to properly understand the design decisions, approaches and other details, overall description of LizardFS background for the whole project is presented in this chapter. 1.1. Introduction to LizardFS LizardFS [13] is an open-source, distributed le system. Project's goal is to deliver fault- tolerant, highly available, scalable and POSIX-compliant alternative to other solutions. Source code is freely available and can be found at https://github.com/lizardfs/lizardfs. 1.1.1. Architecture Conceptually, LizardFS can be divided into 4 parts: 1. metadata server, which can be either of: master, shadow, metalogger, 2. chunkserver, 3. mount, 4. client-side extra features, namely: CGI web server, console administration tool. Master is a centralized metadata management process. It holds all metadata present in the system in RAM, which makes it a primary target for memory usage optimization. Clients gather all needed information about les from master before performing any reads/writes. Chunkservers are managed from master server only decisions on performing a replication between chunkservers or getting rid of certain chunks are commissioned directly from master. 7 Shadow is an online copy of master server, ready to become promoted to master once a failure occurs. Shadows are used in high-availability solutions and keep the same memory structure as masters, so they are the target of RAM optimization as well. Metalogger is an oine copy of system's metadata and is not memory-consuming, so it will not be discussed further in this thesis. Chunkserver is responsible for: keeping chunks of les on internal storage, ensuring that all chunks hold valid information (via checksum calculation), sending and receiving le data from clients. All les in the system are divided into chunks that are up to 64 MiB in size. Chunks are then distributed over chunkservers in order to keep a safe number of copies and follow the rules of georeplication. Each le holds information on how many copies should be kept and on which groups of chunkservers. This information as a whole is called a goal.
Load more

Recommended publications
  • University of California at Berkeley College of Engineering Department of Electrical Engineering and Computer Science
    University of California at Berkeley College of Engineering Department of Electrical Engineering and Computer Science EECS 61C, Fall 2003 Lab 2: Strings and pointers; the GDB debugger PRELIMINARY VERSION Goals To learn to use the gdb debugger to debug string and pointer programs in C. Reading Sections 5.1-5.5, in K&R GDB Reference Card (linked to class page under “resources.”) Optional: Complete GDB documentation (http://www.gnu.org/manual/gdb-5.1.1/gdb.html) Note: GDB currently only works on the following machines: • torus.cs.berkeley.edu • rhombus.cs.berkeley.edu • pentagon.cs.berkeley.edu Please ssh into one of these machines before starting the lab. Basic tasks in GDB There are two ways to start the debugger: 1. In EMACS, type M-x gdb, then type gdb <filename> 2. Run gdb <filename> from the command line The following are fundamental operations in gdb. Please make sure you know the gdb commands for the following operations before you proceed. 1. How do you run a program in gdb? 2. How do you pass arguments to a program when using gdb? 3. How do you set a breakpoint in a program? 4. How do you set a breakpoint which which only occurs when a set of conditions is true (eg when certain variables are a certain value)? 5. How do you execute the next line of C code in the program after a break? 1 6. If the next line is a function call, you'll execute the call in one step. How do you execute the C code, line by line, inside the function call? 7.
    [Show full text]
  • Hard Disk Drives
    37 Hard Disk Drives The last chapter introduced the general concept of an I/O device and showed you how the OS might interact with such a beast. In this chapter, we dive into more detail about one device in particular: the hard disk drive. These drives have been the main form of persistent data storage in computer systems for decades and much of the development of file sys- tem technology (coming soon) is predicated on their behavior. Thus, it is worth understanding the details of a disk’s operation before building the file system software that manages it. Many of these details are avail- able in excellent papers by Ruemmler and Wilkes [RW92] and Anderson, Dykes, and Riedel [ADR03]. CRUX: HOW TO STORE AND ACCESS DATA ON DISK How do modern hard-disk drives store data? What is the interface? How is the data actually laid out and accessed? How does disk schedul- ing improve performance? 37.1 The Interface Let’s start by understanding the interface to a modern disk drive. The basic interface for all modern drives is straightforward. The drive consists of a large number of sectors (512-byte blocks), each of which can be read or written. The sectors are numbered from 0 to n − 1 on a disk with n sectors. Thus, we can view the disk as an array of sectors; 0 to n − 1 is thus the address space of the drive. Multi-sector operations are possible; indeed, many file systems will read or write 4KB at a time (or more). However, when updating the disk, the only guarantee drive manufacturers make is that a single 512-byte write is atomic (i.e., it will either complete in its entirety or it won’t com- plete at all); thus, if an untimely power loss occurs, only a portion of a larger write may complete (sometimes called a torn write).
    [Show full text]
  • Virtual Memory
    Chapter 4 Virtual Memory Linux processes execute in a virtual environment that makes it appear as if each process had the entire address space of the CPU available to itself. This virtual address space extends from address 0 all the way to the maximum address. On a 32-bit platform, such as IA-32, the maximum address is 232 − 1or0xffffffff. On a 64-bit platform, such as IA-64, this is 264 − 1or0xffffffffffffffff. While it is obviously convenient for a process to be able to access such a huge ad- dress space, there are really three distinct, but equally important, reasons for using virtual memory. 1. Resource virtualization. On a system with virtual memory, a process does not have to concern itself with the details of how much physical memory is available or which physical memory locations are already in use by some other process. In other words, virtual memory takes a limited physical resource (physical memory) and turns it into an infinite, or at least an abundant, resource (virtual memory). 2. Information isolation. Because each process runs in its own address space, it is not possible for one process to read data that belongs to another process. This improves security because it reduces the risk of one process being able to spy on another pro- cess and, e.g., steal a password. 3. Fault isolation. Processes with their own virtual address spaces cannot overwrite each other’s memory. This greatly reduces the risk of a failure in one process trig- gering a failure in another process. That is, when a process crashes, the problem is generally limited to that process alone and does not cause the entire machine to go down.
    [Show full text]
  • The Linux Kernel Module Programming Guide
    The Linux Kernel Module Programming Guide Peter Jay Salzman Michael Burian Ori Pomerantz Copyright © 2001 Peter Jay Salzman 2007−05−18 ver 2.6.4 The Linux Kernel Module Programming Guide is a free book; you may reproduce and/or modify it under the terms of the Open Software License, version 1.1. You can obtain a copy of this license at http://opensource.org/licenses/osl.php. This book is distributed in the hope it will be useful, but without any warranty, without even the implied warranty of merchantability or fitness for a particular purpose. The author encourages wide distribution of this book for personal or commercial use, provided the above copyright notice remains intact and the method adheres to the provisions of the Open Software License. In summary, you may copy and distribute this book free of charge or for a profit. No explicit permission is required from the author for reproduction of this book in any medium, physical or electronic. Derivative works and translations of this document must be placed under the Open Software License, and the original copyright notice must remain intact. If you have contributed new material to this book, you must make the material and source code available for your revisions. Please make revisions and updates available directly to the document maintainer, Peter Jay Salzman <[email protected]>. This will allow for the merging of updates and provide consistent revisions to the Linux community. If you publish or distribute this book commercially, donations, royalties, and/or printed copies are greatly appreciated by the author and the Linux Documentation Project (LDP).
    [Show full text]
  • Memory Management and Garbage Collection
    Overview Memory Management Stack: Data on stack (local variables on activation records) have lifetime that coincides with the life of a procedure call. Memory for stack data is allocated on entry to procedures ::: ::: and de-allocated on return. Heap: Data on heap have lifetimes that may differ from the life of a procedure call. Memory for heap data is allocated on demand (e.g. malloc, new, etc.) ::: ::: and released Manually: e.g. using free Automatically: e.g. using a garbage collector Compilers Memory Management CSE 304/504 1 / 16 Overview Memory Allocation Heap memory is divided into free and used. Free memory is kept in a data structure, usually a free list. When a new chunk of memory is needed, a chunk from the free list is returned (after marking it as used). When a chunk of memory is freed, it is added to the free list (after marking it as free) Compilers Memory Management CSE 304/504 2 / 16 Overview Fragmentation Free space is said to be fragmented when free chunks are not contiguous. Fragmentation is reduced by: Maintaining different-sized free lists (e.g. free 8-byte cells, free 16-byte cells etc.) and allocating out of the appropriate list. If a small chunk is not available (e.g. no free 8-byte cells), grab a larger chunk (say, a 32-byte chunk), subdivide it (into 4 smaller chunks) and allocate. When a small chunk is freed, check if it can be merged with adjacent areas to make a larger chunk. Compilers Memory Management CSE 304/504 3 / 16 Overview Manual Memory Management Programmer has full control over memory ::: with the responsibility to manage it well Premature free's lead to dangling references Overly conservative free's lead to memory leaks With manual free's it is virtually impossible to ensure that a program is correct and secure.
    [Show full text]
  • The Case for Compressed Caching in Virtual Memory Systems
    THE ADVANCED COMPUTING SYSTEMS ASSOCIATION The following paper was originally published in the Proceedings of the USENIX Annual Technical Conference Monterey, California, USA, June 6-11, 1999 The Case for Compressed Caching in Virtual Memory Systems _ Paul R. Wilson, Scott F. Kaplan, and Yannis Smaragdakis aUniversity of Texas at Austin © 1999 by The USENIX Association All Rights Reserved Rights to individual papers remain with the author or the author's employer. Permission is granted for noncommercial reproduction of the work for educational or research purposes. This copyright notice must be included in the reproduced paper. USENIX acknowledges all trademarks herein. For more information about the USENIX Association: Phone: 1 510 528 8649 FAX: 1 510 548 5738 Email: [email protected] WWW: http://www.usenix.org The Case for Compressed Caching in Virtual Memory Systems Paul R. Wilson, Scott F. Kaplan, and Yannis Smaragdakis Dept. of Computer Sciences University of Texas at Austin Austin, Texas 78751-1182 g fwilson|sfkaplan|smaragd @cs.utexas.edu http://www.cs.utexas.edu/users/oops/ Abstract In [Wil90, Wil91b] we proposed compressed caching for virtual memory—storing pages in compressed form Compressed caching uses part of the available RAM to in a main memory compression cache to reduce disk pag- hold pages in compressed form, effectively adding a new ing. Appel also promoted this idea [AL91], and it was level to the virtual memory hierarchy. This level attempts evaluated empirically by Douglis [Dou93] and by Russi- to bridge the huge performance gap between normal (un- novich and Cogswell [RC96]. Unfortunately Douglis’s compressed) RAM and disk.
    [Show full text]
  • File Systems and Disk Layout I/O: the Big Picture
    File Systems and Disk Layout I/O: The Big Picture Processor interrupts Cache Memory Bus I/O Bridge Main I/O Bus Memory Disk Graphics Network Controller Controller Interface Disk Disk Graphics Network 1 Rotational Media Track Sector Arm Cylinder Platter Head Access time = seek time + rotational delay + transfer time seek time = 5-15 milliseconds to move the disk arm and settle on a cylinder rotational delay = 8 milliseconds for full rotation at 7200 RPM: average delay = 4 ms transfer time = 1 millisecond for an 8KB block at 8 MB/s Bandwidth utilization is less than 50% for any noncontiguous access at a block grain. Disks and Drivers Disk hardware and driver software provide basic facilities for nonvolatile secondary storage (block devices). 1. OS views the block devices as a collection of volumes. A logical volume may be a partition ofasinglediskora concatenation of multiple physical disks (e.g., RAID). 2. OS accesses each volume as an array of fixed-size sectors. Identify sector (or block) by unique (volumeID, sector ID). Read/write operations DMA data to/from physical memory. 3. Device interrupts OS on I/O completion. ISR wakes up process, updates internal records, etc. 2 Using Disk Storage Typical operating systems use disks in three different ways: 1. System calls allow user programs to access a “raw” disk. Unix: special device file identifies volume directly. Any process that can open thedevicefilecanreadorwriteany specific sector in the disk volume. 2. OS uses disk as backing storage for virtual memory. OS manages volume transparently as an “overflow area” for VM contents that do not “fit” in physical memory.
    [Show full text]
  • Linux Physical Memory Analysis
    Linux Physical Memory Analysis Paul Movall International Business Machines Corporation 3605 Highway 52N, Rochester, MN Ward Nelson International Business Machines Corporation 3605 Highway 52N, Rochester, MN Shaun Wetzstein International Business Machines Corporation 3605 Highway 52N, Rochester, MN Abstract A significant amount of this information can alo be found in the /proc filesystem provided by the kernel. We present a tool suite for analysis of physical memory These /proc entries can be used to obtain many statis- usage within the Linux kernel environment. This tool tics including several related to memory usage. For ex- suite can be used to collect and analyze how the physi- ample, /proc/<pid>/maps can be used to display a pro- cal memory within a Linuxenvironment is being used. cess' virtual memory map. Likewise, the contents of / proc/<pid>/status can be used to retreive statistics about virtual memory usage as well as the Resident Set Size (RSS). Embedded subsystems are common in today's computer In most cases, this process level detail is sufficient to systems. These embedded subsystems range from the analyze memory usage. In systems that do not have very simple to the very complex. In such embedded sys- backing store, more details are often needed to analyze tems, memory is scarce and swap is non-existent. When memory usage. For example, it's useful to know which adapting Linux for use in this environment, we needed pages in a process' address map are resident, not just to keep a close eye on physical memory usage. how many. This information can be used to get some clues on the usage of a shared library.
    [Show full text]
  • Memory Protection at Option
    Memory Protection at Option Application-Tailored Memory Safety in Safety-Critical Embedded Systems – Speicherschutz nach Wahl Auf die Anwendung zugeschnittene Speichersicherheit in sicherheitskritischen eingebetteten Systemen Der Technischen Fakultät der Universität Erlangen-Nürnberg zur Erlangung des Grades Doktor-Ingenieur vorgelegt von Michael Stilkerich Erlangen — 2012 Als Dissertation genehmigt von der Technischen Fakultät Universität Erlangen-Nürnberg Tag der Einreichung: 09.07.2012 Tag der Promotion: 30.11.2012 Dekan: Prof. Dr.-Ing. Marion Merklein Berichterstatter: Prof. Dr.-Ing. Wolfgang Schröder-Preikschat Prof. Dr. Michael Philippsen Abstract With the increasing capabilities and resources available on microcontrollers, there is a trend in the embedded industry to integrate multiple software functions on a single system to save cost, size, weight, and power. The integration raises new requirements, thereunder the need for spatial isolation, which is commonly established by using a memory protection unit (MPU) that can constrain access to the physical address space to a fixed set of address regions. MPU-based protection is limited in terms of available hardware, flexibility, granularity and ease of use. Software-based memory protection can provide an alternative or complement MPU-based protection, but has found little attention in the embedded domain. In this thesis, I evaluate qualitative and quantitative advantages and limitations of MPU-based memory protection and software-based protection based on a multi-JVM. I developed a framework composed of the AUTOSAR OS-like operating system CiAO and KESO, a Java implementation for deeply embedded systems. The framework allows choosing from no memory protection, MPU-based protection, software-based protection, and a combination of the two.
    [Show full text]
  • Project Snowflake: Non-Blocking Safe Manual Memory Management in .NET
    Project Snowflake: Non-blocking Safe Manual Memory Management in .NET Matthew Parkinson Dimitrios Vytiniotis Kapil Vaswani Manuel Costa Pantazis Deligiannis Microsoft Research Dylan McDermott Aaron Blankstein Jonathan Balkind University of Cambridge Princeton University July 26, 2017 Abstract Garbage collection greatly improves programmer productivity and ensures memory safety. Manual memory management on the other hand often delivers better performance but is typically unsafe and can lead to system crashes or security vulnerabilities. We propose integrating safe manual memory management with garbage collection in the .NET runtime to get the best of both worlds. In our design, programmers can choose between allocating objects in the garbage collected heap or the manual heap. All existing applications run unmodified, and without any performance degradation, using the garbage collected heap. Our programming model for manual memory management is flexible: although objects in the manual heap can have a single owning pointer, we allow deallocation at any program point and concurrent sharing of these objects amongst all the threads in the program. Experimental results from our .NET CoreCLR implementation on real-world applications show substantial performance gains especially in multithreaded scenarios: up to 3x savings in peak working sets and 2x improvements in runtime. 1 Introduction The importance of garbage collection (GC) in modern software cannot be overstated. GC greatly improves programmer productivity because it frees programmers from the burden of thinking about object lifetimes and freeing memory. Even more importantly, GC prevents temporal memory safety errors, i.e., uses of memory after it has been freed, which often lead to security breaches. Modern generational collectors, such as the .NET GC, deliver great throughput through a combination of fast thread-local bump allocation and cheap collection of young objects [63, 18, 61].
    [Show full text]
  • Memory Safety Without Garbage Collection for Embedded Applications
    Memory Safety Without Garbage Collection for Embedded Applications DINAKAR DHURJATI, SUMANT KOWSHIK, VIKRAM ADVE, and CHRIS LATTNER University of Illinois at Urbana-Champaign Traditional approaches to enforcing memory safety of programs rely heavily on run-time checks of memory accesses and on garbage collection, both of which are unattractive for embedded ap- plications. The goal of our work is to develop advanced compiler techniques for enforcing memory safety with minimal run-time overheads. In this paper, we describe a set of compiler techniques that, together with minor semantic restrictions on C programs and no new syntax, ensure memory safety and provide most of the error-detection capabilities of type-safe languages, without using garbage collection, and with no run-time software checks, (on systems with standard hardware support for memory management). The language permits arbitrary pointer-based data structures, explicit deallocation of dynamically allocated memory, and restricted array operations. One of the key results of this paper is a compiler technique that ensures that dereferencing dangling pointers to freed memory does not violate memory safety, without annotations, run-time checks, or garbage collection, and works for arbitrary type-safe C programs. Furthermore, we present a new inter- procedural analysis for static array bounds checking under certain assumptions. For a diverse set of embedded C programs, we show that we are able to ensure memory safety of pointer and dy- namic memory usage in all these programs with no run-time software checks (on systems with standard hardware memory protection), requiring only minor restructuring to conform to simple type restrictions. Static array bounds checking fails for roughly half the programs we study due to complex array references, and these are the only cases where explicit run-time software checks would be needed under our language and system assumptions.
    [Show full text]
  • Extracting Compressed Pages from the Windows 10 Virtual Store WHITE PAPER | EXTRACTING COMPRESSED PAGES from the WINDOWS 10 VIRTUAL STORE 2
    white paper Extracting Compressed Pages from the Windows 10 Virtual Store WHITE PAPER | EXTRACTING COMPRESSED PAGES FROM THE WINDOWS 10 VIRTUAL STORE 2 Abstract Windows 8.1 introduced memory compression in August 2013. By the end of 2013 Linux 3.11 and OS X Mavericks leveraged compressed memory as well. Disk I/O continues to be orders of magnitude slower than RAM, whereas reading and decompressing data in RAM is fast and highly parallelizable across the system’s CPU cores, yielding a significant performance increase. However, this came at the cost of increased complexity of process memory reconstruction and thus reduced the power of popular tools such as Volatility, Rekall, and Redline. In this document we introduce a method to retrieve compressed pages from the Windows 10 Memory Manager Virtual Store, thus providing forensics and auditing tools with a way to retrieve, examine, and reconstruct memory artifacts regardless of their storage location. Introduction Windows 10 moves pages between physical memory and the hard disk or the Store Manager’s virtual store when memory is constrained. Universal Windows Platform (UWP) applications leverage the Virtual Store any time they are suspended (as is the case when minimized). When a given page is no longer in the process’s working set, the corresponding Page Table Entry (PTE) is used by the OS to specify the storage location as well as additional data that allows it to start the retrieval process. In the case of a page file, the retrieval is straightforward because both the page file index and the location of the page within the page file can be directly retrieved.
    [Show full text]