DOCSLIB.ORG

Resilient Filesystem

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Resilient Filesystem Resilient Filesystem Henry Georges Information Security Submission date: May 2018 Supervisor: Stefan Axelsson, IIK Co-supervisor: Rune Nordvik, Politihøgskolen Norwegian University of Science and Technology Department of Information Security and Communication Technology ReFS Henry Georges 22-05-2018 Master’s Thesis Master of Science in Information Security 30 ECTS Department of Computer Science and Media Technology Norwegian University of Science and Technology, 2018 Supervisor 1: Professor Stefan Axelsson, NTNU Supervisor 2: Assistant Professor Rune Nordvik, PHS ReFS Acknowledgement I would like to thank my supervisor, Assistant Professor Rune Nordvik, for his patience and engagement. H.G. i ReFS Abstract MICROSOFT developed a new file system, REFS. This Resilient FileSystem is in- tended to replace NTFS, hence the importance and usage of REFS should in- crease over the next few years. Although we have been able to use REFS since the presence of WINDOWS SERVER 2012 and WINDOWS 8, there is almost no documentation on this file system available, especially not from MICROSOFT. The aim of this master thesis is: to examine the structure of REFS; to enhance the knowledge about this file system and how it works; to examine how MICROSOFT designed the file system to improve data-integrity. Because of the missing documentation and the lack of support of REFS by the main forensic tools, reverse engineering was the method of choice and utilized for the examination of prepared REFS partitions. The results of this master thesis will help other forensicators to verify the results provided by other, usually commercial, tools, or, in the case that the tools do not provide the needed functions, to do these jobs manually. This master thesis will describe the experimental setup and the way of reverse engineering. Furthermore it will provide the results in a detailed way, supple- mented by tables for a quick access to the results. To round of the master thesis, it will also provide a self coded python tool, which can be used for analyzing a REFS a partition, at least with REFS version number v1.2. This code was used within the analysis as a proof of concept for the results. Thereby my python tool, described in pull-outs within this master thesis, is the only open source tool, which is able to handle REFS. Because of comparing my results with the output of ENCASE, the only commercial tool I found, which was able to handle REFS, I was able to find faulty outputs from ENCASE, what is also a proof, that this work is necessary. ii ReFS Contents Acknowledgement ............................... i Abstract ..................................... ii Contents .................................... iii List of Figures ................................. v List of Tables .................................. vii Listings ..................................... viii 1 Introduction ................................ 1 1.1 Target Group.............................2 1.2 The Significance of Open Source Knowledge in Forensics.....2 1.3 ReFS - Reverse Engineering of a File System............3 1.4 Overall Problem............................7 1.5 Research Questions..........................9 1.6 Limitations.............................. 12 1.7 Master Thesis Outline........................ 14 2 Background ................................ 16 2.1 Literature Research.......................... 16 2.1.1 Building the next generation file system for Windows: ReFS 16 2.1.2 The Microsoft ReFS On-Disk Layout............. 18 2.1.3 Verifying the properties of ReFS............... 18 2.1.4 Open Source Digital Forensic Tools............. 20 2.1.5 Specialization Project.................... 23 2.2 File System.............................. 24 2.3 B-Tree................................. 25 2.4 Encase................................. 27 2.5 File System Recognition Structure.................. 27 2.6 Reverse Engineering......................... 28 3 Methodology ................................ 29 3.1 Research Design........................... 29 3.2 Testing Environment......................... 30 3.3 Proof of Concept........................... 31 4 Results ................................... 33 4.1 VBR.................................. 33 4.2 $Tree_Control and System Files................... 34 4.2.1 Objects............................ 35 4.2.2 Allocation........................... 36 4.2.3 Attributes........................... 41 iii ReFS 4.3 EntryBlock............................... 41 4.3.1 Starting Area......................... 41 4.3.2 Record Area.......................... 44 5 Discussion ................................. 47 6 Conclusions ................................ 50 6.1 Research Questions.......................... 50 6.2 Future Work.............................. 52 Bibliography .................................. 54 A Appendix .................................. 58 A.1 Preparation.............................. 58 A.1.1 HDD.............................. 58 A.1.2 Preparation Machine..................... 58 A.1.3 Examination Machine.................... 59 A.1.4 Other Hardware / Software................. 59 A.2 Detailed Results............................ 60 A.2.1 VBR.............................. 60 A.2.2 $Tree_Control......................... 61 A.2.3 Second Node......................... 63 A.2.4 FNA.............................. 66 A.2.5 $Recycle.bin......................... 74 A.3 Coding................................. 79 A.3.1 Basics............................. 79 A.3.2 Vbr............................... 80 A.3.3 Tree-Control......................... 82 A.3.4 Analysis of the extracted information............ 90 A.4 Templates............................... 94 A.4.1 VBR.............................. 94 A.4.2 $Tree_Control......................... 94 A.4.3 $Object_Tree......................... 95 A.4.4 $Allocator_Lrg, $Allocator_Med, $Allocator_Sml..... 96 A.4.5 $Object............................ 96 A.4.6 EntryBlock.......................... 97 A.4.7 Directory Metadata Record (0x10000000)......... 98 A.4.8 FileNameAttribute File (0x30000100)........... 98 A.4.9 FileNameAttribute Folder (0x30000200).......... 99 A.4.10 MetaDataAttribute...................... 99 A.4.11 DataRunAttribute....................... 100 A.4.12 Child Attribute (0x20000080)................ 101 iv ReFS List of Figures 1 Structure of the Master Thesis.................... 15 2 Graphic construction of REFS.................... 16 3 Graphic abstraction of B-TREE-structure.............. 17 4 Schematic representation of a B-TREE[1].............. 25 5 Hex-dump from a VBR ........................ 33 6 Schematic layout of a REFS partition................ 34 7 $TREE_CONTROL ........................... 34 8 Dump from record area of the $OBJECT_TREE ........... 35 9 Dump from the record area of the $OBJECT ............ 36 10 Dump from the record area of the $ALLOCATOR_LRG ....... 36 11 How does allocation work...................... 39 12 Interaction between the $ALLOCATOR files............. 41 13 Starting of an ENTRYBLOCK ..................... 42 14 Visualization for the usage of extents................ 43 15 Record Area of an ENTRYBLOCK ................... 44 16 Timestamps.............................. 45 17 Flags-scheme, no flags........................ 46 18 Flags-scheme, hidden flag...................... 46 19 Flags-scheme, ead only flag..................... 46 20 Flags-scheme, archive flag...................... 46 21 VBR .................................. 60 22 $TREE_CONTROL ........................... 61 23 ENTRYBLOCK 0x19E, second node.................. 63 24 ENTRYBLOCK 0x19D, second node.................. 63 25 ENTRYBLOCK 0x19C, second node.................. 64 26 FNA of sys.info............................ 64 27 Output from the $fsuti l....................... 65 28 ENTRYBLOCK-DESCRIPTOR 0x19C.................. 65 29 ENTRYBLOCK-DESCRIPTOR 0x19D.................. 65 30 ENTRYBLOCK-DESCRIPTOR 0x19E.................. 66 31 ENTRYBLOCK-DESCRIPTOR 0x19F.................. 66 32 FILENAME-attribute HUGE-FILE, part 1............... 66 33 FILENAME-attribute,HUGE-FILE, part 2............... 68 34 FILENAME-attribute HUGE-FILE, part 3............... 70 35 FileNameAttribute - Folder, screenshot ENCASE .......... 74 36 FileNameAttribute - Folder, dump from ENTRYBLOCK 0x190... 74 v ReFS 37 NODE 0X703, deleted file...................... 75 38 $RECYCLE.BIN, records 4 and 5................... 76 39 $RECYCLE.BIN, Record 5, DATARUN-ATTRIBUTE .......... 76 40 $RECYCLE.BIN, filename of deleted file............... 77 41 $RECYCLE.BIN, record 7....................... 77 42 $RECYCLE.BIN, record 6....................... 78 43 $RECYCLE.BIN, record 6 DATARUN-ATTRIBUTE ........... 78 44 $RECYCLE.BIN, original records of deleted file........... 78 45 Screenshot from the Main Menu of my tool............ 80 46 Screenshot from the Fsstat of my tool................ 90 47 Screenshot from the visualized directory structure......... 91 48 Screenshot from the list of current files............... 91 49 Screenshot from file details..................... 92 50 Screenshot from the $RECYCLE.BIN ................. 92 51 Screenshot from file details (deleted files)............. 93 vi ReFS List of Tables 1 REFS capacitive parameters [2]................... 20 2 Breakdown of the $ALLOCATOR_LRG ................ 37 3 Flags.................................. 46 4 VBR .................................. 60 5 Breakdown 2nd extent $TREE_CONTROL .............. 62 6 Breakdown of the ENTRYBLOCK-Descriptor............. 66 7 Breakdown for FILENAME-attribute................. 67 8 Breakdown for METADATA-attribute................
Load more

Recommended publications
  • File Protection – Using Rsync Whitepaper
    File Protection – Using Rsync Whitepaper Contents 1. Introduction ..................................................................................................................................... 2 Documentation .................................................................................................................................................................. 2 Licensing ............................................................................................................................................................................... 2 Terminology ........................................................................................................................................................................ 2 2. Rsync technology ............................................................................................................................ 3 Overview ............................................................................................................................................................................... 3 Implementation ................................................................................................................................................................. 3 3. Rsync data hosts .............................................................................................................................. 5 Third Party data host ......................................................................................................................................................
    [Show full text]
  • Active @ UNDELETE Users Guide | TOC | 2
    Active @ UNDELETE Users Guide | TOC | 2 Contents Legal Statement..................................................................................................4 Active@ UNDELETE Overview............................................................................. 5 Getting Started with Active@ UNDELETE........................................................... 6 Active@ UNDELETE Views And Windows......................................................................................6 Recovery Explorer View.................................................................................................... 7 Logical Drive Scan Result View.......................................................................................... 7 Physical Device Scan View................................................................................................ 8 Search Results View........................................................................................................10 Application Log...............................................................................................................11 Welcome View................................................................................................................11 Using Active@ UNDELETE Overview................................................................. 13 Recover deleted Files and Folders.............................................................................................. 14 Scan a Volume (Logical Drive) for deleted files..................................................................15
    [Show full text]
  • Cyber502x Computer Forensics
    CYBER502x Computer Forensics Unit 5: Windows File Systems CYBER 502x Computer Forensics | Yin Pan Basic concepts in Windows • Clusters • The basic storage unit of a disk • The piece of storage that an operating system can actually place data into • Different disk formats have different cluster sizes • Slack space • If they are not filled up-which, the last one almost never is –this excess capacity in the last cluster Old Data Old New Data Overwrites CYBER 502x Computer Forensics | Yin Pan What does a file system do? • Make a structure for an operating system to stores files • For you to access them by name, location, date, or other characteristic. • File System Format • The process of turning a partition into a recognizable file system CYBER 502x Computer Forensics | Yin Pan Windows File Systems • File Allocation Table (FAT) • FAT 12 • FAT 16 • FAT 32 • exFAT • NTFS, a file system for Windows NT/2K • NTFS4 • NTFS5 • ReFS, a file system for Windows Server 2012 CYBER 502x Computer Forensics | Yin Pan FAT File System Structure • The boot record • The File Allocation Tables • The root directory • The data area CYBER 502x Computer Forensics | Yin Pan Boot record • The first sector of a FAT12 or FAT16 volume • The first 3 sectors of a FAT 32 volume • Defines the volume, the offset of the other three areas • Contains boot program if it is bootable CYBER 502x Computer Forensics | Yin Pan FAT (File Allocation Table ) • A lookup table to see which cluster comes next • File Allocation Table for FAT 16 • One entry is 16 bits representing one cluster • Each entry can be • The cluster contains defective sectors (FFF7) • the address of the next cluster in the same file (A8F7) • a special value for "not allocated" (0000) • a special value for "this is the last cluster in the chain“ (FFFF) CYBER 502x Computer Forensics | Yin Pan Directory entry structure • Starting from the root directory.
    [Show full text]
  • MY PASSPORT™ SSD Portable Hard Drive User Manual Accessing Online Support Visit Our Product Support Website at and Choose from These Topics
    MY PASSPORT™ SSD Portable Hard Drive User Manual Accessing Online Support Visit our product support website at http://support.wdc.com and choose from these topics: ▪ Downloads — Download software and updates for your WD product ▪ Registration — Register your WD product to get the latest updates and special offers at http://register.wdc.com. You can also register using WD Discovery software. ▪ Warranty & RMA Services — Get warranty, product replacement (RMA), RMA status, and data recovery information ▪ Knowledge Base — Search by keyword, phrase, or Answer ID ▪ Installation — Get online installation help for your WD product or software ▪ WD Community — Share your thoughts and connect with other WD users at http://community.wdc.com Table of Contents _________ Accessing Online Support.................................................................................ii _________ 1 About Your WD Drive.................................................................................... 1 Features.............................................................................................................................1 Kit Contents......................................................................................................................2 Optional Accessories.......................................................................................................2 Operating System Compatibility....................................................................................2 Disk Drive Format............................................................................................................
    [Show full text]
  • Refs: Is It a Game Changer? Presented By: Rick Vanover, Director, Technical Product Marketing & Evangelism, Veeam
    Technical Brief ReFS: Is It a Game Changer? Presented by: Rick Vanover, Director, Technical Product Marketing & Evangelism, Veeam Sponsored by ReFS: Is It a Game Changer? OVERVIEW Backing up data is more important than ever, as data centers store larger volumes of information and organizations face various threats such as ransomware and other digital risks. Microsoft’s Resilient File System or ReFS offers a more robust solution than the old NT File System. In fact, Microsoft has stated that ReFS is the preferred data volume for Windows Server 2016. ReFS is an ideal solution for backup storage. By utilizing the ReFS BlockClone API, Veeam has developed Fast Clone, a fast, efficient storage backup solution. This solution offers organizations peace of mind through a more advanced approach to synthetic full backups. CONTEXT Rick Vanover discussed Microsoft’s Resilient File System (ReFS) and described how Veeam leverages this technology for its Fast Clone backup functionality. KEY TAKEAWAYS Resilient File System is a Microsoft storage technology that can transform the data center. Resilient File System or ReFS is a valuable Microsoft storage technology for data centers. Some of the key differences between ReFS and the NT File System (NTFS) are: ReFS provides many of the same limits as NTFS, but supports a larger maximum volume size. ReFS and NTFS support the same maximum file name length, maximum path name length, and maximum file size. However, ReFS can handle a maximum volume size of 4.7 zettabytes, compared to NTFS which can only support 256 terabytes. The most common functions are available on both ReFS and NTFS.
    [Show full text]
  • FWD-47W800P 47" BRAVIA Professional Full HD LED Display
    FWD-47W800P 47" BRAVIA Professional Full HD LED display Overview Slim, energy-saving screen for corporate display and digital signage applications This slim, energy efficient 47” Full HD LED display is the smart way to make your point in boardrooms and offices, public spaces, retail venues and schools. It’s easy to install, with plentiful connections and Wi-Fi networking on board. USB playback and support for web-friendly HTML5 simplifies low-cost signage applications. Features Edge LED Backlight with Frame Dimming Impress your audience with high-contrast Full HD images; Frame Dimming intelligently adjusts backlight levels to save energy. HTML support for simple box-free digital signage HTML5 browser displays networked content – including text, graphics, video and web feeds – with no dedicated hardware player needed. D-Sub 15 pin and HDMI input connections Easily link BRAVIA to a PC or signage player via the display’s standard D-Sub 15 pin connector, or via HDMI. Customisable display settings © 2004 - 2021 Sony Corporation. All rights reserved. 1 Reproduction in whole or in part without written permission is prohibited. Features and specifications are subject to change without notice. The values for mass and dimension are approximate. All trademarks are the property of their respective owners. Customise and store display settings and features for certain business requirements. Settings can be copied from display to display via USB flash memory. Styled to impress Enhance any business environment or public space with stylish, contemporary ‘Quartz Edge’ design and super- slim 17mm bezel. Integrated media player Play videos and other media content direct from USB flash memory in wide range of formats.
    [Show full text]
  • File Allocation Table Example Download Comprehensive Guide to Formatting USB Drive to Exfat
    file allocation table example download Comprehensive Guide to Formatting USB Drive to exFAT. If you work in an environment where you constantly use a flash drive between a Windows and Mac computer, you may find that you constantly have to format USB drive. One way to permanently solve this problem is to format usb flash drive to exFAT, a platform-independent file system. Which Format to Choose? FAT32, NTFS, or exFAT? Before we get into the actual process of formatting USB drive to exFAT, we need to understand exFAT and other files systems, specifically, FAT32 and NTFS. FAT32 : FAT32 is the oldest file system. FAT is an acronym for File Allocation Table. FAT32 was introduced way back in Windows 95 and was the successor to the older FAT16 that was used on Dos and Windows 3. It currently works on all Windows versions, Mac and Linux. This is the reason it is also one of the most ubiquitous file systems and comes pre-installed on almost all USB you buy at a store. Unfortunately, FAT32 comes with limitations. One of the biggest drawbacks is a maximum file limit size of 4GB. In today's world where video files can often be larger than that, FAT32 is often impractical. FAT32 also limits partition sizes to 8TB. NTFS : NTFS or NT file system is the default file system used by Windows. NTFS has a huge file size and partition limits that are theoretically impossible to surpass. It originally debuted in Windows NT and later in Windows XP. NTFS is compatible with Windows but files can only be opened in read-only mode in Mac and some Linux distributions.
    [Show full text]
  • OWNER's MANUAL. Contents
    Contents A-Z OWNER'S MANUAL. MINI COUNTRYMAN. Online Edition for Part no. 01 40 2 976 632 - X/16 MINI Owner's Manual for the vehicle Thank you for choosing a MINI. The more familiar you are with your vehicle, the better control you will have on the road. We therefore strongly suggest: Read this Owner's Manual before starting off in your new MINI. Also use the Integrated Owner's Manual in your vehicle. It con‐ tains important information on vehicle operation that will help you make full use of the technical features available in your MINI. The manual also contains information designed to en‐ hance operating reliability and road safety, and to contribute to maintaining the value of your MINI. Any updates made after the editorial deadline can be found in the appendix of the printed Owner's Manual for the vehicle. Get started now. We wish you driving fun and inspiration with your MINI. Online Edition for Part no. 01 40 2 976 632 - X/16 © 2016 Bayerische Motoren Werke Aktiengesellschaft Munich, Germany Reprinting, including excerpts, only with the written consent of BMW AG, Munich. US English ID5 X/16, 11 16 490 Printed on environmentally friendly paper, bleached without chlorine, suitable for recycling. Online Edition for Part no. 01 40 2 976 632 - X/16 Contents The fastest way to find information on a partic‐ MOBILITY ular topic or item is by using the index, refer to 204 Refueling page 268. 206 Fuel 208 Wheels and tires 224 Engine compartment 6 Information 226 Engine oil AT A GLANCE 230 Coolant 14 Cockpit 232 Maintenance 18 Onboard monitor
    [Show full text]
  • SATA-IP Exfat Reference Design Introduction
    exFAT Reference Design Introduction Ver1.1E Direct file-system access from SATA-IP 2013/9/16 Design Gateway Page 1 What is exFAT? • Industrial standard File System – Compatible among Windows(XP/Vista/7/8),Mac,Linux... • Suitable for flash storage such as SDXC,USB memory. – Also applicable to SSD/HDD • Maximum file size = 16 Ei-Byte (2^60byte) – For FAT32, max file size is only 4GByte exFAT file system is supported by popular OS. 2013/9/16 Design Gateway Page 2 Application merit of exFAT with SATA-IP 1 • Direct access to the recorded data from the PC – Record data as an exFAT file by this design application. – Remove drive and reconnect to SATA port of Host PC. – PC can detect drive and can access to the recorded file. Record data PC can directly access to the by exFAT Remove drive recorded data and reconnect to the PC SATA PC can directly access to the recorded data 2013/9/16 Design Gateway Page 3 Application merit of exFAT with SATA-IP 2 • Playback pattern data recorded by the PC – Save pattern data to the drive as an exFAT file. – Remove drive and reconnect to the FPGA via SATA-IP. – FPGA can directly read data from the connected drive. PC can directly read data for Save pattern data by exFAT Remove drive playback operation and reconnect to the FPGA SATA Data playback from FPGA saved by the PC 2013/9/16 Design Gateway Page 4 exFAT reference design summary 1 • Reference design for Kintex-7/Zynq-7000 – Operation on KC705/ZC706+AB09-FMCRAID environment – Optional product for exFAT application development • Real read/write access to connected
    [Show full text]
  • Refs V2 Cloning, Projecting, and Moving Data
    ReFS v2 Cloning, projecting, and moving data J.R. Tipton [email protected] What are we talking about? • Two technical things we should talk about • Block cloning in ReFS • ReFS data movement & transformation • What I would love to talk about • Super fast storage (non-volatile memory) & file systems • What is hard about adding value in the file system • Technically • Socially/organizationally • Things we actually have to talk about • Context Agenda • ReFS v1 primer • ReFS v2 at a glance • Motivations for v2 • Cloning • Translation • Transformation ReFS v1 primer • Windows allocate-on-write file system • A lot of Windows compatibility • Merkel trees verify metadata integrity • Data integrity verification optional • Online data correction from alternate copies • Online chkdsk (AKA salvage AKA fsck) • Gets corruptions out of the namespace quickly ReFS v2 intro • Available in Windows Server Technical Preview 4 • Efficient, reliable storage for VMs: fast provisioning, fast diff merging, & tiering • Efficient erasure encoding / parity in mainline storage • Write tiering in the data path • Automatically redirect data to fastest tier • Data spills efficiently to slower tiers • Read caching • Block cloning • End-to-end optimizations for virtualization & more • File system-y optimizations • Redo log (for durable AKA O_SYNC/O_DSYNC/FUA/write-through) • B+ tree layout optimizations • Substantially more parallel • “Sparse VDL” – efficient uninitialized data tracking • Efficient handling of 4KB IO Why v2: motivations • Cheaper storage, but not
    [Show full text]
  • End-User License Agreement for Microsoft Exfat/NTFS for USB by Paragon Software
    End-user License Agreement for Microsoft exFAT/NTFS for USB by Paragon Software This End-User License Agreement (‘EULA’) is a legally valid contract between the end user of the software and Paragon Software GmbH, Wiesentalstraße 22, D-79115 Freiburg (hereinafter referred to as «Paragon»). Please read this Agreement thoroughly before acquiring the software. Upon acquisition of the software, the following provisions will be deemed to have been agreed to with binding effect. 1. Copyright This software product, including any handbooks, instructions and/or other information material, is protected by copyright. Any copy protection present in the software, a copyright notice, a registration number recorded in it or other features serving to identify the program may not be removed. 2. License agreement Unless determined otherwise, Paragon grants the software’s end user the simple right to install the software on devices and use it for an unlimited period of time. The right to use is limited to the software’s object code. It will expire if the end user violates the conditions of use established in this Agreement. Paragon is not obligated to provide the end user with the source code. Unless determined otherwise in the following, the acquisition of this license does not entitle the end user to create and/or distribute copies of the software, transfer the software from one device to another by electronic means or over a network after its original download or installation on a device, modify, decompile, adapt or translate the software or combine it with other software, or reengineer or disassembly the software.
    [Show full text]
  • This Video Looks at the Four File Systems Supported by Windows
    This video looks at the four file systems supported by Windows. These are ReFS, NTFS, FAT and exFAT. The video looks at what each file system is capable of and its limitations. Copyright 2014 © http://ITFreeTraining.com Resilient File System (ReFS) The Resilient File System is a new file system built from scratch by Microsoft. Since it is a new file system it requires Windows 8 or Windows Server 2012 in order to operate. The main design difference between it and previous operating systems is that it is designed to fix problems while the operating system is online. For this reason the check disk feature that is found in previous operating systems that can be run to fix problems no longer exists. Given a new approach has been taken in the operating system, it is better at ensuring data integrity and corruption than previous operating systems. Copyright 2014 © http://ITFreeTraining.com ReFS Limitations ReFS was designed to replace NTFS, but at the present time there are some limitations which may mean that you will need to stay with NTFS. Disk quotas: Disk quotes are not supported. Microsoft states in a blog post that this is a feature that can be supported outside the file system so it is possible for this feature to be supported in software. Possibly Microsoft will add this feature later on or some 3rd party software is available that will add this feature. NTFS compression and EFS: File compression and encryption (Encrypting File System) are not supported. Hard links: Hard links are not supported which is required by data duplication.
    [Show full text]