Petascale Data Management: Guided by Measurement
Total Page:16
File Type:pdf, Size:1020Kb
Load more
Recommended publications
-
Advfs Command Line and Application Programming Interface
AdvFS Command Line and Application Programming Interface External Reference Specification Version 1.13 JA CASL Not Inspected/Date Inspected Building ZK3 110 Spit Brook Road Nashua, NH 03062 Copyright (C) 2008 Hewlett-Packard Development Company, L.P. Page 1 of 65 Preface Version 1.4 of the AdvFS Command Line and API External Reference Specification is being made available to all partners in order to allow them to design and implement code meeting the specifications contained herein. 1 INTRODUCTION ....................................................................................................................................................................4 1.1 ABSTRACT .........................................................................................................................................................................4 1.2 AUDIENCE ..........................................................................................................................................................................4 1.3 TERMS AND DEFINITIONS ...................................................................................................................................................4 1.4 RELATED DOCUMENTS ......................................................................................................................................................4 1.5 ACKNOWLEDGEMENTS ......................................................................................................................................................4 -
Hierarchical Storage Management with SAM-FS
Hierarchical Storage Management with SAM-FS Achim Neumann CeBiTec / Bielefeld University Agenda ● Information Lifecycle Management ● Backup – Technology of the past ● Filesystems, SAM-FS, QFS and SAM-QFS ● Main Functions of SAM ● SAM and Backup ● Rating, Perspective and Literature June 13, 2006 Achim Neumann 2 ILM – Information Lifecylce Management ● 5 Exabyte data per year (92 % digital, 8 % analog) ● 20% fixed media (disk), 80% removable media ● Managing data from Creation to Cremation (and everything that is between (by courtesy of Sun Microsystems Inc). June 13, 2006 Achim Neumann 3 Different classes of Storagesystems (by courtesy of Sun Microsystems Inc). June 13, 2006 Achim Neumann 4 Backup – Technology of the past ● Backup – protect data from data loss ● Backup is not the problem ● Ability to restore data when needed − Disaster − Hardware failure − User error − Old versions from a file June 13, 2006 Achim Neumann 5 Backup – Technology of the past (cont.) ● Triple data in 2 – 3 years ● Smaller backup window ☞ more data in less time Stream for # Drives for # Drives for #TB 24 h 24 h 6 h 1 12,1 MB/s 2 5 10 121 MB/s 13 49 30 364 MB/s 37 146 June 13, 2006 Achim Neumann 6 Backup – Technology of the past (cont.) ● Limited number of tape drives in library ● Limited number of I/O Channels in server ● Limited scalability of backup software ● Inability to keep all tape drives streaming at one time ● At least every full backup has a copy of the same file ● It's not unusual to have 8 or 10 copies of the same file ☞ excessive media usage and -
Oracle Database Administrator's Reference for UNIX-Based Operating Systems
Oracle® Database Administrator’s Reference 10g Release 2 (10.2) for UNIX-Based Operating Systems B15658-06 March 2009 Oracle Database Administrator's Reference, 10g Release 2 (10.2) for UNIX-Based Operating Systems B15658-06 Copyright © 2006, 2009, Oracle and/or its affiliates. All rights reserved. Primary Author: Brintha Bennet Contributing Authors: Kevin Flood, Pat Huey, Clara Jaeckel, Emily Murphy, Terri Winters, Ashmita Bose Contributors: David Austin, Subhranshu Banerjee, Mark Bauer, Robert Chang, Jonathan Creighton, Sudip Datta, Padmanabhan Ganapathy, Thirumaleshwara Hasandka, Joel Kallman, George Kotsovolos, Richard Long, Rolly Lv, Padmanabhan Manavazhi, Matthew Mckerley, Sreejith Minnanghat, Krishna Mohan, Rajendra Pingte, Hanlin Qian, Janelle Simmons, Roy Swonger, Lyju Vadassery, Douglas Williams This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited. The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing. If this software or related documentation is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, the following notice is applicable: U.S. GOVERNMENT RIGHTS Programs, software, databases, and related documentation and technical data delivered to U.S. -
Kratka Povijest Unixa Od Unicsa Do Freebsda I Linuxa
Kratka povijest UNIXa Od UNICSa do FreeBSDa i Linuxa 1 Autor: Hrvoje Horvat Naslov: Kratka povijest UNIXa - Od UNICSa do FreeBSDa i Linuxa Licenca i prava korištenja: Svi imaju pravo koristiti, mijenjati, kopirati i štampati (printati) knjigu, prema pravilima GNU GPL licence. Mjesto i godina izdavanja: Osijek, 2017 ISBN: 978-953-59438-0-8 (PDF-online) URL publikacije (PDF): https://www.opensource-osijek.org/knjige/Kratka povijest UNIXa - Od UNICSa do FreeBSDa i Linuxa.pdf ISBN: 978-953- 59438-1- 5 (HTML-online) DokuWiki URL (HTML): https://www.opensource-osijek.org/dokuwiki/wiki:knjige:kratka-povijest- unixa Verzija publikacije : 1.0 Nakalada : Vlastita naklada Uz pravo svakoga na vlastito štampanje (printanje), prema pravilima GNU GPL licence. Ova knjiga je napisana unutar inicijative Open Source Osijek: https://www.opensource-osijek.org Inicijativa Open Source Osijek je član udruge Osijek Software City: http://softwarecity.hr/ UNIX je registrirano i zaštićeno ime od strane tvrtke X/Open (Open Group). FreeBSD i FreeBSD logo su registrirani i zaštićeni od strane FreeBSD Foundation. Imena i logo : Apple, Mac, Macintosh, iOS i Mac OS su registrirani i zaštićeni od strane tvrtke Apple Computer. Ime i logo IBM i AIX su registrirani i zaštićeni od strane tvrtke International Business Machines Corporation. IEEE, POSIX i 802 registrirani i zaštićeni od strane instituta Institute of Electrical and Electronics Engineers. Ime Linux je registrirano i zaštićeno od strane Linusa Torvaldsa u Sjedinjenim Američkim Državama. Ime i logo : Sun, Sun Microsystems, SunOS, Solaris i Java su registrirani i zaštićeni od strane tvrtke Sun Microsystems, sada u vlasništvu tvrtke Oracle. Ime i logo Oracle su u vlasništvu tvrtke Oracle. -
LSF ’07: 2007 Linux Storage & Filesystem Workshop Namically Adjusted According to the Current Popularity of Its Hot Zone
June07login1summaries_press.qxd:login summaries 5/27/07 10:27 AM Page 84 PRO: A Popularity-Based Multi-Threaded Reconstruction overhead of PRO is O(n), although if a priority queue is Optimization for RAID-Structured Storage Systems used in the PRO algorithm the computation overhead can Lei Tian and Dan Feng, Huazhong University of Science and be reduced to O(log n). The entire PRO implementation in Technology; Hong Jiang, University of Nebraska—Lincoln; Ke the RAIDFrame software only added 686 lines of code. Zhou, Lingfang Zeng, Jianxi Chen, and Zhikun Wang, Hua- Work on PRO is ongoing. Future work includes optimiz - zhong University of Science and Technology and Wuhan ing the time slice, scheduling strategies, and hot zone National Laboratory for Optoelectronics; Zhenlei Song, length. Currently, PRO is being ported into the Linux soft - Huazhong University of Science and Technology ware RAID. Finally, the authors plan on further investigat - ing use of access patterns to help predict user accesses and Hong Jiang began his talk by discussing the importance of of filesystem semantic knowledge to explore accurate re - data recovery. Disk failures have become more common in construction. RAID-structured storage systems. The improvement in disk capacity has far outpaced improvements in disk band - The first questioner asked about the average rate of recov - width, lengthening the overall RAID recovery time. Also, ery for PRO. Hong answered that the average reconstruc - disk drive reliability has improved slowly, resulting in a tion time is several hundred seconds in the experimental very high overall failure rate in a large-scale RAID storage setup. -
Performance and Scalability of a Sensor Data Storage Framework
View metadata, citation and similar papers at core.ac.uk brought to you by CORE provided by Aaltodoc Publication Archive Aalto University School of Science Degree Programme in Computer Science and Engineering Paul Tötterman Performance and Scalability of a Sensor Data Storage Framework Master’s Thesis Helsinki, March 10, 2015 Supervisor: Associate Professor Keijo Heljanko Advisor: Lasse Rasinen M.Sc. (Tech.) Aalto University School of Science ABSTRACT OF Degree Programme in Computer Science and Engineering MASTER’S THESIS Author: Paul Tötterman Title: Performance and Scalability of a Sensor Data Storage Framework Date: March 10, 2015 Pages: 67 Major: Software Technology Code: T-110 Supervisor: Associate Professor Keijo Heljanko Advisor: Lasse Rasinen M.Sc. (Tech.) Modern artificial intelligence and machine learning applications build on analysis and training using large datasets. New research and development does not always start with existing big datasets, but accumulate data over time. The same storage solution does not necessarily cover the scale during the lifetime of the research, especially if scaling up from using common workgroup storage technologies. The storage infrastructure at ZenRobotics has grown using standard workgroup technologies. The current approach is starting to show its limits, while the storage growth is predicted to continue and accelerate. Successful capacity planning and expansion requires a better understanding of the patterns of the use of storage and its growth. We have examined the current storage architecture and stored data from different perspectives in order to gain a better understanding of the situation. By performing a number of experiments we determine key properties of the employed technologies. The combination of these factors allows us to make informed decisions about future storage solutions. -
Lustrefs and Its Ongoing Evolution for High Performance Computing and Data Analysis Solutions
LustreFS and its ongoing Evolution for High Performance Computing and Data Analysis Solutions Roger Goff Senior Product Manager DataDirect Networks, Inc. 2014 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved. What is Lustre? Parallel/shared file system for clusters Aggregates many systems into one file system Hugely scalable - 100s of GB/s throughput, 10s of thousands of native clients Scales I/O throughput and capacity Widely adopted in HPC community Open source, community driven development Supports RDMA over high speed low latency interconnects and Ethernet Runs on commodity storage and purpose built HPC storage platforms Abstracts hardware view of storage from software Highly available 2 2014 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved. Lustre Node Types Management Servers (MGS) Object Storage Servers (OSS) Meta-Data Servers (MDS) Clients LNET Routers 3 2014 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved. Lustre MGS ManaGement Server Server node which manages cluster configuration database All clients, OSS and MDS need to know how to contact the MGS MGS provides all other components with information about the Lustre file system(s) it manages MGS can be separated from or co-located with MDS 4 2014 Storage Developer Conference. © Insert Your Company Name. All Rights Reserved. Lustre MDS Meta Data Server Manages the names and directories in the Lustre file system Stores metadata (such as filenames, directories, permissions and file layout) on a MetaData Target (MDT) Usually MDSs are configured in an active/passive failover pair Lustre inode ext3 inode 5 2014 Storage Developer Conference. © Insert Your Company Name. -
Guido Van Rossum on PYTHON 3 Get Your Sleep From
JavaScript | Inform 6 & 7 | Falcon | Sleep | Enlightenment | PHP LINUX JOURNAL ™ THE STATE OF LINUX AUDIO SOFTWARE LANGUAGES Since 1994: The Original Magazine of the Linux Community OCTOBER 2008 | ISSUE 174 Inform 7 REVIEWED JavaScript | Inform 6 & 7 | Falcon | Sleep Enlightenment PHP Audio | Inform 6 & 7 Falcon JavaScript Don’t Get Eaten HP Media by a Grue! Vault 5150 Scalent’s Managing Virtual Operating PHP Code Environment Guido van Rossum on PYTHON 3 Get Your Sleep from OCTOBER Java www.linuxjournal.com 2008 $5.99US $5.99CAN 10 ISSUE Martin Messner Enlightenment E17 Insights from SUSE’s Lightweight Alternative 174 + Security Team Lead to KDE and GNOME 0 09281 03102 4 MULTIPLY ENERGY EFFICIENCY AND MAXIMIZE COOLING. THE WORLD’S FIRST QUAD-CORE PROCESSOR FOR MAINSTREAM SERVERS. THE NEW QUAD-CORE INTEL® XEON® PROCESSOR 5300 SERIES DELIVERS UP TO 50% 1 MORE PERFORMANCE*PERFORMANCE THAN PREVIOUS INTEL XEON PROCESSORS IN THE SAME POWERPOWER ENVELOPE.ENVELOPE. BASEDBASED ONON THETHE ULTRA-EFFICIENTULTRA-EFFICIENT INTEL®INTEL® CORE™CORE™ MICROMICROARCHITECTURE, ARCHITECTURE IT’S THE ULTIMATE SOLUTION FOR MANAGING RUNAWAY COOLING EXPENSES. LEARN WHYWHY GREAT GREAT BUSINESS BUSINESS COMPUTING COMPUTING STARTS STARTS WITH WITH INTEL INTEL INSIDE. INSIDE. VISIT VISIT INTEL.CO.UK/XEON INTEL.COM/XEON. RELION 2612 s 1UAD #ORE)NTEL®8EON® RELION 1670 s 1UAD #ORE)NTEL®8EON® PROCESSOR PROCESSOR s 5SERVERWITHUPTO4" s )NTEL@3EABURG CHIPSET s )DEALFORCOST EFFECTIVE&ILE$" WITH-(ZFRONTSIDEBUS APPLICATIONS s 5PTO'"2!-IN5CLASS s 2!32ELIABILITY !VAILABILITY LEADINGMEMORYCAPACITY 3ERVICEABILITY s -ANAGEMENTFEATURESTOSUPPORT LARGECLUSTERDEPLOYMENTS 34!24).'!4$2429.00 34!24).'!4$1969.00 Penguin Computing provides turnkey x86/Linux clusters for high performance technical computing applications. -
Sun Storedge™ QFS and Sun Storedge SAM-FS Software Installation and Configuration Guide
Sun StorEdge™ QFS and Sun StorEdge SAM-FS Software Installation and Configuration Guide Release 4.2 Sun Microsystems, Inc. www.sun.com Part No. 817-7722-10 September 2004, Revision A Submit comments about this document at: http://www.sun.com/hwdocs/feedback Copyright 2004 Sun Microsystems, Inc., 4150 Network Circle, Santa Clara, California 95054, U.S.A. All rights reserved. Sun Microsystems, Inc. has intellectual property rights relating to technology that is described in this document. In particular, and without limitation, these intellectual property rights may include one or more of the U.S. patents listed at http://www.sun.com/patents and one or more additional patents or pending patent applications in the U.S. and in other countries. This document and the product to which it pertains are distributed under licenses restricting their use, copying, distribution, and decompilation. No part of the product or of this document may be reproduced in any form by any means without prior written authorization of Sun and its licensors, if any. Third-party software, including font technology, is copyrighted and licensed from Sun suppliers. Parts of the product may be derived from Berkeley BSD systems, licensed from the University of California. UNIX is a registered trademark in the U.S. and in other countries, exclusively licensed through X/Open Company, Ltd. Sun, Sun Microsystems, the Sun logo, AnswerBook2, docs.sun.com, Solaris, SunOS, SunSolve, Java, JavaScript, Solstice DiskSuite, and StorEdge are trademarks or registered trademarks of Sun Microsystems, Inc. in the U.S. and in other countries. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. -
Ted Ts'o on Linux File Systems
Ted Ts’o on Linux File Systems An Interview RIK FARROW Rik Farrow is the Editor of ;login:. ran into Ted Ts’o during a tutorial luncheon at LISA ’12, and that later [email protected] sparked an email discussion. I started by asking Ted questions that had I puzzled me about the early history of ext2 having to do with the perfor- mance of ext2 compared to the BSD Fast File System (FFS). I had met Rob Kolstad, then president of BSDi, because of my interest in the AT&T lawsuit against the University of California and BSDi. BSDi was being sued for, among other things, Theodore Ts’o is the first having a phone number that could be spelled 800-ITS-UNIX. I thought that it was important North American Linux for the future of open source operating systems that AT&T lose that lawsuit. Kernel Developer, having That said, when I compared the performance of early versions of Linux to the current version started working with Linux of BSDi, I found that they were closely matched, with one glaring exception. Unpacking tar in September 1991. He also archives using Linux (likely .9) was blazingly fast compared to BSDi. I asked Rob, and he served as the tech lead for the MIT Kerberos explained that the issue had to do with synchronous writes, finally clearing up a mystery for me. V5 development team, and was the architect at IBM in charge of bringing real-time Linux Now I had a chance to ask Ted about the story from the Linux side, as well as other questions in support of real-time Java to the US Navy. -
View the Slides
SMB3.1.1 POSIX Protocol Extensions: Summary and Current Implementation Status Steve French Azure Storage – Microsoft Samba Team And SMB Jeremy Allison Google/Samba Team 3.1.1 Legal Statement This work represents the views of the author(s) and does not necessarily reflect the views of Microsoft or Google Linux is a registered trademark of Linus Torvalds. Other company, product, and service names may be trademarks or service marks of others. Outline Linux is a lot more than POSIX ... Why do these extensions matter? Implementation Status What works today? Some details How to handle Linux continuing to extend APIs? Wireshark and Tracing Linux > POSIX Currently huge number of syscalls! (try “git grep SYSCALL_DEFINE” well over 850 and 500+ are even documented “man syscalls” FS layer has 223). Verified today vs Only about 100 POSIX API calls 513 syscalls with man pages! +12 just since last year’s SDC! Some examples of new fs ones from past 9 months ... Syscall name Kernel Version introduced io_uring_enter 5.1 io_uring_register 5.1 io_uring_setup 5.1 move_mount 5.2 open_tree 5.2 fsconfig 5.2 fsmount 5.2 fsopen 5.2 fspick 5.2 Repeating an old slide ... Remember LINUX > POSIX And not just new syscalls … new flags ... 2 examples of richer Linux vs. simpler POSIX fallocate has 7 flags – Insert range – Unshare range – Zero range – Keep size – But POSIX fallocate has no flags Rename (renameat2) has 3 flags – noreplace, whiteout and exchange – POSIX rename has none Network File systems matter ● these extensions to most popular network fs protocol (SMB3) are important ● block devices struggle to do file system tasks: locking, security, leases, consistent metadata Linux Apps need to work over network mounts and continue to work as Linux evolves Improve common situations where customers have Linux and Windows and Mac clients Make sure extensions work with most secure, most optimal SMB3.1.1 dialect (don’t encourage less secure network file systems, or even SMB1/CIFS) Quick Overview of Status ● Linux kernel client: – 5.1 kernel or later can be used. -
Creating a Hierarchical Database Backup System Using Oracle RMAN and Oracle SAM QFS with the Sun ZFS Storage Appliance
An Oracle White Paper August 2011 Creating a Hierarchical Database Backup System using Oracle RMAN and Oracle SAM QFS with the Sun ZFS Storage Appliance Creating a Hierarchical Database Backup System using Oracle RMAN and Oracle SAM QFS with the Sun ZFS Storage Appliance Introduction ......................................................................................... 2 Overview of the Hierarchical Database Backup Solution.................... 2 Physical Architecture of a Hierarchical Database Backup System ..... 3 Hierarchical Database Backup System Logical Architecture .............. 5 SAM Server Configuration .............................................................. 6 Configuring the Oracle RMAN Server............................................. 9 Oracle RMAN Configuration ......................................................... 10 Performance Considerations ............................................................ 11 Sizing Oracle RMAN Backups .......................................................... 12 Additional Considerations ................................................................. 14 Conclusion ........................................................................................ 15 Appendix A. Reference Scripts and Configurations.......................... 16 Appendix B. Solution Component Features...................................... 19 Oracle Recovery Manager ............................................................ 19 Sun Quick File System and Storage Archive Manager................. 20 Sun ZFS