Lab 7C: Using the BASH Shell
Total Page:16
File Type:pdf, Size:1020Kb

Load more
Recommended publications
-
GNU CPIO GNU Cpio 2.5 June 2002
GNU CPIO GNU cpio 2.5 June 2002 by Robert Carleton Copyright c 1995, 2001, 2002 Free Software Foundation, Inc. This is the first edition of the GNU cpio documentation, and is consistent with GNU cpio 2.5. Published by the Free Software Foundation 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA Permission is granted to make and distribute verbatim copies of this manual provided the copyright notice and this permission notice are preserved on all copies. Permission is granted to copy and distribute modified versions of this manual under the con- ditions for verbatim copying, provided that the entire resulting derived work is distributed under the terms of a permission notice identical to this one. Permission is granted to copy and distribute translations of this manual into another lan- guage, under the above conditions for modified versions, except that this permission notice may be stated in a translation approved by the Free Software Foundation. Chapter 2: Tutorial 1 1 Introduction GNU cpio copies files into or out of a cpio or tar archive, The archive can be another file on the disk, a magnetic tape, or a pipe. GNU cpio supports the following archive formats: binary, old ASCII, new ASCII, crc, HPUX binary, HPUX old ASCII, old tar, and POSIX.1 tar. The tar format is provided for compatability with the tar program. By default, cpio creates binary format archives, for compatibility with older cpio programs. When extracting from archives, cpio automatically recognizes which kind of archive it is reading and can read archives created on machines with a different byte-order. -
Real Time Operating Systems Rootfs Creation: Summing Up
Real Time Operating Systems RootFS Creation: Summing Up Luca Abeni Real Time Operating Systems – p. System Boot System boot → the CPU starts executing from a well-known address ROM address: BIOS → read the first sector on the boot device, and executes it Bootloader (GRUB, LILO, U-Boot, . .) In general, load a kernel and an “intial ram disk” The initial fs image isn’t always needed (example: netboot) Kernel: from arm-test-*.tar.gz Initial filesystem? Loaded in RAM without the kernel help Generally contains the boot scripts and binaries Real Time Operating Systems – p. Initial Filesystem Old (2.4) kernels: Init Ram Disk (initrd); New (2.6) kernels: Init Ram Filesystem (initramfs) Generally used for modularized disk and FS drivers Example: if IDE drivers and Ext2 FS are modules (not inside the kernel), how can the kernel load them from disk? Solution: boot drivers can be on initrd / initramfs The bootloader loads it from disk with the kernel The kernel creates a “fake” fs based on it Modules are loaded from it Embedded systems can use initial FS for all the binaries Qemu does not need a bootloader to load kernel and initial FS (-kernel and -initrd) Real Time Operating Systems – p. Init Ram Filesystem Used in 2.6 kernels It is only a RAM FS: no real filesystem metadata on a storage medium All the files that must populate the FS are stored in a cpio package (similar to tar or zip file) The bootloader loads the cpio file in ram At boot time, the kernel “uncompresses” it, creating the RAM FS, and populating it with the files contained in the archive The cpio archive can be created by using the cpio -o -H newc command (see man cpio) Full command line: find . -
Lossless Compression of Internal Files in Parallel Reservoir Simulation
Lossless Compression of Internal Files in Parallel Reservoir Simulation Suha Kayum Marcin Rogowski Florian Mannuss 9/26/2019 Outline • I/O Challenges in Reservoir Simulation • Evaluation of Compression Algorithms on Reservoir Simulation Data • Real-world application - Constraints - Algorithm - Results • Conclusions 2 Challenge Reservoir simulation 1 3 Reservoir Simulation • Largest field in the world are represented as 50 million – 1 billion grid block models • Each runs takes hours on 500-5000 cores • Calibrating the model requires 100s of runs and sophisticated methods • “History matched” model is only a beginning 4 Files in Reservoir Simulation • Internal Files • Input / Output Files - Interact with pre- & post-processing tools Date Restart/Checkpoint Files 5 Reservoir Simulation in Saudi Aramco • 100’000+ simulations annually • The largest simulation of 10 billion cells • Currently multiple machines in TOP500 • Petabytes of storage required 600x • Resources are Finite • File Compression is one solution 50x 6 Compression algorithm evaluation 2 7 Compression ratio Tested a number of algorithms on a GRID restart file for two models 4 - Model A – 77.3 million active grid blocks 3.5 - Model K – 8.7 million active grid blocks 3 - 15.6 GB and 7.2 GB respectively 2.5 2 Compression ratio is between 1.5 1 compression ratio compression - From 2.27 for snappy (Model A) 0.5 0 - Up to 3.5 for bzip2 -9 (Model K) Model A Model K lz4 snappy gzip -1 gzip -9 bzip2 -1 bzip2 -9 8 Compression speed • LZ4 and Snappy significantly outperformed other algorithms -
The Ark Handbook
The Ark Handbook Matt Johnston Henrique Pinto Ragnar Thomsen The Ark Handbook 2 Contents 1 Introduction 5 2 Using Ark 6 2.1 Opening Archives . .6 2.1.1 Archive Operations . .6 2.1.2 Archive Comments . .6 2.2 Working with Files . .7 2.2.1 Editing Files . .7 2.3 Extracting Files . .7 2.3.1 The Extract dialog . .8 2.4 Creating Archives and Adding Files . .8 2.4.1 Compression . .9 2.4.2 Password Protection . .9 2.4.3 Multi-volume Archive . 10 3 Using Ark in the Filemanager 11 4 Advanced Batch Mode 12 5 Credits and License 13 Abstract Ark is an archive manager by KDE. The Ark Handbook Chapter 1 Introduction Ark is a program for viewing, extracting, creating and modifying archives. Ark can handle vari- ous archive formats such as tar, gzip, bzip2, zip, rar, 7zip, xz, rpm, cab, deb, xar and AppImage (support for certain archive formats depends on the appropriate command-line programs being installed). In order to successfully use Ark, you need KDE Frameworks 5. The library libarchive version 3.1 or above is needed to handle most archive types, including tar, compressed tar, rpm, deb and cab archives. To handle other file formats, you need the appropriate command line programs, such as zipinfo, zip, unzip, rar, unrar, 7z, lsar, unar and lrzip. 5 The Ark Handbook Chapter 2 Using Ark 2.1 Opening Archives To open an archive in Ark, choose Open... (Ctrl+O) from the Archive menu. You can also open archive files by dragging and dropping from Dolphin. -
Bzip2 and Libbzip2, Version 1.0.8 a Program and Library for Data Compression
bzip2 and libbzip2, version 1.0.8 A program and library for data compression Julian Seward, https://sourceware.org/bzip2/ bzip2 and libbzip2, version 1.0.8: A program and library for data compression by Julian Seward Version 1.0.8 of 13 July 2019 Copyright© 1996-2019 Julian Seward This program, bzip2, the associated library libbzip2, and all documentation, are copyright © 1996-2019 Julian Seward. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: • Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. • The origin of this software must not be misrepresented; you must not claim that you wrote the original software. If you use this software in a product, an acknowledgment in the product documentation would be appreciated but is not required. • Altered source versions must be plainly marked as such, and must not be misrepresented as being the original software. • The name of the author may not be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE AUTHOR "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DIS- CLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -
Working with Compressed Archives
Working with compressed archives Archiving a collection of files or folders means creating a single file that groups together those files and directories. Archiving does not manipulate the size of files. They are added to the archive as they are. Compressing a file means shrinking the size of the file. There are software for archiving only, other for compressing only and (as you would expect) other that have the ability to archive and compress. This document will show you how to use a windows application called 7-zip (seven zip) to work with compressed archives. 7-zip integrates in the context menu that pops-up up whenever you right-click your mouse on a selection1. In this how-to we will look in the application interface and how we manipulate archives, their compression and contents. 1. Click the 'start' button and type '7-zip' 2. When the search brings up '7-zip File Manager' (as the best match) press 'Enter' 3. The application will startup and you will be presented with the 7-zip manager interface 4. The program integrates both archiving, de/compression and file-management operations. a. The main area of the window provides a list of the files of the active directory. When not viewing the contents of an archive, the application acts like a file browser window. You can open folders to see their contents by just double-clicking on them b. Above the main area you have an address-like bar showing the name of the active directory (currently c:\ADG.Becom). You can use the 'back' icon on the far left to navigate back from where you are and go one directory up. -
Z/VM Version 7 Release 2
z/VM Version 7 Release 2 OpenExtensions User's Guide IBM SC24-6299-01 Note: Before you use this information and the product it supports, read the information in “Notices” on page 201. This edition applies to Version 7.2 of IBM z/VM (product number 5741-A09) and to all subsequent releases and modifications until otherwise indicated in new editions. Last updated: 2020-09-08 © Copyright International Business Machines Corporation 1993, 2020. US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Contents Figures................................................................................................................. xi Tables................................................................................................................ xiii About this Document........................................................................................... xv Intended Audience..................................................................................................................................... xv Conventions Used in This Document......................................................................................................... xv Escape Character Notation................................................................................................................... xv Case-Sensitivity.....................................................................................................................................xv Typography............................................................................................................................................xv -
Gzip, Bzip2 and Tar EXPERT PACKING
LINUXUSER Command Line: gzip, bzip2, tar gzip, bzip2 and tar EXPERT PACKING A short command is all it takes to pack your data or extract it from an archive. BY HEIKE JURZIK rchiving provides many bene- fits: packed and compressed Afiles occupy less space on your disk and require less bandwidth on the Internet. Linux has both GUI-based pro- grams, such as File Roller or Ark, and www.sxc.hu command-line tools for creating and un- packing various archive types. This arti- cle examines some shell tools for ar- chiving files and demonstrates the kind of expert packing that clever combina- tained by the packing process. If you A gzip file can be unpacked using either tions of Linux commands offer the com- prefer to use a different extension, you gunzip or gzip -d. If the tool discovers a mand line user. can set the -S (suffix) flag to specify your file of the same name in the working di- own instead. For example, the command rectory, it prompts you to make sure that Nicely Packed with “gzip” you know you are overwriting this file: The gzip (GNU Zip) program is the de- gzip -S .z image.bmp fault packer on Linux. Gzip compresses $ gunzip screenie.jpg.gz simple files, but it does not create com- creates a compressed file titled image. gunzip: screenie.jpg U plete directory archives. In its simplest bmp.z. form, the gzip command looks like this: The size of the compressed file de- Listing 1: Compression pends on the distribution of identical Compared gzip file strings in the original file. -
Deduplicating Compressed Contents in Cloud Storage Environment
Deduplicating Compressed Contents in Cloud Storage Environment Zhichao Yan, Hong Jiang Yujuan Tan* Hao Luo University of Texas Arlington Chongqing University University of Nebraska Lincoln [email protected] [email protected] [email protected] [email protected] Corresponding Author Abstract Data compression and deduplication are two common approaches to increasing storage efficiency in the cloud environment. Both users and cloud service providers have economic incentives to compress their data before storing it in the cloud. However, our analysis indicates that compressed packages of different data and differ- ently compressed packages of the same data are usual- ly fundamentally different from one another even when they share a large amount of redundant data. Existing data deduplication systems cannot detect redundant data among them. We propose the X-Ray Dedup approach to extract from these packages the unique metadata, such as the “checksum” and “file length” information, and use it as the compressed file’s content signature to help detect and remove file level data redundancy. X-Ray Dedup is shown by our evaluations to be capable of breaking in the boundaries of compressed packages and significantly Figure 1: A user scenario on cloud storage environment reducing compressed packages’ size requirements, thus further optimizing storage space in the cloud. will generate different compressed data of the same con- tents that render fingerprint-based redundancy identifi- cation difficult. Third, very similar but different digital 1 Introduction contents (e.g., files or data streams), which would other- wise present excellent deduplication opportunities, will Due to the information explosion [1, 3], data reduc- become fundamentally distinct compressed packages af- tion technologies such as compression and deduplica- ter applying even the same compression algorithm. -
Is It Time to Replace Gzip?
bioRxiv preprint doi: https://doi.org/10.1101/642553; this version posted May 20, 2019. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY 4.0 International license. Is it time to replace gzip? Comparison of modern compressors for molecular sequence databases Kirill Kryukov*, Mahoko Takahashi Ueda, So Nakagawa, Tadashi Imanishi Department of Molecular Life Science, Tokai University School of Medicine, Isehara, Kanagawa 259-1193, Japan. *Correspondence: [email protected] Abstract Nearly all molecular sequence databases currently use gzip for data compression. Ongoing rapid accumulation of stored data calls for more efficient compression tool. We systematically benchmarked the available compressors on representative DNA, RNA and Protein datasets. We tested specialized sequence compressors 2bit, BLAST, DNA-COMPACT, DELIMINATE, Leon, MFCompress, NAF, UHT and XM, and general-purpose compressors brotli, bzip2, gzip, lz4, lzop, lzturbo, pbzip2, pigz, snzip, xz, zpaq and zstd. Overall, NAF and zstd performed well in terms of transfer/decompression speed. However, checking benchmark results is necessary when choosing compressor for specific data type and application. Benchmark results database is available at: http://kirr.dyndns.org/sequence-compression-benchmark/. Keywords: compression; benchmark; DNA; RNA; protein; genome; sequence; database. Molecular sequence databases store and distribute DNA, RNA and protein sequences as compressed FASTA files. Currently, nearly all databases universally depend on gzip for compression. This incredible longevity of the 26-year-old compressor probably owes to multiple factors, including conservatism of database operators, wide availability of gzip, and its generally acceptable performance. -
Software Development Kits Comparison Matrix
Software Development Kits > Comparison Matrix Software Development Kits Comparison Matrix DATA COMPRESSION TOOLKIT FOR TOOLKIT FOR TOOLKIT FOR LIBRARY (DCL) WINDOWS WINDOWS JAVA Compression Method DEFLATE DEFLATE64 DCL Implode BZIP2 LZMA PPMd Extraction Method DEFLATE DEFLATE64 DCL Implode BZIP2 LZMA PPMd Wavpack Creation Formats Supported ZIP BZIP2 GZIP TAR UUENCODE, XXENCODE Large Archive Creation Support (ZIP64) Archive Extraction Formats Supported ZIP BZIP2 GZIP, TAR, AND .Z (UNIX COMPRESS) UUENCODE, XXENCODE CAB, BinHex, ARJ, LZH, RAR, 7z, and ISO/DVD Integration Format Static Dynamic Link Libraries (DLL) Pure Java Class Library (JAR) Programming Interface C C/C++ Objects C#, .NET interface Java Support for X.509-Compliant Digital Certificates Apply digital signatures Verify digital signatures Encrypt archives www.pkware.com Software Development Kits > Comparison Matrix DATA COMPRESSION TOOLKIT FOR TOOLKIT FOR TOOLKIT FOR LIBRARY (DCL) WINDOWS WINDOWS JAVA Signing Methods SHA-1 MD5 SHA-256, SHA-384 and SHA-512 Message Digest Timestamp .ZIP file signatures ENTERPRISE ONLY Encryption Formats ZIP OpenPGP ENTERPRISE ONLY Encryption Methods AES (256, 192, 128-bit) 3DES (168, 112-bit) 168 ONLY DES (56-bit) RC4 (128, 64, 40-bit), RC2 (128, 64, 40-bit) CAST5 and IDEA Decryption Methods AES (256, 192, 128-bit) 3DES (168, 112-bit) 168 ONLY DES (56-bit) RC4 (128, 64, 40-bit), RC2 (128, 64, 40-bit) CAST5 and IDEA Additional Features Supports file, memory and streaming operations Immediate archive operation Deferred archive operation Multi-Thread -
Backing up Data
Backing up data On LINUX and UNIX Logical backup • Why? Job security • For user data. • Remember endian order, character set issues. • Full backup + incremental = backup set No differential (see find command). • Backup without a restore test is just a tape • mt or rmt used for generic tape management. See mt, rmt manpage. See /dev/rmt0 • See text for CD/DVD procedures • Use commercial products for business. Why? Tape catalog management. Backup Commands • cp (duh) • ftp (eh) • rcp (no, no, no) • scp • rsync • tar • cpio • pax (see cpio, tar) • dump/restore – the standard The usual suspects • cp –rp source destination • ftp hostname user name cd source or destination lcd destination/source put/get filename quit • scp source user@destination:/pathtofile scp user@source:/pathtofile /destination • rcp source user@destination:/pathtofile rcp user@ source:/pathtofile /destination uses .rhosts see manpage on hosts.equiv rsync • Support from rsync.samba.org • rsync copies files either to or from a remote host, or locally on the current host but does not support copying files between two remote hosts. • rsync can reduce the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. • There are two different ways for rsync to contact a remote system: using a remote- shell program as the transport (such as ssh or rsh) or contacting an rsync daemon directly via TCP. See the manpage on rsyncd.conf. • rsync –avh /source /destination • rsync -avze ssh /home/user/directory/ user@remotehost:home/user/directory/ • Other Options -a, --archive archive mode; equals -r, --recursive recurse into directories -u, --update skip files that are newer on the receiver Tape ARchive • Most portable backup utility between systems.