Prac4: Unix System Internals
Total Page:16
File Type:pdf, Size:1020Kb
Load more
Recommended publications
-
Web Vmstat Any Distros, Especially Here’S Where Web Vmstat Comes Those Targeted at In
FOSSPICKS Sparkling gems and new releases from the world of FOSSpicks Free and Open Source Software Mike Saunders has spent a decade mining the internet for free software treasures. Here’s the result of his latest haul… Shiny statistics in a browser Web VMStat any distros, especially Here’s where Web VMStat comes those targeted at in. It’s a system monitor that runs Madvanced users, ship an HTTP server, so you can connect with shiny system monitoring tools to it via a web browser and see on the desktop. Conky is one such fancy CSS-driven charts. Before you tool, while GKrellM was all the rage install it, you’ll need to get the in the last decade, and they are websocketd utility, which you can genuinely useful for keeping tabs find at https://github.com/ on your boxes, especially when joewalnes/websocketd. Helpfully, you’re an admin in charge of the developer has made pre- various servers. compiled executables available, so Now, pretty much all major you can just grab the 32-bit or distros include a useful command 64-bit tarball, extract it and there line tool for monitoring system you have it: websocketd. (Of course, Here’s the standard output for vmstat – not very interesting, right? resource usage: vmstat. Enter if you’re especially security vmstat 1 in a terminal window and conscious, you can compile it from copy the aforementioned you’ll see a regularly updating (once its source code.) websocketd into the same place. per second) bunch of statistics, Next, clone the Web VMStat Git Then just enter: showing CPU usage, free RAM, repository (or grab the Zip file and ./run swap usage and so forth. -
CSE 220: Systems Programming Input and Output
CSE 220: Systems Programming Input and Output Ethan Blanton Department of Computer Science and Engineering University at Buffalo Introduction Unix I/O Standard I/O Buffering Summary References I/O Kernel Services We have seen some text I/O using the C Standard Library. printf() fgetc() … However, all I/O is built on kernel system calls. In this lecture, we’ll look at those services vs. standard I/O. © 2020 Ethan Blanton / CSE 220: Systems Programming Introduction Unix I/O Standard I/O Buffering Summary References Everything is a File These services are particularly important on Unix systems. On Unix, “everything is a file”. Many devices and services are accessed by opening device nodes. Device nodes behave like (but are not) files. Examples: /dev/null: Always readable, contains no data. Always writable, discards anything written to it. /dev/urandom: Always readable, reads a cryptographically secure stream of random data. © 2020 Ethan Blanton / CSE 220: Systems Programming Introduction Unix I/O Standard I/O Buffering Summary References File Descriptors All access to files is through file descriptors. A file descriptor is a small integer representing an open file in a particular process. There are three “standard” file descriptors: 0: standard input 1: standard output 2: standard error …sound familiar? (stdin, stdout, stderr) © 2020 Ethan Blanton / CSE 220: Systems Programming Introduction Unix I/O Standard I/O Buffering Summary References System Call Failures Kernel I/O (and most other) system calls return -1 on failure. When this happens, the global variable errno is set to a reason. Include errno.h to define errno in your code. -
Introduction to Unix
Introduction to Unix Rob Funk <[email protected]> University Technology Services Workstation Support http://wks.uts.ohio-state.edu/ University Technology Services Course Objectives • basic background in Unix structure • knowledge of getting started • directory navigation and control • file maintenance and display commands • shells • Unix features • text processing University Technology Services Course Objectives Useful commands • working with files • system resources • printing • vi editor University Technology Services In the Introduction to UNIX document 3 • shell programming • Unix command summary tables • short Unix bibliography (also see web site) We will not, however, be covering these topics in the lecture. Numbers on slides indicate page number in book. University Technology Services History of Unix 7–8 1960s multics project (MIT, GE, AT&T) 1970s AT&T Bell Labs 1970s/80s UC Berkeley 1980s DOS imitated many Unix ideas Commercial Unix fragmentation GNU Project 1990s Linux now Unix is widespread and available from many sources, both free and commercial University Technology Services Unix Systems 7–8 SunOS/Solaris Sun Microsystems Digital Unix (Tru64) Digital/Compaq HP-UX Hewlett Packard Irix SGI UNICOS Cray NetBSD, FreeBSD UC Berkeley / the Net Linux Linus Torvalds / the Net University Technology Services Unix Philosophy • Multiuser / Multitasking • Toolbox approach • Flexibility / Freedom • Conciseness • Everything is a file • File system has places, processes have life • Designed by programmers for programmers University Technology Services -
Storage Devices, Basic File System Design
CS162 Operating Systems and Systems Programming Lecture 17 Data Storage to File Systems October 29, 2019 Prof. David Culler http://cs162.eecs.Berkeley.edu Read: A&D Ch 12, 13.1-3.2 Recall: OS Storage abstractions Key Unix I/O Design Concepts • Uniformity – everything is a file – file operations, device I/O, and interprocess communication through open, read/write, close – Allows simple composition of programs • find | grep | wc … • Open before use – Provides opportunity for access control and arbitration – Sets up the underlying machinery, i.e., data structures • Byte-oriented – Even if blocks are transferred, addressing is in bytes What’s below the surface ?? • Kernel buffered reads – Streaming and block devices looks the same, read blocks yielding processor to other task • Kernel buffered writes Application / Service – Completion of out-going transfer decoupled from the application, File descriptor number allowing it to continue - an int High Level I/O streams • Explicit close Low Level I/O handles 9/12/19 The cs162file fa19 L5system abstraction 53 Syscall registers • File File System descriptors File Descriptors – Named collection of data in a file system • a struct with all the info I/O Driver Commands and Data Transfers – POSIX File data: sequence of bytes • about the files Disks, Flash, Controllers, DMA Could be text, binary, serialized objects, … – File Metadata: information about the file • Size, Modification Time, Owner, Security info • Basis for access control • Directory – “Folder” containing files & Directories – Hierachical -
Performance, Scalability on the Server Side
Performance, Scalability on the Server Side John VanDyk Presented at Des Moines Web Geeks 9/21/2009 Who is this guy? History • Apple // • Macintosh • Windows 3.1- Server 2008R2 • Digital Unix (Tru64) • Linux (primarily RHEL) • FreeBSD Systems Iʼve worked with over the years. Languages • Perl • Userland Frontier™ • Python • Java • Ruby • PHP Languages Iʼve worked with over the years (Userland Frontier™ʼs integrated language is UserTalk™) Open source developer since 2000 Perl/Python/PHP MySQL Apache Linux The LAMP stack. Time to Serve Request Number of Clients Performance vs. scalability. network in network out RAM CPU Storage These are the basic laws of physics. All bottlenecks are caused by one of these four resources. Disk-bound •To o l s •iostat •vmstat Determine if you are disk-bound by measuring throughput. vmstat (BSD) procs memory page disk faults cpu r b w avm fre flt re pi po fr sr tw0 in sy cs us sy id 0 2 0 799M 842M 27 0 0 0 12 0 23 344 2906 1549 1 1 98 3 3 0 869M 789M 5045 0 0 0 406 0 10 1311 17200 5301 12 4 84 3 5 0 923M 794M 5219 0 0 0 5178 0 27 1825 21496 6903 35 8 57 1 2 0 931M 784M 909 0 0 0 146 0 12 955 9157 3570 8 4 88 blocked plenty of RAM, idle processes no swapping CPUs A disk-bound FreeBSD machine. b = blocked for resources fr = pages freed/sec cs = context switches avm = active virtual pages in = interrupts flt = memory page faults sy = system calls per interval vmstat (RHEL5) # vmstat -S M 5 25 procs ---------memory-------- --swap- ---io--- --system- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 1 0 0 1301 194 5531 0 0 0 29 1454 2256 24 20 56 0 0 3 0 0 1257 194 5531 0 0 0 40 2087 2336 34 27 39 0 0 2 0 0 1183 194 5531 0 0 0 53 1658 2763 33 28 39 0 0 0 0 0 1344 194 5531 0 0 0 34 1807 2125 29 19 52 0 0 no blocked busy but not processes overloaded CPU in = interrupts/sec cs = context switches/sec wa = time waiting for I/O Solving disk bottlenecks • Separate spindles (logs and databases) • Get rid of atime updates! • Minimize writes • Move temp writes to /dev/shm Overview of what weʼre about to dive into. -
The Linux Command Line
The Linux Command Line Fifth Internet Edition William Shotts A LinuxCommand.org Book Copyright ©2008-2019, William E. Shotts, Jr. This work is licensed under the Creative Commons Attribution-Noncommercial-No De- rivative Works 3.0 United States License. To view a copy of this license, visit the link above or send a letter to Creative Commons, PO Box 1866, Mountain View, CA 94042. A version of this book is also available in printed form, published by No Starch Press. Copies may be purchased wherever fine books are sold. No Starch Press also offers elec- tronic formats for popular e-readers. They can be reached at: https://www.nostarch.com. Linux® is the registered trademark of Linus Torvalds. All other trademarks belong to their respective owners. This book is part of the LinuxCommand.org project, a site for Linux education and advo- cacy devoted to helping users of legacy operating systems migrate into the future. You may contact the LinuxCommand.org project at http://linuxcommand.org. Release History Version Date Description 19.01A January 28, 2019 Fifth Internet Edition (Corrected TOC) 19.01 January 17, 2019 Fifth Internet Edition. 17.10 October 19, 2017 Fourth Internet Edition. 16.07 July 28, 2016 Third Internet Edition. 13.07 July 6, 2013 Second Internet Edition. 09.12 December 14, 2009 First Internet Edition. Table of Contents Introduction....................................................................................................xvi Why Use the Command Line?......................................................................................xvi -
Filesystem Hierarchy Standard
Filesystem Hierarchy Standard LSB Workgroup, The Linux Foundation Filesystem Hierarchy Standard LSB Workgroup, The Linux Foundation Version 3.0 Publication date March 19, 2015 Copyright © 2015 The Linux Foundation Copyright © 1994-2004 Daniel Quinlan Copyright © 2001-2004 Paul 'Rusty' Russell Copyright © 2003-2004 Christopher Yeoh Abstract This standard consists of a set of requirements and guidelines for file and directory placement under UNIX-like operating systems. The guidelines are intended to support interoperability of applications, system administration tools, development tools, and scripts as well as greater uniformity of documentation for these systems. All trademarks and copyrights are owned by their owners, unless specifically noted otherwise. Use of a term in this document should not be regarded as affecting the validity of any trademark or service mark. Permission is granted to make and distribute verbatim copies of this standard provided the copyright and this permission notice are preserved on all copies. Permission is granted to copy and distribute modified versions of this standard under the conditions for verbatim copying, provided also that the title page is labeled as modified including a reference to the original standard, provided that information on retrieving the original standard is included, and provided that the entire resulting derived work is distributed under the terms of a permission notice identical to this one. Permission is granted to copy and distribute translations of this standard into another language, under the above conditions for modified versions, except that this permission notice may be stated in a translation approved by the copyright holder. Dedication This release is dedicated to the memory of Christopher Yeoh, a long-time friend and colleague, and one of the original editors of the FHS. -
Linux Kernel and Driver Development Training Slides
Linux Kernel and Driver Development Training Linux Kernel and Driver Development Training © Copyright 2004-2021, Bootlin. Creative Commons BY-SA 3.0 license. Latest update: October 9, 2021. Document updates and sources: https://bootlin.com/doc/training/linux-kernel Corrections, suggestions, contributions and translations are welcome! embedded Linux and kernel engineering Send them to [email protected] - Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 1/470 Rights to copy © Copyright 2004-2021, Bootlin License: Creative Commons Attribution - Share Alike 3.0 https://creativecommons.org/licenses/by-sa/3.0/legalcode You are free: I to copy, distribute, display, and perform the work I to make derivative works I to make commercial use of the work Under the following conditions: I Attribution. You must give the original author credit. I Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under a license identical to this one. I For any reuse or distribution, you must make clear to others the license terms of this work. I Any of these conditions can be waived if you get permission from the copyright holder. Your fair use and other rights are in no way affected by the above. Document sources: https://github.com/bootlin/training-materials/ - Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 2/470 Hyperlinks in the document There are many hyperlinks in the document I Regular hyperlinks: https://kernel.org/ I Kernel documentation links: dev-tools/kasan I Links to kernel source files and directories: drivers/input/ include/linux/fb.h I Links to the declarations, definitions and instances of kernel symbols (functions, types, data, structures): platform_get_irq() GFP_KERNEL struct file_operations - Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 3/470 Company at a glance I Engineering company created in 2004, named ”Free Electrons” until Feb. -
A High Performance Ldm Data Cluster
55 A HIGH PERFORMANCE LDM DATA CLUSTER Robert C. Lipschutz, David Hagerty*, Paul Hamer, Peter Lannigan*, and Chris MacDermaid Cooperative Institute for Research in the Atmosphere (CIRA) Colorado State University, Fort Collins, Colorado USA, and NOAA Earth System Research Laboratory (ESRL), Boulder, Colorado USA *NOAA Earth System Research Laboratory (ESRL), Boulder, Colorado USA Contract with Riverside Technology, Inc., Fort Collins, Colorado USA 1. INTRODUCTION Data Manager (LDM) have enabled us to construct a high performance real-time data Within the NOAA Earth System Research processing environment that serves the GSD Laboratory (ESRL), the Global Systems Division community. Given the central role of LDM in the (GSD) develops weather information systems, architecture, we particularly focus on the how weather forecast models, and other applications LDM provides cluster services for data transport, in support of the National Weather Service, the and on the use of event notifications in LDM. Federal Aviation Administration, and other agencies. Well-known GSD products include the 2. LIMITATIONS OF THE HA ARCHITECTURE Rapid Update Cycle (RUC) model, the Local Analysis and Prediction System (LAPS), the While GSD's HA pairs have operated Meteorological Assimilation Data Ingest System successfully for a number of years (MADIS), and Science On a Sphere® (SOS). A (Lipschutz and MacDermaid, 2005), and provided common feature of these and other projects is robust availability through their automated fail- that they require observational and model data over mechanism, a major drawback has been the provided by acquisition systems running within need for two machines, one of which is mostly GSD’s Central Facility. These systems handle idle, to support a single processing configuration. -
Oracle® Linux 7 Managing File Systems
Oracle® Linux 7 Managing File Systems F32760-07 August 2021 Oracle Legal Notices Copyright © 2020, 2021, Oracle and/or its affiliates. This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited. The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing. If this is software or related documentation that is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, then the following notice is applicable: U.S. GOVERNMENT END USERS: Oracle programs (including any operating system, integrated software, any programs embedded, installed or activated on delivered hardware, and modifications of such programs) and Oracle computer documentation or other Oracle data delivered to or accessed by U.S. Government end users are "commercial computer software" or "commercial computer software documentation" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, the use, reproduction, duplication, release, display, disclosure, modification, preparation of derivative works, and/or adaptation of i) Oracle programs (including any operating system, integrated software, any programs embedded, installed or activated on delivered hardware, and modifications of such programs), ii) Oracle computer documentation and/or iii) other Oracle data, is subject to the rights and limitations specified in the license contained in the applicable contract. -
File and Console I/O
File and Console I/O CS449 Spring 2016 What is a Unix(or Linux) File? • File: “a resource for storing information [sic] based on some kind of durable storage” (Wikipedia) • Wider sense: “In Unix, everything is a file.” (a.k.a “In Unix, everything is a stream of bytes.”) – Traditional files, directories, links – Inter-process communication (pipes, shared memory, sockets) – Devices (interactive terminals, hard drives, printers, graphic card) • Usually mounted under /dev/ directory – Process Links (for getting process information) • Usually mounted under /proc/ directory Stream of Bytes Abstraction • A file, in abstract, is a stream of bytes • Can be manipulated using five system calls: – open: opens a file for reading/writing and returns a file descriptor • File descriptor: index into an OS array called open file table – read: reads current offset through file descriptor – write: writes current offset through file descriptor – lseek: changes current offset in file – close: closes file descriptor • Some files do not support certain operations (e.g. a terminal device does not support lseek) C Standard Library Wrappers • C Standard Library wraps file system calls in library functions – For portability across multiple systems – To provide additional features (buffering, formatting) • All C wrappers buffered by default – Buffering can be controlled using “setbuf” or “setlinebuf” calls (remember those?) • Works on FILE * instead of file descriptor – FILE is a library data structure that abstracts a file – Contains file descriptor, current offset, buffering mode etc. Wrappers for the Five System Calls Function Prototype Description FILE *fopen(const char *path, const Opens the file whose name is the string pointed to char *mode); by path and associates a stream with it. -
Filesystem Considerations for Embedded Devices ELC2015 03/25/15
Filesystem considerations for embedded devices ELC2015 03/25/15 Tristan Lelong Senior embedded software engineer Filesystem considerations ABSTRACT The goal of this presentation is to answer a question asked by several customers: which filesystem should you use within your embedded design’s eMMC/SDCard? These storage devices use a standard block interface, compatible with traditional filesystems, but constraints are not those of desktop PC environments. EXT2/3/4, BTRFS, F2FS are the first of many solutions which come to mind, but how do they all compare? Typical queries include performance, longevity, tools availability, support, and power loss robustness. This presentation will not dive into implementation details but will instead summarize provided answers with the help of various figures and meaningful test results. 2 TABLE OF CONTENTS 1. Introduction 2. Block devices 3. Available filesystems 4. Performances 5. Tools 6. Reliability 7. Conclusion Filesystem considerations ABOUT THE AUTHOR • Tristan Lelong • Embedded software engineer @ Adeneo Embedded • French, living in the Pacific northwest • Embedded software, free software, and Linux kernel enthusiast. 4 Introduction Filesystem considerations Introduction INTRODUCTION More and more embedded designs rely on smart memory chips rather than bare NAND or NOR. This presentation will start by describing: • Some context to help understand the differences between NAND and MMC • Some typical requirements found in embedded devices designs • Potential filesystems to use on MMC devices 6 Filesystem considerations Introduction INTRODUCTION Focus will then move to block filesystems. How they are supported, what feature do they advertise. To help understand how they compare, we will present some benchmarks and comparisons regarding: • Tools • Reliability • Performances 7 Block devices Filesystem considerations Block devices MMC, EMMC, SD CARD Vocabulary: • MMC: MultiMediaCard is a memory card unveiled in 1997 by SanDisk and Siemens based on NAND flash memory.