Introduction to Unix History of Unix Systems Base Concepts Grégory Mounié Linux Installation

Presentation of the lecture History of unix systems Base concepts Linux installation CLI Conﬁguration Distribution and Scalability Presentation of the lecture History of unix systems Base concepts Linux installation CLI Conﬁguration Distribution and Scalability Outline

Presentation of the lecture

SAaNS: Introduction to Unix History of unix systems Base concepts Grégory Mounié Linux installation

2014-2015 CLI

Conﬁguration

Distribution and Scalability

1 / 75 2 / 75

History of unix systems • [email protected] • Associate professor at Ensimag Base concepts • Oﬃce: Building of Inria Montbonnot (11 km of the campus) • Current teaching: mainly Operating Systems; Networks and Linux installation distributed systems; Python; • Former teaching: Networks, Java, Ada, Operational research, CLI OpenGL • Research: Scheduling for High Performance Computing Conﬁguration (HPC), or the art to put cubes in boxes

Distribution and Scalability

3 / 75 3 / 75 Presentation of the lecture History of unix systems Base concepts Linux installation CLI Conﬁguration Distribution and Scalability Presentation of the lecture History of unix systems Base concepts Linux installation CLI Conﬁguration Distribution and Scalability Content of the course Bibliography

Main reference Unix® and Linux® System Administration Handbook, Fourth Edition, Video Enhanced Edition. Evi Nemeth; Garth Snyder; Trent R. Hein; Ben Whaley • OS basics and OS administration in 3 hours Basics • Network basics and Network administration in 3 hours • Security in 3 hours OS Modern Operating Systems, Tanenbaum • Perl in (3 + 3 hours) Network Computer Networks, Tanenbaum Network and system programming Unix programming, Stevens (a bit old but still the reference for many low levels and raw stuﬀ)

4 / 75 5 / 75

• People with suﬃcent background may skip the basics part of • Man pages the lecture and do the challenge instead. • GNU Texinfo (Html with automatic index) Challenge for the OS lecture • Package-speciﬁc doc • typically in /usr/share/doc/ You are employee of X and you will go to Paques Island to sell some Y. The major provider of Y is an US society. You will cross • RFC and Internet documents the US border with commercial documents in your laptop. • beware of version mismatch Choose a way to protect your documents, implement it and present it to others lecturers (10 min at the end of the lecture)

6 / 75 7 / 75 Presentation of the lecture History of unix systems Base concepts Linux installation CLI Conﬁguration Distribution and Scalability Presentation of the lecture History of unix systems Base concepts Linux installation CLI Conﬁguration Distribution and Scalability Duties of the system administrator Origin of UNIX

• Account provisioning • Adding and removing hardware • Performing backups in the beginning was multics (65-68) • Installing and upgrading software • failure ⇒ Bell decides to stop making operating systems • Monitoring the system • Kernigham and Ritchie et al. develop a small operating system • Troubleshooting of their own • Maintaining local documentation • by derision they call it UNICS (69) • and create a programming language (C) to recode it • Vigilantly monitoring security • property of Bell Labs • Fire ﬁghting • restrained diﬀusion when successful

Diﬃcult point Each point is simple. Complexity come from the cross-references between points.

8 / 75 9 / 75

• sources was given (Unix V6) to users (device drivers)

Berkeley University (76) • no free (as in speech) version • ﬁrst version of BSD • development of internet: • ⇒ ideal conditions for the development of free unices: • large diﬀusion in universities • strong demand • skilled peoples Parallel evolution, in 1983 • means enabling common development • 4.3BSD (SunOS) • Unix System V

10 / 75 11 / 75 Presentation of the lecture History of unix systems Base concepts Linux installation CLI Conﬁguration Distribution and Scalability Presentation of the lecture History of unix systems Base concepts Linux installation CLI Conﬁguration Distribution and Scalability MINIX and Linux Unixes Today

Philosophy 1. Linux and its embedded variants: • Android, Firefox OS, Sailﬁsh OS (ex-Mer, ex-Meego), Ubuntu • Minix (85) written by Prof. A. Tannenbaum for his courses Touch, Tizen (ex-Meego), WebOS, Plasma Active • Desktop distributions: Debian, Fedora, etc. • internet ⇒ large diﬀusion and improvements 2. BSD family: • but restriction on user rights • Darwin (MacOS X, IOS), NetBSD, FreeBSD, OpenBSD • Linux written by a student Linus 3. A given number of (vanishing) proprietary unixes : Torvalds • Solaris, AIX, HP/UX • support of recent architectures 4. other exotic unixes: Minix, Hurd, Haiku (BeOS clone) . . . • GPL licence

12 / 75 13 / 75

• free as in speech not as in beer ! • based on open standards • diﬀerent licences : BSD, GNU Public Licence, . . . Freedom 2 • defending users rights : The freedom to improve the program, and 1. The freedom to run the program, for any purpose release your improvements to the public. 2. The freedom to study how the program works, and adapt it to Access to the source code is a precondition your needs. for this. • Access to the source code is a precondition for this. 3. The freedom to redistribute copies 4. The freedom to redistribute modiﬁed copies.

14 / 75 15 / 75 Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability Free software today Computer are fast, stupid and huge but we have to work with Computer are: GNU/Linux increasingly popular • stupid [G. Berry]:: only know the 4 operations in base 2 and • Huge hidden base: Android Smartphone, xDSL box, router, data movement rice cooker, fast do 1 operation while a photon travels few centimeters huge The number of bits in a disk is similar to the solar mass of our • Desktop user groups in almost all countries galaxy • in France : user groups in every major city • huge amount of different projects with public on-going source Computer science challenge code How to build a non-stupid result using numerous simple operations • http://github.com ? • http://gitorious.org • http://freshmeat.net Computer science 4 pillars [G. Doweq] • http://sourceforge.net 1. Algorithmic :: Express human idea in computer logic 2. Data :: data structure and storage to store human ideas efficiently for the computer logic

16 / 75 3. Compilation :: programming languages to explain what to do 17 / 75 to the computer (fast, correct, easy to understand, etc. ) 4. Hardware :: How to build a computer Presentation of the lecture History of unix systems Base concepts Linux installation CLI Conﬁguration Distribution and Scalability Presentation of the lecture History of unix systems Base concepts Linux installation CLI Conﬁguration Distribution and Scalability Are we using 40 years old tools ??? Does Unix 40 year history means. . .

• we are using 40 year old tools ?

Yes some tools worked on dot-matrix printer before screen age: • ed (brother of sed) line-by-line text editor is still • Most concepts are present in Multics installed in MacOS X, and vi is installed in • Some tools are still the same busybox used by many embedded systems • however, many tools evolved (Android phones, DSLboxes, linux-base network • time ﬁlters things : only the most router) valuable things are kept over time No git and vlc are 10 years old • ⇒ unix has some simple and elegant concepts

18 / 75 19 / 75 Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability Unix concepts Everything is a file

composition and K.I.S.S • lots of small and efficient tools • easy composition of different tools to obtain • Why ? powerful effects In computers, all devices are about IO modularity Unix = • a kernel handling low level tasks • reading data from a device • a set of applications • writing data to a device (almost) everything is a file • sockets, devices, pipes, . . . • File interface for all device ⇒ Fewer system calls • uniform way of handling all resources

20 / 75 21 / 75

22 / 75 23 / 75 Presentation of the lecture History of unix systems Base concepts Linux installation CLI Conﬁguration Distribution and Scalability Presentation of the lecture History of unix systems Base concepts Linux installation CLI Conﬁguration Distribution and Scalability Multitasking Controlling process

Process attributes • an executing program is called a process • PID unique process ID number • several processes execute simultaneously • PPID, UID, GID, priority • sharing of memory (code and data) • sharing of CPU resources • sharing of peripheral devices Process control

Warning • ps, kill, nice process != program. A process is a program in execution • strace, gdb • /proc ﬁlesystem

24 / 75 25 / 75

• Users have credentials to manipulate the files (everything is a • several users on the same machine file) • notion of user accounts • home directory, ID number (UID), password, name, . . . Basic model user/group/other: chmod, setuid Advanced model ACL • each file is owned by one of the users • processes are owned by the user launching them Basic security rule • several processes belonging to different users coexist in the Limit the credentials to the minimum (It is a lot of work) same system The system may refuse some actions even with administrative rights (security, DRM, containment)

26 / 75 27 / 75 Presentation of the lecture History of unix systems Base concepts Linux installation CLI Conﬁguration Distribution and Scalability Presentation of the lecture History of unix systems Base concepts Linux installation CLI Conﬁguration Distribution and Scalability Memory handling Filesystem

• logical organization (tree) of physical space • two types of memory : • dependent on the OS • numerous formats • physical memory : the one really existing inside the machine • DOS/Windows 98 (FAT16, FAT32), Windows NT and later • virtual memory : the address space used by processes (NTFS), Linux (ext2, ext3, ext4, btrfs, reiserfs, . . . ), . . . • less physical than virtual • CDROM, DVD • if physical becomes overused ⇒ swapping on disk • ... • laziness : memory is physically reserved at the ﬁrst r/w • a partition has to be mounted to become accessible • hardware translation from virtual to physical (MMU) • software management of hardware translation (Page Table) Example (mount) mount (1) to manage mounted partitions Allow to use multiple partitions and hardwares in a single tree.

28 / 75 29 / 75

• Hardware space is a large block array • Every normal file data are stored in a particular sequence of block • All filesystems expose the same interface thus similar concept as the inheritance of object programming may be applied: • The sequence of block of a file is described in a structure • Programs and systems use filesytem through the same generic (inode) stored in a (fixed size) table, with the file metadata interface (VFS) (UID, access right . . . save the file name) • Filesystems differences are hidden from common users • A directory is just a file (a single block or a sequence) storing • Matroska stack for network filesystems: NFS expose VFS pairs of (file name, inode number) interface and access (remotely) the data through VFS)

Example (stat) stat (1) / stat (2) to get information on a particular ﬁle

30 / 75 31 / 75 Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability High level: Main directories Special files

standard rules /bin /sbin base system programs /usr/bin programs • Directories /usr/share data files for programs in /usr/bin • Character (eg. mouse) and Block (HD, SSD) device files /usr/share/doc program and system documentation • Sockets /home user home directories • Named pipes /var /var/log log files, changing system files • Symbolic link /tmp temporary files /etc configuration files /usr/local/ locally compiled or installed file

32 / 75 33 / 75

• processes handling • creation, destruction, time-sharing • Distribution = a set of programs distributed with the kernel • virtual memory handling • base tools for OS configuration and maintenance • allocation, freeing, swap • user applications • one process == one memory zone • Distribution = hierarchy of directories • peripheral devices accesses • file positions • keyboard, mouse, hard-disk/SSD, network, . . . • configuration files • one (loose) norm : FHS (Filesystem Hierarchy Standard) • process communication and synchronization • energy saving

34 / 75 35 / 75 Presentation of the lecture History of unix systems Base concepts Linux installation CLI Conﬁguration Distribution and Scalability Presentation of the lecture History of unix systems Base concepts Linux installation CLI Conﬁguration Distribution and Scalability Main Desktop Linux distributions Counter-example: Android

• Debian (Ubuntu, Mint) • Redhat (Fedora, RHEL, CentOS, OpenSuse) • The kernel is the same • Mageïa (ex Mandrake, ex Mandriva) • The set of program is very diﬀerent: • Oracle Linux (ex Netware) • JVM (Dalvik): most programs are Java program • C library (Bionic, not the full blend GNU Library) • Gentoo • Some other ad-hoc daemons • Arch linux • Slackware There are 300+ others http://distrowatch.com.

36 / 75 37 / 75

two types of hardware : 1. handled by the kernel • preparation • serial port, disk controllers, keyboard, . . . • boot 2. handled partially by applications (user-space drivers) • disk partitioning • graphical card, some USB gadgets, . . . • files installations • base system configuration golden rules for System Administration • accounts creations • no material too recent • lots of online documentation to read first

38 / 75 39 / 75 Presentation of the lecture History of unix systems Base concepts Linux installation CLI Conﬁguration Distribution and Scalability Presentation of the lecture History of unix systems Base concepts Linux installation CLI Conﬁguration Distribution and Scalability Preparation Disk partitioning

• swap partition • check if the disk can be erased • : main system partition • else re-partitioning / • : eventually • boot on: CD/DVD, USB key, network (eg. PXE), hard /boot disk/SSD, floppy disk • /home : users files • UEFI may stop you from booting anything but signed kernel • /var : changing system files • GPT is much modern alternative than standard PC partitioning • /tmp : temporary files • Raid (hard/soft) and LVM helps a lot but should be used with Why so many partitions ? caution Containing problems in case of filled-up partitions (eg. /var/log/)

40 / 75 41 / 75

• choice of type of installation The needs change with time: storage increase with time, or change • desktop technology. • workstation • • server Ext4 is still the standard for desktop and phone • scientiﬁc computing • ZFS (Sun/Oracle) allows plenty of nice things (integrity, • graphic station storage pool, snapshot, copy-on-write, deduplication) but is • media center not part of the standard linux kernel and distribution (FUSE) • ... • Btrfs is still not ready yet, but should arrive very soon • more ﬁne tune by manual packages selection

42 / 75 43 / 75 Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability Packages Different packages systems

Definition (Package) application / library / set of applications • DEB : debian, ubuntu, mint. . . • add or removes packages instead of files • RPM : redhat, mandriva, suse, . . . • goal : ease administration • TAR : just an archive file without dependencies • handling of dependencies between packages (The main • some conversion tools between packages (alien) difference with GooglePlay, AppleStore, etc.)

44 / 75 45 / 75

• Role-based: at least two categories of accounts • administrators : root • creating a new user account • user accounts • username, home directory, password • possibilities to delegate root powers to standard users (setuid, • all users are referenced in the /etc/passwd ﬁle sudo) • encrypted passwords are to be found in the /etc/shadow ﬁle • adduser command (useradd: low level) Warning the root account should only be used for administration

46 / 75 47 / 75 Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability Presentation of the lecture History of unix systems Base concepts Linux installation CLI Configuration Distribution and Scalability Advanced account management Configuration files

PAM: Pluggable Authentication Modules (Debian) libpam has 66 modules. Most are different ways to authenticate a user (password, ssh keys, kerberos, two-factor, ldap, • configuration files are text files mysql, geoIP, etc.) • easy to modify using any text editor Kerberos third party cryptographic authentication • global configuration files in /etc ACL Access control lists fine grain chmod on files • local configuration files hidden in home directory root account strong password, indirect login (su/sudo), lost root • one file for every application password POSIX capabilities program/shell may withdraw some rights LDAP and Active Directory centralizing account management

48 / 75 49 / 75

• each file has an owner : • initially, the file creator • each user belongs to a one or more groups • rights are associated to each file • possibility to share files among all users of a group • security and flexibility • reading, writing, execution rights • the list of groups is in /etc/group • for the owner • the root user can create new groups • for the members of the group • for all other users

50 / 75 51 / 75 Presentation of the lecture History of unix systems Base concepts Linux installation CLI Conﬁguration Distribution and Scalability Presentation of the lecture History of unix systems Base concepts Linux installation CLI Conﬁguration Distribution and Scalability Mounting points Command line

• text based interface to the system • under DOS/WINDOWS : different peripheral with different file • one main disadvantage : less intuitive than mouse interface systems • many advantages : • a: c: d: ... • scripts to automate actions • under linux : only one root directory (/) • very useful for distant administration (ssh) • adding new devices with the mount command • same interface everywhere • configuration file : /etc/fstab • powerful for mass operations (e.g. renaming 200 files) • very quick : automatic completion • completion allows to avoid typing mistakes

52 / 75 53 / 75

cat concatenate files content (many UUOC) ls list files in directory cut select columns cat display a file find find a file rm remove a file seq enumerate an interval (for) cd change directory sort sort mkdir make new directory uniq remove consecutive duplicate mv move a file tee save pipe in a file cp copy a file wc word count grep filter file content head and tail first lines and last lines of a file xargs take input as argument of the following command basename remove suffix in a name

54 / 75 55 / 75 Presentation of the lecture History of unix systems Base concepts Linux installation CLI Conﬁguration Distribution and Scalability Presentation of the lecture History of unix systems Base concepts Linux installation CLI Conﬁguration Distribution and Scalability Script internal commands Example

• Basic control ﬂow: • branch (if-then-else) • loops Simple command, complex interaction • arrays Find the number of line of the 5 largest ﬁles in the current directory • regular expressions and its sub-directories • matching find . -type f -exec ls -l ’{}’ ’;’ | sort -k 5 | • literal and special character tail -5 | xargs wc -l • The syntax varies with the interpreters • sh/bash/dash, csh/tcsh, ksh/zsh

56 / 75 57 / 75

• all devices are accessible as files in the /dev directory • all commands are programs • access rights to devices == standard files rights • manual pages : man • special files created using the mknod command • other doc in /usr/share/doc : • standard commands (cat, . . . ) working ! • programs documentation • • HOWTO files example : /dev/mouse

58 / 75 59 / 75 Presentation of the lecture History of unix systems Base concepts Linux installation CLI Conﬁguration Distribution and Scalability Presentation of the lecture History of unix systems Base concepts Linux installation CLI Conﬁguration Distribution and Scalability Common devices Disks

• IDE (ATA) disks • /dev/printer • /dev/hda : primary master • /dev/modem • /dev/hdb : primary slave • /dev/mouse • SCSI and SATA disks • /dev/dsp • /dev/sda, /dev/sdb,... • • /dev/cdrom ﬁrst partition on ﬁrst SCSI : /dev/sda1 • SDcard disks (Android) • /dev/fb0 • /dev/block/mmcblk0

60 / 75 61 / 75

• all configuration in text files • allow easy automatic modifications using scripts • /etc/fstab : all mount points • almost all files in /etc • /etc/modules.conf : kernel modules configuration • there exists some graphical frontends and global management • /etc/X11/xorg.conf : X11 configuration file systems • /etc/cups/ : all config files for CUPS service • /etc/resolv.conf : DNS config (often automatically generated) Warning • ... Configuration files vary from distribution to distribution as do frontends.

62 / 75 63 / 75 Presentation of the lecture History of unix systems Base concepts Linux installation CLI Conﬁguration Distribution and Scalability Presentation of the lecture History of unix systems Base concepts Linux installation CLI Conﬁguration Distribution and Scalability Services Services

• service : functionality offered by the machine • program running in the background : daemon Cron configuration • examples : • one file containing a list of jobs : /etc/crontab • printing • man crontab • email • web server mm hh dd MMM DDD tâche > log • ssh • ftp server • ...

64 / 75 65 / 75

• delayed execution • cron works only if the machine is always on • at time action • in other cases is preferable • useful for use of machines during night anacron • • examples : asynchronous cron • at 10pm wget http://linuxfr.org • execution if not done recently • at 2am tomorrow launch_computation

66 / 75 67 / 75 Presentation of the lecture History of unix systems Base concepts Linux installation CLI Conﬁguration Distribution and Scalability Presentation of the lecture History of unix systems Base concepts Linux installation CLI Conﬁguration Distribution and Scalability Kernel Modules

• drivers can be included in the kernel • some commands : • or separated in modules • insmod : load a module • only loading useful drivers • rmmod : remove a module • dynamic loading / unloading at runtime • lsmod : display all loaded modules • modprobe : load a module with dependencies • auto-detection by hotplug

68 / 75 69 / 75

• program enabling OS choice • if not secured : security weakness • Managing 10 machines the same way as a single machine is • several bootloaders : tedious • LILO • Managing 1000 machines required other tools • /etc/lilo.conf • Managing 100000 required other tools than previous one • GRUB • /boot/grub/menu.lst

70 / 75 71 / 75 Presentation of the lecture History of unix systems Base concepts Linux installation CLI Conﬁguration Distribution and Scalability Presentation of the lecture History of unix systems Base concepts Linux installation CLI Conﬁguration Distribution and Scalability Authenticating on a large set of machine Executing any command on a large set of machine

LDAP command-line is your friend Remote BD often dedicated to directory of people and their • scriptable: easy to repeat a sequence of operation administrative right. Easy to interconnect with local PAM service • remote access: easy do do it remotely using ssh NIS Old school (No black hat). trivial to set up but no security Parallel ssh Scalability • parallel launchers exist. Use them ! Using a large set of machine on a single central service may become • taktuk/kanif, dsh, pdsh, dish, capistrano, pssh, clustershell a bottleneck. • related to the scalability of authentication

72 / 75 73 / 75

The conﬁguration of the "client" host may be managed by a master: Bittorrent • puppet (http://www.puppetlabs.com) (ruby) Large scale distribution of ﬁle is quite useful ! • chef (http://wiki.opscode.com/display/chef) (ruby/git) • ansible (http://ansible.com) (python) • salt (http://saltstack.org/) (python)

74 / 75 75 / 75