../rhd256.bowez_notes.2006-07-20.txt Thu Jul 20 15:31:29 2006 1 ==> ./01_intro/notes.txt <== ------2.3 Bash Special Characaters 1.3 Prerequisates ------Familiarity: Redirection (Stream Manipulation) ------the classroom setup (server1, station1, etc...) the "pub" directory every process has three default file descriptors (streams):
Common Tools: 0: stdin 1: stdout terminal: 2: stderr changing fonts, etc.... - by default, all are connected to the terminal. editors: - In RHL, can be accessed by /dev/std{in,out,err} emacs vi, vim, gvim < redirect descriptor input pico -w, gedit > redirect descriptor output (create or clobber) (less pager) >> redirect descriptor output (create or append)
finding information: 2>&1 bind one descriptor to another (i.e., combine stderr and stdout) --help, -h >& bash shortcut for the same man pages (chapters) info pages (info, pinfo, nautilus) examples: /usr/share/doc grep root /etc/* rpm (-qf filename, -ql, -qi) grep root /etc/* > /tmp/out grep root /etc/* 2> /tmp/err ------grep root /etc/* > /tmp/out2 2> /tmp/err2 grep root /etc/* >& /tmp/all 1.4 Procedures
------Compound Command Grouping anonymous cvs access to notes/sources: ------
CVSROOT=:pserver:anonymous@server:/usr/local/cvsroot | pipe
cvs co gls/curricula/rhd143 rpm -qa | sort cvs export -Dnow gls/curricula/rhd143 ; execute commands in sequence && if first succeeds (return code == 0) , execute second || if first fails (return code != 0) , execute second (...) execute commands in subshell ==> ./03_bash/notes.txt <== ------examples: cd /var/tmp; date > foo; ls; pwd 2.2 Shells ls /tmp > /dev/null && echo "it exists" ls /tmpfoo > /dev/null || echo "it no exists" ------(cd /usr/share/kudzu; ls); pwd
Bash Red Hat Specific Initialization ------login shells: Command Line Expansions /etc/profile ------/etc/profile.d/*.sh ˜/.bash_profile brace {} touch foo{a,b,c} tilde ˜ cd ˜elvis login and non-login shells: parameter $ cd $HOME ˜/.bashrc command ‘...‘, $(...) touch System-map-‘uname -r‘ /etc/bashrc arithmetic $((...)) pathname *,?,[...],[^...] ls /etc/*.conf ../rhd256.bowez_notes.2006-07-20.txt Thu Jul 20 15:31:29 2006 2 history ! !1024, !v echo ’good teacher’ else Expansions happen __before__ command is run! echo ’no apple for you’ fi Compare: find /etc -name *.conf "man test" for more conditionals touch a.conf b.conf; find /etc -name *.conf if [ x$A -eq "xapple" ]; then (set -x can be helpful in deciphering these) echo ’good teacher’ else ------echo ’no apple for you’ Interactive Tricks fi ------[ -x /usr/bin/foo ] && /usr/bin/foo
------for i in dog cat mouse gerbil; do Compound Syntax echo "nice $i" ------done
------for i in $*; do branches echo "nice $i" ------done
if _command_ for i in $(users); do then echo "nice $i" _command_ done _command_ else _command_ set -x fi # vi:ts=4
if grep elvis /etc/passwd; then echo "elvis is in the house" ==> ./04_gcc/notes.txt <== else ------echo "no elvis" fi Section 3 - gcc compiler
------if grep -q elvis /etc/passwd; then echo "elvis is in the house" executable: gcc else rpm: gcc echo "no elvis" layout: /usr/lib/gcc-lib/ fi (the compiler table)
if test $A -eq "apple"; then demo - hello world echo ’good teacher’ illustrate stages else link against other executables echo ’no apple for you’ fi gcc *.c gcc -o hello *.c if test x$A -eq "xapple"; then gcc -E hello.c ../rhd256.bowez_notes.2006-07-20.txt Thu Jul 20 15:31:29 2006 3 gcc -S hello.c w/o greatly increasing compilation time: gcc -c hello.c thread-jumps, defer-pop, delayed-branch -O2 all but loop unrolling, function inlining +------+ -O3 -O2 + inline functions (can be hard to debug) | Stage | utility | rpm | gcc | gcc | input | output | -Os Optomize for size | | | | switch | option | suffix | suffix | |------+------+------+------+------+------+------| preprocessor options: |preprocess |cpp |cpp |-E |-Wp |.c |.i | |------+------+------+------+------+------+------| -Dmacro[=def] define macro |compile |gcc |gcc |-S | |.i |.s | -Umacro undefine macro |------+------+------+------+------+------+------| -M produce makefile dependency info only |assemble |as |binutils |-c |-WS |.s |.o | -MM same as -M, but don’t include system headers |------+------+------+------+------+------+------| |link |ld |binutils | |-Wl |.o |executable | -include, -imacros, -idirafter, -iwithprefix, -nostdinc +------+ modify include search path documentation: linker options:
man pages - examine gcc -shared generate shared object libraries (dflt) info pages - note gnome-help-browser -static generate static object libraries options: ------
compilation stages: cpp options
-E preprocess only ------S preprocess and compile only -c preprocess, compile, and assemble only details: -o filename set output file to filename -ansi directory options: - disables non ANSI features and keywords (asm, -Dunix, // comments) -Idir include directory in include file search path - defines __STRICT_ANSI__ -Ldir include directory in library search path - enables trigraphs -Bdir compiler base directory (useful for cross compilations) -no-asm target options: non ANSI code not flatout rejected, use in conjunction w/ -pedantic
-b arch (cross)compile for architecture arch ------language options: gcc extensions -ansi strict ansi conformance -traditional ??? ------
warning options: - local (block) labels
-Wall issue all warnings - nested functions -pedantic fail on non-ansi compliance -Werror promote all warnings to errors - typeof() -w disable warnings #define max(a,b) \ debugging: ({ typeof (a) _a = (a); \ typeof (b) _b = (b); \ -g produce debugging symbols _a > _b ? _a : _b; }) -ggdb produce debugging symbols w/ gdb extentions -p enable profiling with ’prof’ - int int (doubleword integers) -pg enable profiling with ’gprof’ -save-temps save temporary files - zero length arrays
optimization: struct line { int length; (none) optimize compilation time and debugging consistancey char contents[0]; -O,-O1 try to reduce code size and run time, }; ../rhd256.bowez_notes.2006-07-20.txt Thu Jul 20 15:31:29 2006 4
attach to a running program: attach 3204
arugments: show args set args arg1...
environment: # vi: ts=4 show environment [VARNAME] set environment VARNAME [=VALUE] ==> ./04_gdb/notes.txt <== ------forking: What gdb can do ------set follow-fork-mode parent|child|ask
a) start program (w/ any args,env) threads: b) set breakpoints, conditional breakpoints c) stopped program analysis features: d) dynamic changes - new thread notification - thread specific breakpoints Supports gcc, g++
------thread THREAD: shift threads compiling - info threads ------thread apply: apply cmd to all threads
gcc -g -O0 ------stopping (gdb can handle optimized code, will not be ’faithful’ ------to source file) break - break at position watch - break when value changes ------catch - break on events (signals, forks, execs) invoking ------breaking:
gdb break [file:]func | [file:]line | *addr | +/- offset gdb a.out gdb a.out core (read core file) break ... if cond.... gdb a.out 2341 (attach to running process) ignore num count
------rbreak regex (fcns which match regex) common commands tbreak ... (temporary breakpoint) ------hbreak ... ("hardware assisted") (needed for ROM/EPROM)
help watching:
run [arglist] watch expr... break when expr modified c|next|step {r,a}watch expr... break when expr read, or either
break [file:]function catching:
bt catch fork | exec | throw | catch | load | unload ... print expr removing breakpoints: list [file:]function {dis,en}able num quit clear [file:]{line,fcn} ------delete num running ------side effects:
start a program w/in gdb: run [arg1 arg2...] break commands: ../rhd256.bowez_notes.2006-07-20.txt Thu Jul 20 15:31:29 2006 5 disasemble addr1 addr2 commands bnum cmd1... cmd1... ------end scripting ------autodisplay: ˜/.gdbinit display expr -x filename threads: -batch break ... thread THREADNO [if ...] ------keeping track: continuing ------info: state of your program next [ignore_count] step address files registers target finish all-registers float remote-process terminal continue args frame scope threads breakpoints functions selectors tracepoints ------catch handle set types Viewing Data classes line sharedlibrary udot ------common locals signals variables copying macro source vector print EXPR dcache mem sources warranty x ADDR display proc stack watchpoints display EXPR extensions program symbol
info registers info float info breakpoints info mem info terminal info threads dump memory FILENAME START_ADDR END_ADDR restore FILENAME BIAS START END show: state of gdb
------signals Features: ------
handle SIG stop|print|pass completion: command names ------symbol names: functions, data frames ------logging: set logging on|off frame [NUM] - show or jump to a certain frame number set logging file FILENAME backtrace info frame|args|locals
------# vi: ts=4 source files ------==> ./06_i18n/notes.txt <==
list EXPR edit EXPR (uses EDITOR env variable) Gettext Overview (from info gettext) search REGEX nouns: directory DIRNAME (addd DIRNAME to front of search path) ".pot" - portable object tempalte info line NUM ".po" - portable object - contains translated strings ".mo" - machine object - compiled strings easily indexed at runtime ../rhd256.bowez_notes.2006-07-20.txt Thu Jul 20 15:31:29 2006 6 textdomain (PACKAGE);
Original C Sources ---> hand edit ---> Marked C Sources ---. - mark strings | .------<--- GNU gettext Library | - strings in function context: .--- make <---+ | | ‘------<------+------’ printf("hello world"); --> printf(_("hello world")); | | | .-----<--- PACKAGE.pot <--- xgettext <---’ .---<--- PO Compendium - strings in non-function context: | | | ^ | | ‘---. | ------| ‘---. +--> hand edit --. | +----> msgmerge ------> LANG.po ---->------’ | { | .---’ | static const char *msgs[] = { | | | "some very meaningful message", | ‘------<------. | "and another one" | +--- New LANG.po <------’ }; | .--- LANG.gmo <--- msgfmt <---’ | | const char *string; | ‘---> install ---> /.../LANG/PACKAGE.mo ---. ... | +---> "Hello world!" string = index > 1 ? "a default message" :msgs[index]; ‘------> install ---> /.../bin/PROGRAM ------’ fputs (string); ... - Goals of gettext: }
- remove burden of i18n from programmer as much as possible --- becomes ------
- isolate role of translator #define gettext_noop(String) String
- easy incorporation for end user { static const char *msgs[] = { stages: gettext_noop ("some very meaningful message"), gettext_noop ("and another one") 1) mark C sources }; 2) use xgettext to extract strings into PACKAGE.pot file const char *string; 3) cp PACKAGE.pot to LANG.po and translate ... 4) run msgfmt to convert LANG.po into LANG.mo string = 5) combine and insall LANG.mo into .../LANG/PACKAGE.mo index > 1 ? gettext ("a default message") : Periodically update LANG.po file with msgmerge gettext (msgs[index]);
------fputs (string); mark C sources ... ------}
- In a common header ------
#ifdef USE_NLS - string writing guidelines #include
- In main (or some startup fcn) xgettext --output-dir po --keyword=_ *.c *.h
setlocale (LC_ALL, ""); mv messages.po PACKAGE.pot bindtextdomain (PACKAGE, LOCALEDIR); ../rhd256.bowez_notes.2006-07-20.txt Thu Jul 20 15:31:29 2006 7 cp PACKAGE.pot LANG.po msgfmt *.po (--> builds messages.mo)
------cp PACKAGE.pot to LANG.po and translate ------special topics ------>> # SOME DESCRIPTIVE TITLE. >> # Copyright (C) YEAR THE PACKAGE’S COPYRIGHT HOLDER passing translation notes >> # This is distributed under the same license as the... >> # FIRST AUTHOR
.po file entries: ------
WHITE-SPACE ------# TRANSLATOR-COMMENTS General Makefile Syntax #. AUTOMATIC-COMMENTS ------#: REFERENCE... #, FLAG... TARGET ... : PREREQUISITE ... msgid UNTRANSLATED-STRING \t COMMAND msgstr TRANSLATED-STRING \t ...
recognized flags: ------Trivial Example "fuzzy" - translation might not be full accurate ------"{c,python,tcl,...}-format" - string contains format escapes dependent : independent tools to help maintain .po files: cat independent > dependent
msg{init,cat,merge,conv,grep,filter,uniq,comm} ------Almost Trivial Example ------run msgfmt to convert LANG.po into LANG.gmo dependent : independent ------cat independent > dependent ../rhd256.bowez_notes.2006-07-20.txt Thu Jul 20 15:31:29 2006 8 clean: rm dependent hello3.o : hello3.c gcc -c $^ ------More Realistic Example (taken from chapter 3) clean: ------rm -f $(OBJS) a.out *.s $(EXE)
hello: hello.o hello2.o hello3.o ------gcc -o hello hello.o hello2.o hello3.o Implicit Rules ------hello.o : hello.c gcc -c hello.c Implicit rules allow Makefiles to be reduced to pattern matching
hello2.o : hello2.c %.o: %.c gcc -c hello2.c gcc -c $^
hello3.o : hello3.c %: %.o gcc -c hello3.c gcc -o $@ $(OBJS)
clean: The Makefile above can now be reduced to just this: rm -f *.o a.out *.s hello EXE = hello ------OBJS = hello.o hello2.o hello3.o Variables ------$(EXE): $(OBJS) gcc -o $@ $^ EXE = hello OBJS = hello.o hello2.o hello3.o %.o: %.c gcc -c $^ $(EXE): $(OBJS) gcc -o $(EXE) $(OBJS) clean: rm -f $(OBJS) a.out *.s $(EXE) hello.o : hello.c gcc -c hello.c ------Builtin Implicit Rules hello2.o : hello2.c ------gcc -c hello2.c Implicit rules can be observed by invoking make -p. hello3.o : hello3.c gcc -c hello3.c Note the following excerpts from make -p:
clean: rm -f $(OBJS) a.out *.s $(EXE) COMPILE.c = $(CC) $(CFLAGS) $(CPPFLAGS) $(TARGET_ARCH) -c LINK.o = $(CC) $(LDFLAGS) $(TARGET_ARCH) ------Automatic Variables ------# build object files from C source files %.o: %.c $@ target $(COMPILE.c) $(OUTPUT_OPTION) $< $< first prerequesite $? all newer prerequisites # builld executable from object files $^ all prerequisites %: %.o $(LINK.o) $^ $(LOADLIBES) $(LDLIBS) -o $@ EXE = hello OBJS = hello.o hello2.o hello3.o Consider the following makefile:
$(EXE): $(OBJS) CC = gcc gcc -o $@ $^ CFLAGS = -g
hello.o : hello.c EXE = hello gcc -c $^ OBJS = hello.o hello2.o hello3.o
hello2.o : hello2.c $(EXE): $(OBJS) gcc -c $^ ../rhd256.bowez_notes.2006-07-20.txt Thu Jul 20 15:31:29 2006 9 clean: rm -f $(OBJS) a.out *.s $(EXE) lib: $(MAKE) -C $@ Good variables to know for C development: ------CC: the C compiler to use Miscelaneous CFLAGS: gcc swithces (-g, -pg, -O3, etc...) ------CPPFLAGS: cpp switches (-DDEBUG, -I/opt/foo/include, ...) LDFLAGS: ld switches (-static, ...) gcc -MM LDLIBDES: ld library dirs (-Llib, ...) LDLIBS: ld libraries (-lfoo, ...) each line in command is run in seperate shell!
------Dry Runs: Expansions make -n, --dry-run ------make -q, --
wildcard: $(wildcard *.c) # vi:ts=4 subst: $(subst .c,.o,$(wildcard *.c)) shell: $(shell grep VERSION foo.spec) ==> ./08_lib/notes.txt <======Consider the following: Section 4 - libraries CC = gcc CFLAGS = -g ======EXE = hello SRCS = $(wildcard *.c) two basic approaches: OBJS = $(subst .c,.o,$(SRCS)) static libraries libfoo.a $(EXE): $(OBJS) shared libraries libfoo.so
clean: ------rm -f $(OBJS) $(EXE) static libraries ------Variables ------A) building the library: How defined: 1) command line 1. compile code gcc -c *.c 2) explicit 2. archive object files ar rs libfoo.a *.o 3) environment 3. (generate symbol index) (ranlib libfoo.a) 4) automatic B) linking against library: Assignment: gcc file1.c file2.c -L/my/lib/path -lfoo foo = bar CFLAGS += -g source code MUST come before library references... foo ?= bar (allows environent to override explicit) override CC = gcc (allows explicit to override command line) (WRONG!!) gcc -L/my/lib/path -lfoo file1.c file2.c
------Recursive Make ------command details (binutils...rpm): EXE = malloctest ------CPPFLAGS = -Ilib SRCS = main.c ar - archive OBJS = $(subst .c,.o,$(SRCS)) r insert s build index .PHONY: lib t table (list) d delete $(EXE): $(OBJS) lib x extract gcc -o $(EXE) $(OBJS) -Llib -lboutil ../rhd256.bowez_notes.2006-07-20.txt Thu Jul 20 15:31:29 2006 10 ar rs libfoo.a *.o B) linking against library at compile time:
------gcc file1.c file2.c -L/my/lib/path -lfoo
ranlib - generate symbol index (gcc -shared (dflt) or -static if both foo’s exist)
ranlib libfoo.a C) loading library at runtime:
------either: 1) update /etc/ld.so.conf and run ldconfig nm - list symbols from an archive (creates /etc/ld.so.cache) or: symbol details: (lowercase = local, uppercase = global) 2) set $LD_LIBRARY_PATH environment variable
A absolute ------B BSS (uninitialized data) section D initialized data section compiling the code: T text (code) section U undefined -fPIC = "Placement Independent Code"
nm libfoo.a (avoid -fpic)
------
objdump - examine object info building the library:
-a archive headers gcc -shared -Wl,-soname,libfoo.so.1 -o libfoo.so.1.0 *.o -lc -f file headers -h section headers (or, ld -soname=libfoo.so.1 -o libfoo.so.1.0) -x all headers
-d, -S dissassemble the symlink complexity: -s dump contents libfoo.so (used in subsequent linking) etc... libfoo.so.1 (the soname - built into library, and executable) libfoo.so.1.0 (the actual library implementation) objdump -S libfoo.a ------linking to the library size - report section sizes gcc file1.c file2.c -L/my/lib/path -lfoo size libfoo.a (note that it makes use of libfoo.so) ------
strings - extract (ascii) strings from a file ------
strings libfoo.a ld-linux.so
------dynamically shared library, implicitly linked to at compile time. - responsible for loading all other so’s. shared libraries documentation: man ld.so(8) ------search path: a) $LD_LIBRARY_PATH A) building the library: b) DT_RUNPATH dynamic section of executable c) /etc/ld.so.cache 1. compile code gcc -fPIC -c *.c d) /lib, then /usr/lib
2. build so other interesting enviornment varialbes:
gcc -shared -Wl,-soname,libfoo.so.1 -o libfoo.so.1.0 *.o $LD_PRELOAD - load libraries first (used to override symbols) ../rhd256.bowez_notes.2006-07-20.txt Thu Jul 20 15:31:29 2006 11 Build Steps: ------aclocal: (expands automake macros for autoconf) ldconfig configure.in shared object maintenance: [acinclude.m4] -- aclocal --> aclocal.m4 [user macros] 1) build /etc/ld.so.cache file 2) update symlinks in library directories autoconf: 3) report current cache configure.in [aclocal.m4] -- autoconf --> configure ldconfig - updates /etc/ld.so.cache, sets needed links ldconfig -p - print current cache automake:
How to register library with system configure.in [COPYING, INSTALL] makefile.am -- automake --> Makefile.in, stamp-h.in 1) Place in directory known about by /etc/ld.so.conf [config.{guess,sub}] 2) run ldconfig mkinstalldirs, missing, install-sh
------readelf configure.in ------readelf -a libfoo.so readelf -a a.out - shell code should be portable
standard ordering: # vi: ts=4 1) Boilerplate: ==> ./09_ac/notes.txt <== AC_INIT, AC_INIT_AUTOMAKE, AC_CONFIG_HEADER, AC_REVISION http://www-src.lip6.fr/homepages/Alexandre.Duret-Lutz/autotools.html 2) Options: small project AC_ARG_ENABLE(... ------Programmer Inputs: Makefile.am, configure.in 3) Programs: ------AC_CHECK_PROG Makefile.am: bare bones project requirements AC_PATH_TOOL
- what has to be built 4) Libraries (required for other checks) - where installed - only libc, libm, libX11 can be taken for granted configure.in: - generally only effect Makefiles
- template macros and shell fragments AC_CHECK_LIB
- mandatory init macros 5) Headers
AC_INIT(main.c) 6) Typedefs and structures AM_INIT_AUTOMAKE(foonly, 1.0) 7) Functions - build program requirements AC_REPLACE_FUNCS(strstr) AC_PROG_CC AC_CHECK_FUNCS(inet_aton inet_addr, break) AM_PROG_LEX AC_PROG_YACC 8) Output: AC_OUTPUT(..)
- output (foo.in --> foo) (?) Targeting the System instead of Feature: AC_OUTPUT(Makefile) - avoid if possible ../rhd256.bowez_notes.2006-07-20.txt Thu Jul 20 15:31:29 2006 12 RPM - Red Hat Package Management AC_CANONICAL_{HOST,SYSTEM}: sets $host variable ------case ${host} in i[[3456]]86-*-linux-gnu*) do foo;; ------sparc*-sun-solaris*) do foo;; composed of multiple components mips*-*-elf*) do foo;; ------esac 1. local database: /var/lib/rpm/ - note doubling of square brackets to survive m4 2. rpm executable 3. package files: foo-1.2-4.i386.rpm ------automake (4. metadirectories and RHN and yum) ------
1) build ------2) check The rpm executable 3) clean ------4) install 5) distribute major modes:
general rules: system admin: -i, --install # comment passed through -U, --upgrade ## comment isn’t - used to comment Makfile.am -F, --freshen -e, --erase include $(top_srcdir)/config/Make-rules - top of source tree -q, --query include $(srcdir)/Make-rules - local directory -V, --verify
- subdirs not easily handled (??) building: rpmbuild -b - avoid CFLAGS and friends, use AM_CFLAGS, AM_CXXFLAGS, etc... ------primaries: installing ------DATA HEADERS -i install package file SCRIPTS -U upgrade (completely remove old, completely install new) MANS -F freshen (upgrade if alreay installed, ignore otherwise) TEXINFOS -h hash marks PROGRAMS -v verbose information LIBRARIES --nodeps, --replace-files, --force LTLIBRARIES rpm -ihv foo-1.2-4.i386.rpm bin_PROGRAMS = foo rpm -Uhv http://server1/pub/RedHat/RPMS/foo-1.2-4.i386.rpm foo_SOURCES = main.c rpm -ihv ftp://server1/pub/RedHat/RPMS/foo-* foo_LDADD = @FOO_OBJ@ rpm -Fhv ftp://server1/pub/updates/*.rpm foo_DEPENDENCIES = @FOO_OBJ@ EXTRA_foo_SOURCES = foo.c -e erase
subdirectories: rpm -e foo
SUBDIRS = . m4 tests ------querying ------
All queries consist of two parts:
rpm -q (what pkgs) (what info) # vi: ts=4 package selection options: ==> ./10_rpm/notes.txt <== ------a all installed pkgname package name ../rhd256.bowez_notes.2006-07-20.txt Thu Jul 20 15:31:29 2006 13 -f filename package which owns a file | |-- gzip-1.3.tar.gz -p pkgfilename directly query package file | ‘-- lsof_4.51_W.tar.gz |-- SPECS spec file (recipe for building) information options: | |-- gzip.spec | ‘-- lsof.spec (default) package name ‘-- SRPMS -i info header -l list of files --requires installation requirements --scripts install/post install scripts ------recipe for an open source build --queryformat "%10{size} %{name}\n" ------
------Traditionally, there has been a recipe for building open verifying source software: ------1) obtain sources and patches installed package verification: 2) untar source and apply patches 3) ./configure rpm -V samba 4) make rpm -Va 5) make install
package file signature verificiation This recipe is anticipated by the rpm build design
(rpm --import /mnt/cdrom/RPM-GPG-KEY) ------rpm --checksig foo-1.2-4.i386.rpm The RPM spec file ------Major portions: Source RPMs and RPM building Header - database info ------%description - text paragraph(s) %prep - extract sources and apply pattches ------%build - compile installing source RPMS: %install - install products into $RPM_BUILD_ROOT ------%clean - cleanup $RPM_BUILD_ROOT %{post,pre}{,un} - scripts to run on the target rpm -ihv /mnt/cdrom/SRPMS/foo-1.2-3.src.rpm %files - itemize all files %changelog - log changes - source rpms’s are not databased - source rpm’s install completely into /usr/src/redhat ------Headers ------structure of /usr/src/redhat # ------# Required Datbase Fields # after installing the lsof and gzip srpms: # Name: Package Name # Summary: Package Name . # Version: Should Reflect Version of "Pristine Source" |-- BUILD used during build process # Release: Should Reflect "Internal" Revision (patches) |-- RPMS stores completed RPM # License: | |-- athlon # Group: "Approved" Groups in /usr/share/doc/rpm-*/GROUPS | |-- i386 Summary: The GNU data compression program. | |-- i486 Name: gzip | |-- i586 Version: 1.3.3 | |-- i686 Release: 13 | ‘-- noarch License: GPL |-- SOURCES "pristine" source and patches Group: Applications/File | |-- gzip-1.2.4-zforce.patch | |-- gzip-1.2.4a-dirinfo.patch # | |-- gzip-1.3-mktemp.patch # Optional Fields; URL, ... | |-- gzip-1.3-stderr.patch # | |-- gzip-1.3-zgreppipe.patch URL: http://www.gzip.org/ ../rhd256.bowez_notes.2006-07-20.txt Thu Jul 20 15:31:29 2006 14 %setup -q # %patch0 -p1 # Identify Sources and Patches - only filname is used, all %patch1 -p1 # are expected to be found in %%{_topdir}/SOURCES #patch2 -p1 # %patch3 -p1 Source: ftp://alpha.gnu.org/gnu/gzip/gzip-%{version}.tar.gz %patch4 -p1 -b .nixi Patch0: gzip-1.3-openbsd-owl-tmp.diff %patch5 -p1 -b .rsync Patch1: gzip-1.2.4-zforce.patch %patch6 -p1 -b .window-size Patch2: gzip-1.2.4a-dirinfo.patch Patch3: gzip-1.3-stderr.patch ------Patch4: gzip-1.3.1-zgreppipe.patch build Patch5: gzip-1.3-rsync.patch ------Patch6: gzip-1.3.3-window-size.patch # # # compile the source in %%{_topdir}/BUILD directory. # Prereq: must be on target _before_ installation # # Requires: must be on target _after_ installation # helpful macros: # BuildPrereq:must be on build host # # # %%configure - run configure script Prereq: /sbin/install-info # Requires: mktemp less %build # export DEFS="-DNO_ASM" # Buildroot: Directory on build host that servers as root directory export CPPFLAGS="-DHAVE_LSTAT" # for target system - files should be "installed" to this directory. %configure --bindir=/bin # # usually formulaic: %%{_tmppath}/%{name}-%{version}-root make # make gzip.info Buildroot: %{_tmppath}/gzip-%{version}-root ------install Description ------# # easy enough... # install products into $RPM_BUILD_DIR # %description The gzip package contains the popular GNU gzip data compression %install program. Gzipped files have a .gz extension. rm -rf ${RPM_BUILD_ROOT} %makeinstall bindir=${RPM_BUILD_ROOT}/bin Gzip should be installed on your Red Hat Linux system, because it is a mkdir -p ${RPM_BUILD_ROOT}%{_bindir} very commonly used data compression program. ln -sf ../../bin/gzip ${RPM_BUILD_ROOT}%{_bindir}/gzip ln -sf ../../bin/gunzip ${RPM_BUILD_ROOT}%{_bindir}/gunzip ------prep for i in zcmp zegrep zforce zless znew \ ------gzexe zdiff zfgrep zgrep zmore ; do mv ${RPM_BUILD_ROOT}/bin/$i ${RPM_BUILD_ROOT}%{_bindir}/$i # done # goal of prep faze is to extract and patch sources into the # %%{_topdir}/BUILD directory. gzip -9nf ${RPM_BUILD_ROOT}%{_infodir}/gzip.info* # # helpful macros: # cat > ${RPM_BUILD_ROOT}%{_bindir}/zless <
%preun provided by rcs...rpm if [ $1 = 0 ]; then /sbin/install-info --delete %{_infodir}/gzip.info.gz %{_infodir}/dir Collection of commands: fi ci check in %files co check out %defattr(-,root,root) rcsdiff diff’s between revisions and working copy %doc NEWS README AUTHORS ChangeLog THANKS TODO rlog revision log /bin/* %{_bindir}/* What RCS provides: %{_mandir}/*/* %{_infodir}/gzip.info* revision history %attr(775,root,uucp) /bin/grep revision log %dir /var/log/grep/ probhibits concurrent (and possible contradictory) modifications
%changelog Common concepts: * Tue Jun 15 2004 Elliot Lee
* Fri Feb 13 2004 Elliot Lee
------$Id: notes.rcs,v 1.1.1.1 2004/02/21 22:35:13 bowe Exp $ lab guidelines $Log: notes.rcs,v $ ------Revision 1.1.1.1 2004/02/21 22:35:13 bowe initial import of new CVS tree 1) install gzip SRPM from server1 Revision 1.1 2001/11/27 14:24:31 bostrick 2) tar up your sources 06 cvs/rcs notes tar cvzf myproj-1.1.tar.gz myproj-1.1/ 07 rpm notes
3) move tar of sources to /usr/src/redhat/SOURCES static char rcsid[] = "$Id: notes.rcs,v 1.1.1.1 2004/02/21 22:35:13 bowe Exp $ 4) cp gzip.spec to myproj.spec, and customize for your project ";
Source: myproj-1.1.tar.gz locked versus unlocked copies ---Patch0:--- delete -l locked copy %prep -u unlocked copy (default) %setup ------%build make CVS - concurrent versioning system
%install ------mkdir $RPM_BUILD_ROOT mkdir $RPM_BUILD_ROOT/bin provided by cvs...rpm mkdir $RPM_BUILD_ROOT/lib mv myexe $RPM_BUILD_ROOT/bin single front end command, several modes: mv mylib.a $RPM_BUILD_ROOT/lib cvs 5) rpm -bb myproj.spec checkout commit (checkin) diff # vi: ts=4 log
../rhd256.bowez_notes.2006-07-20.txt Thu Jul 20 15:31:29 2006 16 update release 2) import into the repository, specifying repository path to your add root directory, "vendor tag", and "start tag" remove cvs import proj/foo gls init export import 3) mv your project directory to the side
admin cd ..; mv foo /tmp/foo.o init 4) create and cd to a sandbox directory ------repositories mkdir ˜/sandbox ------cd ˜/sandbox
local /usr/local/cvsroot 5) check out your project pserver :pserver:[email protected]:/usr/local/cvsroot rsh :ext:[email protected]:/usr/local/cvsroot cvs co proj/foo
specify via -d or $CVSROOT # vi: ts=4
(note that CVS_RSH=ssh causes ssh to be used instead of rsh) ==> ./13_prof/notes.txt <== ------sample session Oprofile ------export CVSROOT=:ext:elvis@server1:/usr/local/cvsroot export CVS_RSH=ssh http://oprofile.sourceforge.net/
cvs checkout gls/curricula/rhd143 - System wide Profiler
...make edits... - Kernel driver - Daemon cvs update - Post Profiling Analysis Tools
...make edits... ------quick start cvs update ------cvs commit - boot to smp version of the kernel cvs release - install oprofile rpm - opcontrol --vmlinux=/path/to/vmlinux (or opcontrol --no-vmlinux) ------opcontrol --start setting up a CVS server ------(run objects to be profiled)
1) make repository directory w/ appropriate perms - opreport -l /path/to/mybinary - opannotate --source --output-dir=/path/to/annotated-source /path/to/mybinary mkdir /usr/local/cvs - opreport chgrp cvs /usr/local/cvs chmod 2775 /usr/local/cvs ------reporting 2) initialize the directory ------
cvs -d /usr/local/cvs init opreport opreport -l ------opreport --long-filenames -l importing a project to CVS ------opannotate --source --output-dir=annotated /usr/local/bin/gzip
1) Start in base directory of your project ==> ./20_mmap/notes.txt <== cd $HOME/foo ------../rhd256.bowez_notes.2006-07-20.txt Thu Jul 20 15:31:29 2006 17 fundamental data structure: directories struct dirent ------{ long d_ino; /* inode number */ ------off_t d_off; /* offset to next dirent */ what are directories? unsigned short d_reclen; /* length of this dirent */ ------char d_name [NAME_MAX+1]; /* file name (null-terminated) */ } directories contain list of dentries only the d_name and d_ino are useful, only d_name is portable. dentries map a filename to an inode primitives: note that a directory does not know it’s own name! it calls itself "." DIR *opendir(const char *name); [root@server1 root]# mkdir foo struct dirent *readdir(DIR *dir); [root@server1 root]# touch foo/file{1,2} int closedir(DIR *dir); [root@server1 root]# mkdir foo/dir1 void rewinddir(DIR *dir); [root@server1 root]# touch foo/dir1/file{18,19} repositioning: [root@server1 root]# ls -ai1 foo off_t telldir(DIR *dir); 1302 . void seekdir(DIR *dir, off_t offset); 162955 .. 1305 dir1 filtering: 1303 file1 1304 file2 int scandir(const char *dir, struct dirent ***namelist, int (*select)(const struct dirent *), [root@server1 root]# ls -ai1 foo/dir1 int (*compar)(const struct dirent **, const struct dirent **) ); 1305 . int alphasort(const struct dirent **a, const struct dirent **b); 1302 .. 1306 file18 for every entry in DIR, if select return non-zero, 1307 file19 store the name in namelist, sorted by compar.
consider the real world example: ------Directory limitations [root@server1 root]# ls -al /var/log ------
total 3228 entries are stored as linear list, large direcories drwxr-xr-x 7 root root 4096 Nov 28 11:58 . take a *long* time to process. drwxr-xr-x 19 root root 4096 Nov 26 12:04 .. -rw------1 root root 1041 Nov 27 09:48 boot.log compare timing of -rw------1 root root 4928 Nov 28 11:01 cron -rw-r--r-- 1 root root 5081 Nov 26 07:13 dmesg ls -l /etc | grep foo drwxr-xr-x 2 root root 4096 Jun 24 22:09 fax ls -l /dev | grep foo drwxr-xr-x 2 root root 4096 Nov 13 12:15 gdm drwxr-xr-x 2 root root 4096 Nov 26 09:45 httpd (note, if your disk cache is "hot", there might not be an -rw-r--r-- 1 root root 58109 Nov 26 07:13 ksyms.0 impressive difference) -rw-r--r-- 1 root root 19136220 Nov 28 11:19 lastlog -rw------1 root root 680 Nov 28 04:02 maillog how does netscape cacheing avoid this problem? -rw------1 root root 212458 Nov 28 11:34 messages -rw-r--r-- 1 root root 16866 Nov 28 04:02 rpmpkgs [root@server1 root]# ls -F .netscape/cache/ drwx------2 root root 4096 Aug 13 15:54 samba 00/ 03/ 05/ 0F/ 11/ 15/ 17/ 19/ 1D/ 1F/ -rw------1 root root 95495 Nov 28 11:34 secure 01/ 04/ 0E/ 10/ 14/ 16/ 18/ 1C/ 1E/ index.db -rw-r--r-- 1 root root 647225 Nov 28 10:16 up2date drwxr-xr-x 2 root root 4096 Aug 27 18:35 vbox ------rw-rw-r-- 1 root utmp 253440 Nov 28 11:34 wtmp adding/removing filesystem entries -rw-r--r-- 1 root root 21023 Nov 28 09:40 XFree86.0.log ------
------int mkdir(const char *pathname, mode_t mode); directory access int rmdir(const char *pathname); ------../rhd256.bowez_notes.2006-07-20.txt Thu Jul 20 15:31:29 2006 18 the process’s current working directory 1+ read locks y n ------1 write lock n n
char *getcwd(char *buf, size_t size); - locks the inode, not the descriptor int chdir(const char *path); int fchdir(int fd); locks are not duplicated across fork(), dup()
------current locks can be examined in /proc/locks memory mapping ------using fcntl() ------
MAP_FIXED - supplied address must be used, must be page aligned #include
(1) duplicate existing file descriptor (F_DUPFD) ------(2) get/set descriptor flags (F_{G,S}ETFD) using flock() (3) get/set file status flags (F_{G,S}ETFL) ------(4) get/set asynchronous IO param (F_{G,S}ETOWN, F_{G,S}ET_SIG) (5) get/set record locks (F_{G,S}ETLK, F_SETLKW) #include
descriptor flags: FD_CLOEXEC : close on exec (0 or 1) int flock(int fd, int operation)
file status flags: LOCK_{SH,EX,UN}
O_{RD,WR}ONLY, O_RDWR - simpler, kernel implements using fcntl() mechanism O_APPEND, O_NONBLOCK, O_SYNC, O_ASYNC # vi: ts=4 ------==> ./21_sig/notes.txt <== file locking ======
------signals
------======introduction ------1. introduction 2. raising signals 3. alarms, pausing, and sleeping - implementations 4. signal dispositions 5. issues with signal implementation Advisory - only enforced if processes look for the lock 6. blocking signals and terminolgoy Mandatory - enforced on all processes (requires mount option) 7. signal sets, sigprocmask(), sigsuspend() 8. sigaction() - types: read (shared), write (exclusive) ------read write introduction no locks on file y y ../rhd256.bowez_notes.2006-07-20.txt Thu Jul 20 15:31:29 2006 19 ------library fcn, essentially alarm() and then pause(). signals are essentially "software interrupts"
sources of signals: higher precission versions:
- terminal interrupts (^C --> SIGINT, ^\ --> SIGQUIT) void usleep(unsigned long usec); int getitimer(int which, struct itimerval *value); - hardware exceptions (SIGBUS, SIGFPU) int setitimer(int which, const struct itimerval *value, struct itimerval *ovalue); - kill() syscall, kill command (SIGHUP, SIGUSR1, SIGUSR2) struct itimerval { - software conditions struct timeval it_interval; /* next value */ struct timeval it_value; /* current value */ SIGCHLD - child died }; SIGALRM - generated with alarm() syscall SIGURG - OOB transmissions over sockets struct timeval { SIGWINCH - gui window changed size long tv_sec; /* seconds */ SIGPIPE - write to closed pipe long tv_usec; /* microseconds */ ... };
------three different timers:
raising signals ITIMER_REAL - decrements in realtime (SIGALRM) ITIMER_VIRTUAL - decrements in process time (SIGVTALRM) ------ITIMER_PROF - decrements in process and kernel time (SIGPROF)
int kill(pid_t pid, int sig); int raise (int sig); ------
raising signals is easy... signal dispositions, signal()
sig: signal number ------
pid: (same convetions as waitpid) 1) kernel default: terminate, ignore, "puke", suspend, continue (SIG_DFL)
pid < 0 : pgid = |pid| itemized in man signal(7) pid == -1: any child pid == 0 : my pgid 2) ignore the signal (SIG_IGN) pid > 0 : pid 3) implement custom signal handler with signal(). raise() - send signal to myself
------void (*signal(int signum, void (*sighandler)(int)))(int);
alarms, pausing, and sleeping ... or (easer to understand)... ------
unsigned int alarm(unsigned int seconds); typedef void (*sighandler_t)(int signum); int pause(void); sighandler_t signal(int signum, sighandler_t handler); unsigned int sleep(unsigned int seconds);
alarm(): - upon recieving signal, the handler will be called, with the the signal number as it’s lone argument. - after so many seconds, kernel sends SIGALRM to process - returns immediatly - signal returns previous signal handler. - only one outstanding alarm allowed per process - predefined macros: pause(): SIG_DFL - implement default signal handler - process sleeps until signal is received SIG_IGN - ignore signal SIG_ERR - signal command error (-1) sleep(): ../rhd256.bowez_notes.2006-07-20.txt Thu Jul 20 15:31:29 2006 20 ------a signal is handled by the receiving process
issues with the signal() implementation - how handled depends on "disposition" of receiving process.
------+ kernel default + ignored 1) ripe for reentrancy issues + custom signal handler
stick to "reentrant safe" functions w/in signal handler note that "blocking" is not the same as "ignoring"
- avoid global variables repeated raising of a signal is idempotent. - avoid stdio library (multiple signals are *not* delivered, only one signal is delivered) 2) race conditions ------consider the following code snippet (assume SIGALRM handler in place): signal sets, sigprocmask(), sigpending(), and sigsupend() void sleep(int secs) { alarm(secs); ------pause(); } signal sets:
what happens if SIGALRM is recieved between alarm and pause? opaque structure of type sigset_t potentially pause forever! operated on with the following functions: - cannot examine current signal handler without resetting it sig{empty,fill}set(sigset_t s); sighan = signal(SIGUSR1, SIG_DFLT); sig{add,del}set(sigset_t s, int signum); sighan = signal(SIGUSR1, sighan); sigismember(sigset_t s, signum);
3) system calls int sigprocmask(int how, const sigset_t *set, sigset_t *oldset); how to handle an interrupted system call? sets the current state of blocked signals (the "mask"). EINTR can be returned how: ERESTART (or varients of) used internally within kernel SIG_BLOCK new = current | set SIG_UNBLOCK new = current & ˜set ------SIG_SETMASK new = set
blocking signals and signal terminology oldset = previous state
------if set == NULL, only return current state in oldset
signal delivery can be temporarily suspended, or "blocked". int sigpending(sigset_t *set); signal terminology:
"raised" set returns with current state of pending signals - a process has generated the signal
"pending" int sigsuspend(const sigset_t *mask); - the raised signal is not yet delivered - commonly because the signal is being "blocked" atomically: - block signals specified in mask "blocked" - pause until new signal is raised - used in same context as "disposition" (the receiving end) - if a blocked signal were to be raised, it would be "pending" - functionally, an atomic replacement for pause(). until "unblocked" ------"mask" - a process’s current state of blocked signals better registering of signal handlers: sigaction()
"delivered" ------../rhd256.bowez_notes.2006-07-20.txt Thu Jul 20 15:31:29 2006 21 - maximum queue size defined by /proc/sys/kernel/rtsig-max "sigaction" handler is more sophisticated than "signal" handler: # vi: ts=4 void (*sa_handler)(int signum); void (*sa_sigaction)(int signum, siginfo_t *siginfo, void *ptr); ==> ./22_pthd/notes.txt <== sigactions are registered with the sigaction() command, that makes use ======of the following data structure: threads struct sigaction { void (*sa_handler)(int); ======void (*sa_sigaction)(int, siginfo_t *, void *); sigset_t sa_mask; int sa_flags; ------void (*sa_restorer)(void); } __clone()
sa_handler and sa_sigaction are often implemented as a union ------
- true for Linux - why would you want to use both anyway? int __clone(int (*fn) (void *arg), void *child_stack, int flags, void *arg)
sa_mask is additional signals to mask linux specific system call
- masked signals are blocked during exection of signal handler child process (thread) terminates on exit from fn(arg) - the signal being handled is _automatically_ blocked. child_stack points to *top* of stack memory space provided for child sa_flags flags: SA_NOCLDSTOP: if signum == SIGCHLD, ignore if child is just stopped CLONE_VM - same memory CLONE_FS - same filesystem context (cwd, root, etc...) SA_ONESHOT, SA_RESETHAND: CLONE_FILES - same descriptor table restore to kernel default after handling CLONE_SIGHAND - same signal dispositions CLONE_PID - same process id SA_RESTART: certain system calls automatically restart across signals ------
SA_NOMASK, SA_NODEFER: pthreads library do not block signal from it’s own handler ------SA_SIGINFO: fill out extended siginfo information implements thread management in userspace
incorporated into glibc
int sigaction(int signum, const struct sigaction *act, struct sigaction *oldact); to use:
note intelligent implementation of returning old action. 1) Include
2) compile with -D_REENTRANT, link with -lpthread ------or real time signals 2) compile with -pthread ------available in the 2.4 kernel pthread primitive structures - signals between SIGRTMIN (32) and SIGRTMAX (63) ------reliably delivered every time raised (even if blocked) 1) threads ../rhd256.bowez_notes.2006-07-20.txt Thu Jul 20 15:31:29 2006 22 2) mutexes - synchronization 3) conditionals - signaling detachstate: joinable, detached schedpolicy: other, rr(rt), fifo(rt) ------schedparam: priority inheritsched: inherit sched params from parent? basic thread operations scope: scheduling scope: system, process(not implemented) ------mutexes: int p_create(p_t *thread, p_attr_t *attr, kind_np: "fast", "recursive", or "error checking" void *(*start_fn)(void *), void *arg); conditionals: int p_join(p_t th, void **thread_return); N/A
void p_exit(void *retval); ------
int p_cancel(p_t thread); synchronization, invariants, and critical sections
int p_detach(p_t thread); ------
invariants: assumed data conditions - basic thread creation/removal, reminescent of following: example: linked lists p_create: fork and exec p_join: wait 1) a pointer refers to the first list elem p_exit: exit p_cancel: kill -15 2) each element (except the last) contains a reference to the next element
- p_create() add_to_list(struct foo *item, struct foo *head) {
- essentially front end for __clone foo->next = head head = foo - provides its own statck management } - p_join() and p_exit() example: stacks - p_exit() recieves a return value pointer if a stack is implemented as the array int data[100], - p_join() provides a "handle" to receive return value pointer there will be an idx called "top" that points to the next available stack entry. - documented in libc info pages note that the following function temporarily breaks this condition:
------typedef stack { int data[100], int top } stack_t;
posix attribute handling void push(stack_t *s, int item) { s->data[s->top++] = item;}
------where is the (subtle) break?
int p_attr_init(p_attr_t *attr); example: increment int p_attr_destroy(p_attr_t *attr); i++ int p_attr_setATTR(p_attr_t *attr, int policy); int p_attr_getATTR(p_attr_t *attr, int *policy); critical sections:
sections of code that temporarily break invariants. general mechanism: critical sections should be constructed such that breaking of p_{,mutex,cond}attr_{init,destroy}(p_{,mutex,cond}attr_t *attr) invariants occurs _atomically_.
attributes: ------
pthreads: mutexes ../rhd256.bowez_notes.2006-07-20.txt Thu Jul 20 15:31:29 2006 23 s->data[s->top++] = data; ------// but i have to restore them by now int p_mutex_init(p_mutex_t *mutex, const p_mutexattr_t *mutexattr); int p_mutex_destroy(p_mutex_t *mutex); p_mutex_unlock(&mutex);
int p_mutex_lock(p_mutex_t *mutex); int p_mutex_trylock(p_mutex_t *mutex); ------int p_mutex_unlock(p_mutex_t *mutex); conditionals attributes: ------kind_np: "fast", "recursive", or "error checking" (non posix)
initializers: int p_cond_init (p_cond_t *COND, p_condattr_t *cond_ATTR) int p_cond_destroy (p_cond_t *COND) p_mutex_t fastmutex = P_MUTEX_INITIALIZER; p_mutex_t recmutex = P_RECURSIVE_MUTEX_INITIALIZER_NP; int p_cond_broadcast (p_cond_t *COND) p_mutex_t errchkmutex= P_ERRORCHECK_MUTEX_INITIALIZER_NP; int p_cond_signal (p_cond_t *COND) int p_cond_wait (p_cond_t *COND, p_mutex_t *MUTEX) initialize into the unlocked state int p_cond_timedwait (p_cond_t *COND, p_mutex_t *MUTEX, const struct timespec *ABSTIME) lock:
if unlocked, lock the mutex used to signal other threads about possible changes in condition
if locked: always associated with a mutex
if lock held by another thread: example: block until unlocked, the lock and return int i; if lock held by me: p_mutex_init(MUTEX); p_cond_init(COND, cond_ATTR); recursive: increment lock and continue errorchecking: return w/ EDEADLCK thread 1: "i must make sure i does not grow to big" fast: block until unlocked. err... p_mutex_lock(MUTEX) trylock: while (i < 10) { same as lock, but will not block p_cond_wait(COND, MUTEX); } unlock: // modfiy i unlock the mutex i -= 5;
if errorchecking, enforce that lock is held by me p_cond_broadcast(COND); p_mutex_unlock(MUTEX);
Example: thread 2: p_mutex_t mutex; p_mutex_attr mattr; p_mutex_lock(MUTEX);
p_mutex_attr_init(&mattr); // modify i p_mutex_init(&mutex, &mattr); i++;
p_cond_broadcast(COND); ...... p_mutex_unlock(MUTEX);
p_mutex_lock(&mutex);
// now i may break invariants Conditionals and the Thundering Hurd ../rhd256.bowez_notes.2006-07-20.txt Thu Jul 20 15:31:29 2006 24 + "memory coherence" between multiple processors - pthread_cond_broadcast() wakes all waiters + "read/write ordering"
- pthread_cond_signal() wakes at most one + "memory barriers"
- any writes before barrier are valid after barrier - barrier is moving wall, not a "cache flush" command ------memory controller cannot "look past" barrier until it has completed actions before barrier advanced topics... Unix Systems for Modern Architectures [Schimmel, 1994] ------# vi: ts=4 avoiding deadlock ==> ./24_pam/notes.txt <== 1) acquisition heirarchy ------2) try and backoff What is PAM? 3) lock chaining (if algorithm allows it) ------often useful in tree traversal acquire 1 Pluggable Authentication Modules acquire 2 release 1 Keep user authentication is common set of libraries acquire 3 release 2 /lib/security/ binary modules /etc/pam.d/ configuration files /etc/security/ supplementary cfg files memory visibility /usr/share/doc/pam-*/ documentation
1) context before (at point of) p_create will be seen by Each library implements some "nugget" of authentication policy new thread. memory changes after p_create are not garenteed. pam_listfile.so is the user listed in a file? 2) context seen by thread unlocking mutex will be seen by pam_localuser.so is the user local? thread who subsequently locks mutex. pam_nologin.so does /etc/nologin exist? pam_rhosts_auth.so is there a proper ˜/.rhost file? 3) context when thread terminates will be seen by join. pam_rootok.so is the use root? pam_time.so is the right time of day? 4) context seen at signal/broadcast will be seen by conditionals awoken from signal. pam_unix.so the "standard unix thing"
contrast: ------PAM configuration ------lock(mutex1) a = 1; lock(mutex1) 4 types of applicaiton calls: b = 2; unlock(mutex1) auth is user who they claim to be? locala = a account other reasons resources inaccessable (quota? time of day?) localb = b password user is changing credentials (change password) unlock(mutex1) session any other session init or teardown localb = 2 ------3 common types of library policies:
lock(mutex1) requried must pass all required librarires a = 1; lock(mutex1) sufficient if passed, user is done. if failed, user keeps going unlock(mutex1) optional has no bearing on final authentication b = 2; locala = a localb = b /etc/pam.d/login: unlock(mutex1) localb = ? #%PAM-1.0 ------auth required pam_securetty.so auth required pam_stack.so service=system-auth + a memory address can only hold one value - don’t auth required pam_nologin.so let threads "race" to get there first account required pam_stack.so service=system-auth password required pam_stack.so service=system-auth ../rhd256.bowez_notes.2006-07-20.txt Thu Jul 20 15:31:29 2006 25 session required pam_selinux.so multiple session required pam_stack.so service=system-auth session optional pam_console.so const char *pam_strerror(pam_handle_t *pamh, int errnum);
/etc/pam.d/system-auth: ------authenticating the user #%PAM-1.0 ------# This file is auto-generated. # User changes will be destroyed the next time authconfig is run. int pam_authenticate(pam_handle_t *pamh, int flags); auth required /lib/security/pam_env.so auth sufficient /lib/security/pam_unix.so likeauth nullok flags: PAM_SILENT, PAM_DISALLOW_NULL_AUTHTOK auth required /lib/security/pam_deny.so return values: account sufficient /lib/security/pam_succeed_if.so uid < 100 account required /lib/security/pam_unix.so PAM_SUCCESS PAM_AUTH_ERR password requisite /lib/security/pam_cracklib.so retry=3 PAM_CRED_INSUFFICIENT password sufficient /lib/security/pam_unix.so nullok md5 shadow PAM_AUTHINFO_UNAVAIL password required /lib/security/pam_deny.so PAM_USER_UNKNOWN PAM_MAXTRIES session required /lib/security/pam_limits.so session required /lib/security/pam_unix.so ------setting user credentials - something the user possesses ------application writers interface ------int pam_setcred(pam_handle_t *pamh, int flags);
------Initialization and Tear Down flags: ------PAM_SILENT PAM_{ESTABLISH,DELETE,REINTIALIZE,REFRESH}_CRED
int pam_start(const char *service_name, const char *user, returns: const struct pam_conv *pam_conversation, PAM_SUCCESS pam_handle_t **pamh); PAM_USER_UNKNOWN PAM_CRED_{UNAVAIL,EXPIRED,ERR} int pam_end(pam_handle_t *pamh, int pam_status); ------account management PAM Items ------int pam_acct_mgmt(pam_handle_t *pamh, int flags); int pam_set_item(pam_handle_t *pamh, int item_type, const void *item); flags: PAM_SILENT, PAM_DISALLOW_NULL_AUTHTOK
int pam_get_item(const pam_handle_t *pamh, int item_type, returns: const void **item); PAM_SUCCESS PAM_AUTHTOKEN_REQD Items: PAM_ACCT_EXPIRED PAM_SERVICE PAM_AUTH_ERR PAM_USER PAM_PERM_DENIED PAM_USER_PROMPT PAM_USER_UNKNOWN PAM_TTY PAM_RUSER ------PAM_RHOST updating authentication tokens PAM_CONV ------PAM_FAIL_DELAY int pam_chauthtok(pam_handle_t *pamh, const int flags); Return: PAM_SUCCESS flags: PAM_SILENT, PAM_CHANGE_EXPIRED_AUTHTOK PAM_SYSTEM_ERR PAM_PERM_DENIED returns: PAM_BUF_ERR PAM_SUCCESS PAM_BAD_ITEM PAM_AUTHTOK_{ERR,RECOVERY_ERR,LOCK_BUSY,DISABLE_AGING} ../rhd256.bowez_notes.2006-07-20.txt Thu Jul 20 15:31:29 2006 26 PAM_PERM_DENIED PAM_TRY_AGAIN time_t pam_misc_conv_warn_time; PAM_USER_UNKNOWN const char *pam_misc_conv_warn_line; time_t pam_misc_conv_die_time; ------const char *pam_misc_conv_die_line; session handling int pam_misc_conv_died; ------# vi: ts=4 int pam_open_session(pam_handle_t *pamh, int flags); int pam_close_session(pam_handle_t *pamh, int flags); ==> ./misc/18_pipe/notes.txt <== flags: PAM_SILENT ======returns: PAM_SUCCESS, PAM_SESSION_ERR pipes ------PAM environment handling ======------int pam_putenv(pam_handle_t *pamh, const char *name_value); const char *pam_getenv(pam_handle_t *pamh, const char *name); pipe() system call
const char * const *pam_getenvlist(pam_handle_t *pamh); ------
- environment user inherits upon login int pipe(int filedes[2]);
------returns two descriptors configured for half-duplex: conversation functions ------filedes[0] - open for reading filedes[1] - open for writing struct pam_conv { int (*conv)(int num_msg, PIPE_BUF - maximum size for atomic operations const struct pam_message **msg, /* potentially array */ struct pam_response **resp, /* single struct */ what happens when one side of the pipe is closed? void *appdata_ptr); void *appdata_ptr; 1) a read from a pipe with closed writers fairly normal: }; return 0 (EOF)
struct pam_message { 2) a write to a pipe with no readers more problematic: int msg_style; const char *msg; return EPIPE }; signal process with SIGPIPE (dflt handler: terminate)
msg_style: ------PAM_PROMPT_ECHO_{ON,OFF} PAM_ERROR_MSG retrieving command output through a pipe PAM_TEXT_INFO PAM_BINARY_{PROMPT,MSG} /* linux extensions */ ------
struct pam_response { - a process wants to run a command, such as "uptime", and reveive the char *resp; command’s output through a pipe int resp_retcode; /* currently not implemented, set to 0 */ }; - now the dup() syscall makes more sense - used to pass a descriptor to stdin/stdout
provided text console conversation function: original process: int misc_conv(int num_msg, const struct pam_message **msgm, struct pam_response **response, void *appdata_ptr); pipe(fd[2]) fork() - prompts user with comments, obtains inputs parent: child: - supports following timeouts (as globally exported vars): close(fd[1]) close(fd[0]) while() { close(1) ../rhd256.bowez_notes.2006-07-20.txt Thu Jul 20 15:31:29 2006 27 read(fd[0]) dup(fd[1]) } close(fd[1]) ==> ./misc/file/notes.txt <== close(fd[0]) exec("uptime") ------
File Handling - how would you arrange to send stdin to the child instead? ------two approaches to file handling (documented in the libc info pages): popen(), pclose() library calls 1) system calls ------base on file descriptor (integer) - provide wrapper for all the hard work done above system calls: open, close, read, write, lseek, (fcntl) FILE *popen(const char *command, const char *type); int pclose(FILE *stream); 2) the stdio (libc) library
akin to system() call: based on opaque FILE object
- creates pipe #include
how pipe is configured depends on type: library calls:
"r": parent FILE <----- child stdout fopen, fclose, fread, fwrite, fseek, ftell "w": parent FILE -----> child stdin getc, fgetc, putc, fputc, getchar, ungetc ------gets, fgets, puts, fputs, putchar fprintf (and friends), fscanf (and friends) named pipes (fifos) fflush, setvbuf (and friends) ------fdopen, fileno int mkfifo (const char *pathname, mode_t mode ); ferror, feof, clearerr mkfifo commandline executable buffering: subtle difference on how opened: either raw, line buffered, or fully buffered standard (blocking): can be set with setvbuf()
open blocks until other process joins examples: read blocks until other process writes stderr: raw buffer O_NOBLOCK (nonblocking): stdout: line buffered when terminal, fully when file default: fully buffered read returns immediately write error w/ ENXIO # vi: ts=4 common example: ==> ./misc/fs/notes.txt <== how to send stdout to two commands simmultaneiously? ------
files and filesystem prog1 --> /bin/tee --> prog2 | ------> FIFO --> prog3 ------file characteristics ------# vi: ts=4 ../rhd256.bowez_notes.2006-07-20.txt Thu Jul 20 15:31:29 2006 28 name ------type file type design user and group owner ------mode (permissions) size regular files data blocks (direct, indirect) atime directories data blocks with dentries mtime symlinks data blocks with a filename ctime num links block dev major/minor number data char dev major/minor number
helpful commands: ls -l, stat FIFO (special) socket (special) ------file types ------Filesystems regular file /etc/passwd directory /etc ------symlink /etc/X11/X Filesystems structure block devices with relevent dentry and inode char device node /dev/tty1 data structures block device node /dev/fd0 Common filesystems in linux: FIFO (named pipe) /dev/initctl socket /tmp/.X11-unix/X0 ext2 the standard linux filesystem msdos vanilla FAT filesystem ------vfat FAT + long filenames traditional unix filesystem design iso9660 CDROM filesystem ------Virtual filesystems: directories contain dentry’s proc implements /proc filename : inode devpts implements /dev/pts filename : inode tmpfs implements /dev/shm filename : inode Journaling filesystems inodes contain all info about file _except_ the name ext3 new w/ RHL 7.2, basically ext2+journalling type (file, dir, symlink, etc...) reiserfs supported but not emphasized in RHL uid/gid xfs SGI version, recently ported to Linux, no RHL support mode jfs IBM version, recently ported to Linux, no RHL support nlinks size Network filesystem {c,m,a}time nfs nfs version 2 and 3, implemented within kernel data block pointers (or filename for symlinks) ------design of ext2 filesystem nlinks: number of dentries pointing to an inode ------
hard links: ln foo foohlink filesystem layout:
multiple dentries pointing to same inode | boot sector | block group 1 | block group 2 | ... cannot span filesystems no concept of original and copy block group layout:
soft links: ln -s foo fooslink superblock group descriptors for all blockgroups single dentry points to single inode inode contains filename used to look up new dentry block bitmap distinct concept of original inode bitmap possibility of dangling links inode table ../rhd256.bowez_notes.2006-07-20.txt Thu Jul 20 15:31:29 2006 29 free blocks - utime/stime/cputime/etc... zeroed in child free inodes - heritage information (ppid, siblings, etc...) used dirs - file locks cleared in child - pending signals/actions cleared data notably what gets inherited by child useful commands: - environment variables e2fsprogs...rpm - open file descriptors - signal handlers mkfs.ext2 (mke2fs) - existing mmaps dumpe2fs tune2fs ------e2label e2fsck Execing - replace memory context with new executable resize2fs ------# vi: ts=4
==> ./misc/proc/notes.txt <== int execve(const char *filenm, char *const argv[], char *const envp[]); ======int execl (const char *path, const char *arg, ...); processes int execlp(const char *file, const char *arg, ...); int execle(const char *path, const char *arg , ..., char * const envp[]); ======int execv (const char *path, char *const argv[]); int execvp(const char *file, char *const argv[]); 1. What’s a process? 2. forking what you lose - 3. execing 4. the libc environment - text, data, heap, stack 5. system - mmaps, singnal handlers 6. wait what you keep ------credentials, heritage, alarms What’s a process? - cwd, umask, root - file locks, signal masks, pending signals 1) execution context - state, priority/niceness, resource acctg, cwd, root - timing statitistics, resource limits 2) environment - environment variables 3) heritage - pid, ppid, siblings, session group, process group what depends - 4) credentials - uid, gids, capabilities 5) I/0 - file descripters, ipc constructs, signal handlers - open file descriptors (FD_CLOEXEC = 0 by dflt) 6) memory management - environment (how exec was called)
demo: ps aux, ls -l /proc/
------libc execution environment
Forking - duplicate process ------
------startup sequence:
pid_t fork(void); libc startup routines
fork return values: - load dynamic libraries - int main(int argc, char* argv[], char* env[]) -1 error 0 child ways to pass information into a process pid > 0 parent 1) command line arguments what differs in child from parent? 2) environment variables 3) stdin - memory space distinct, initialized to that of parent (copy on write) - fork return value registering exit handlers ../rhd256.bowez_notes.2006-07-20.txt Thu Jul 20 15:31:29 2006 30 WIFSIGNALED(status) - true if child terminated by signal int atexit(void (*function)(void)); int on_exit(void (*function)(int , void *), void *arg); WTERMSIG(status) - returns lethal signal number
ways to exit a process WIFSTOPPED(status) - true if child suspended
exit(int status); WSTOPSIG(status) - returns signal number
- no return - options relevant for waitpid(): - calls exit handlers - cleans up stdio buffers WNOHANG - don’t block - return immediatly - calls _exit() - variants _exit(int status); pid_t wait3(int *status, int opts, struct rusage *rusage); - no return pid_t wait4(pid_t pid, int *status, int opts, struct rusage *rusage); - closes file descriptors - children inherited by pid 1 collect resource usage info - parent is sent a SIGCHLD - status returned to parent as return value # vi: ts=4 abort(void); ==> ./misc/sock/notes.txt <== - send SIGABRT to self ======- cleanup stdio sockets (terminate with signal) ======------1. general concepts system() 2. the system calls ------a. socket() b. connect() int system(const char *cmd); c. bind() d. listen() - forks and execs /bin/sh -c cmd e. accept() - significant security risk - avoid w/ suid executables f. shutdown()
------3. protocol families a. unix wait() b. inet (v4)
------4. Internet name resolution
pid_t wait(int *status); 5. socket configuration pid_t waitpid(pid_t pid, int *status, int options); ------block until child terminates, and collect it’s return value general concepts wait - wait until any child dies waitpid - wait until specified child dies ------
pid < 0 : pgid = |pid| ------pid == -1: any child the stages for socket initialization pid == 0 : my pgid ------pid > 0 : pid socket() - information returned in status - examine with macros - generate socket descriptor WIFEXITED(status) - true if child exited - configured for specific addressing family (domain)
WEXITSTATUS(status) - returns return value domain: PF_UNIX, PF_INET, PF_IPX, ... type: SOCK_STREAM, SOCK_DGRAM, SOCK_RAW, ... ../rhd256.bowez_notes.2006-07-20.txt Thu Jul 20 15:31:29 2006 31 bind(s, a) bind() listen(s) for(;;) { - Associate a family specific address w/ the descriptor d = accept(s) fork() connect() close(d) read(d)/write(d) } close(d) connection based types: connect to supplied address close(s)
connectionless types: associate default recipiant address ------listen() - socket() - configure a socket to accept connection requests - only appropriate for connection based protocols ------socket is no longer usable for writing
accept() - int socket(int domain, int type, int protocol);
- wait for new connections (blocking) - generates new descriptor for each new connection domain (family): - only appropriate for connection based protocols Name Purpose Man page ------PF_UNIX,PF_LOCAL Local communication unix(7) how do i use an initialized socket? PF_INET IPv4 Internet protocols ip(7) ------PF_INET6 IPv6 Internet protocols PF_IPX IPX - Novell protocols write() - simplest interface - it’s a file desriptor PF_NETLINK Kernel user interface device netlink(7) PF_X25 ITU-T X.25 / ISO-8208 protocol x25(7) send() - akin to write, but w/ networking options PF_AX25 Amateur radio AX.25 protocol sendto() - send to a supplied address PF_ATMPVC Access to raw ATM PVCs sendmsg() - send special control messages (OOB, error handling, ...) PF_APPLETALK Appletalk ddp(7) PF_PACKET Low level packet interface packet(7) compliments: read()/recv()/recvfrom()/recvmsg() type: ------usual approach SOCK_STREAM - connection based, reliable, duplex, maybe OOB ------SOCK_DGRAM - connectionless, best effort SOCK_SEQPACKET - connection based w/ fixed length requirements client: SOCK_RAW - wysiwyg ... a = {server address} s = socket() protocol: connect(s, a) write(s)/read(s) - family/type may have multipile underlying protocols close(s) - catalogued in /etc/protocols
server: returns socket descriptor or -1
a = {server address} man socket(2), socket(7) s = socket() bind(s, a) ------listen(s) for(;;) { connect() d = accept(s) read(d)/write(d) ------close(d) } close(s) int connect(int sockfd, struct sockaddr *serv_addr, socklen_t addrlen);
forking server: returns 0/-1 on success/failure a = {server address} s = socket() connection oriented protocols call connect once ../rhd256.bowez_notes.2006-07-20.txt Thu Jul 20 15:31:29 2006 32
connectionless protocols can make multiple calls ------
- changes default server association int shutdown(int sockfd, int how);
------closes down half or all of a (full duplex) socket bind() - often call close() instead ------how:
int bind(int sockfd, struct sockaddr *my_addr, socklen_t addrlen); 0 - receiving end 1 - writing end 2 - both returns 0/-1 on success/failure - returns 0/-1 on success/failure usually only required of server, before listen() and accept()
------PF_UNIX: unix domain sockets listen() ------allow processes on the same host duplex communication via filesystem
int listen(int sockfd, int backlog); - man unix(7)
------returns 0/-1 on success/failure socket() ------only applies to connection oriented sockets (SOCK_{STREAM,SEQPACKET}) unix_socket = socket(PF_UNIX, type, 0); generally, backlog ˜ 4 type:
------SOCK_STREAM: stream oriented SOCK_DGRAM: preserves message boundaries accept() (always reliable and sequential) ------
int accept(int sockfd, struct sockaddr *addr, socklen_t *addrlen); ------address format (for bind(), connect(), and accept()): ------parameters #define UNIX_PATH_MAX 108 sockfd: socket set to LISTEN state by listen() struct sockaddr_un { addr: on return, the address of the client sa_family_t sun_family; /* AF_UNIX */ char sun_path[UNIX_PATH_MAX]; /* pathname */ addrlen: };
on entry: the size of the addr structure - for bind() (server), error if named socket ("file") already exists
on return: the size of the supplied address - for connect() (client), error if named socket doesn’t exist
returns new descriptor or -1 on failure - server should unlink named socket when done
------examples: /tmp/.X11-unix/* (if running an X server)
shutdown() (close()) ------../rhd256.bowez_notes.2006-07-20.txt Thu Jul 20 15:31:29 2006 33
IP - IPv4 - convert dotted quat to in_addr, or back
------inet_ntoa returns "statically allocated structure", which can be overwritten (thread safe?) - man ip(7), udp(7), tcp(7), raw(7) - inet_aton has sane way of retuning error ------socket() ------
tcp_socket = socket(PF_INET, SOCK_STREAM, 0); Internet name resolution udp_socket = socket(PF_INET, SOCK_DGRAM, protocol); raw_socket = socket(PF_INET, SOCK_RAW, protocol); ------
------address format: gethostby{name,addr}() - IPV4 specific ------
struct sockaddr_in { #include
/* Internet address. */ struct hostent { struct in_addr { char *h_name; /* official name of host */ u_int32_t s_addr; /* address in network byte order */ char **h_aliases; /* alias list */ }; int h_addrtype; /* host address type */ int h_length; /* length of address */ char **h_addr_list; /* list of addresses */ network/host ordering: } #define h_addr h_addr_list[0] /* for backward compatibility */ - port and inetaddr are in network byte order (big endian) gethostbyname: - i386 platform is little endian name = hostname, or ascii "dotted quad" - convert short/long ints from host to network and back gethostbyaddr: - htonl(), htons(), ntohl(), ntohs() - should alway call for portability addr = ascii "dotted quad" (?) - if host order == network order, they are noops len = length of addr type = AF_INET (only supported) ------helper functions return values: ------success: hostent structure (you don’t need to free it) failure: NULL, examine h_errno for details #include
- convert "127.0.0.1" to network order in_addr error handling:
- "255.255.255.255" --> -1 !!! (also the error value) extern int h_errno; void herror(const char *s); - don’t use :(. const char * hstrerror(int err);
limitations: only speaks IPV4 int inet_aton(const char *cp, struct in_addr *inp) char *inet_ntoa(struct in_addr in) ------../rhd256.bowez_notes.2006-07-20.txt Thu Jul 20 15:31:29 2006 34 getipnodeby{name,addr}() - IPV4 and IPV6 int ai_flags; ------int ai_family; int ai_socktype; #include
struct hostent *getipnodebyaddr(const void *addr, size_t len, - combines getipnodeby{name,addr}, getservby{name,port} int af, int *error_num); - thread safe void freehostent(struct hostent *ip); - creates socket adress structure (to be passed to bind(), connect()) af = address family (AF_INET | AF_INET6)
flags: on entry: AI_V4MAPPED (w/ AF_INET6) map IPV4 address into IPV6 AI_ALL (w/ AI_V4MAPPED) include both IPV4 and IPV6 hints: AI_ADDRCONFIG only look up family if like interface is active supplies preferred socket type, or NULL for no preference on return: ai_family, ai_socktype, ai_protocol set to prefered type success: hostent structure (dynamically allocated) failure: NULL (examine error_num) to specify "any":
HOST_NOT_FOUND ai_family = PF_UNSPEC NO_ADDRESS (i.e., network IPV4, request IPV6) ai_socktype = 0 NO_RECOVERY dns returned permenant failure ai_protocol = 0 TRY_AGAIN dns returned temporary failure ai_flags: TBA ------services (/etc/services) all other fields: 0 or NULL, as appropriate ------node: #include
on exit: ------getaddrinfo returns dynamically allocated linked list of ------addrinfo structures (linked w/ ai_next field)
int getaddrinfo(const char *node, const char *service, why more than one? const struct addrinfo *hints, struct addrinfo **res); - mutlihomed hosts - service supported by two protocols (udp/tcp) void freeaddrinfo(struct addrinfo *res); const char *gai_strerror(int errcode); ai_family, ai_socktype, ai_protocol: akin to socket() system call
struct addrinfo { ai_canonname: set to canonical hostname, if AI_CANONNAME flag set ../rhd256.bowez_notes.2006-07-20.txt Thu Jul 20 15:31:29 2006 35 ------return value: socket configuration - the monster in the closet 0 success EAI_FAMILY family not supported ------EAI_SOCKTYPE socket type not supported EAI_BADFLAGS ai_flags invalid EAI_NONAME node or service not known Basic means of configuring sockets: EAI_SERVICE service not available w/ requested socket type EAI_ADDRFAMILY host has no addresses w/ requested family EAI_NODATA host exists, but no network addressed defined - system call arguments/flags send()/recv() EAI_MEMORY no memory - file descriptor options fcntl() EAI_FAIL permenant name server failure - i/o driver options ioctl() EAI_AGAIN temproary name server failure - socket options {g,s}etsockopt() EAI_SYSTEM other system error, check errno for details - sysctls /proc/sys/net/*, /etc/sysctl.conf - signals/errors ------getnameinfo ------system call arguments/flags int getnameinfo(const struct sockaddr *sa, socklen_t salen, ------char *host, size_t hostlen, char *serv, size_t servlen, int flags); send()/recv():
- combines gethostbyaddr() and getservbyport() - MSG_OOB, MSG_DONT{WAIT,ROUTE}, MSG_NO_SIGNAL, MSG_CONFIRM
- inverse of getaddrinfo() - man send(2)
- sockaddr is generic sockaddr_in or sockaddr_in6 ------fnctl ------on entry: sa: initialized sockaddr structure fcntl(): salen: sizeof(sa) host: host of size hostlen or NULL F_{G,S}ETOWN, F_{G,S}ETSIG servlen: buffer of size servlen or NULL flags: - man fcntl(2) NI_NOFQDN only hostname part of FQDN NI_NUMERICHOST dotted quad ------NI_NAMEREQD error if no hostname ioctl NI_NUMERICSERV ascii int service ------NI_DGRAM require UDP service ioctl():
on exit: socket layer (man socket(7)): one of or both host, and serv are filled out SIOCGSTAMP, SIOCSPGRP, FIOASYNC, SIOCGPGRP
- suggested sizes for supplied buffers (from getnameinfo(3):) ip layer:
# define NI_MAXHOST 1025 - firewalling (man ipfw(4)) # define NI_MAXSERV 32 - interface configuration (man netdevice(7))
- errors: tcp layer (man tcp(7)):
EAI_AGAIN temporary name resolution error FIONREAD or TIOCINQ, SIOCATMARK, TIOCOUTQ EAI_BADFLAGS malformed flags parameter EAI_FAIL permenant name resolution error ------EAI_FAMILY bad address family or length socket options EAI_MEMORY out of memory. ------EAI_NONAME name doesn’t resolve and NI_NAMEREQD set EAI_SYSTEM general system error occurred. examine errno. int getsockopt(int s, int level, int optname, void *opt-val, socklen_t *optlen); ../rhd256.bowez_notes.2006-07-20.txt Thu Jul 20 15:31:29 2006 36 # vi: ts=4 int setsockopt(int s, int level, int optname, const void *optval, socklen_t optlen); ==> ./misc/sysv/notes.txt <======
system V ipc - note that options can be set at different levels: ======SOL_SOCKET general socket options SOL_IP ip specific configs 1. introductory info SOL_TCP tcp specific configs 2. common constructs (PF_UNIX uses SOL_SOCKET) (udp protocol has no specific options) 3. specific constructs a. shared memeory b. message queues c. semaphores socket layer options (man socket(7)): 4. system administration SO_KEEPALIVE, SO_OOBINLINE, SO_{SND,RCV}LOWAT, a. ipcs SO_{SND,RCV}TIMEOUT, SO_BSDCOMPAT, SO_{PEER,PASS}CRED, b. /proc/sysvipc SO_BINDTODEVICE, SO_DEBUG, SO_REUSEADDR, SO_TYPE, c. /proc/sys/kernel/{msg*, sem, shm*} SO_DONTROUTE, SO_BROADCAST, SO_{SND,RCV}BUF, SO_LINGER, SO_PRIORITY, SO_ERROR ------
ip layer options (man ip(7)): introductory info
IP_OPTIONS, IP_PKTINFO, IP_RECV{TOS,TTL,OPTS}, IP_RETOPTS, ------IP_TOS, IP_TTL, IP_HDRINCL, IP_RECVERR, IP_PMTU_DISCOVER, IP_MTU, IP_ROUTER_ALERT, IP_MULTICAST_{TTL,LOOP}, System V IPC allows IPC through global system strucutures IP_{ADD,DROP}_MEMBERSHIP, IP_MULTICAST_IF - very "ununix like" tcp layer options (man tcp(7)): - does *not* rely on file descriptors TCP_NODELAY, TCP_MAXSEG, TCP_CORK - structures independent of processes and file system ------sysctls structures: ------shared memory constructs socket layer (man socket(7), /proc/sys/net/core/) message queues semaphores {r,w}mem_{default,max} send/recv buffer message_{burst,cost} limit warning ------netdev_max_backlog global input queue common concepts others: hot_list_length, {no,lo,mod}_cong, no_cong_thresh, optmem_max ------
ip layer (man ip(7), /proc/sys/net/ipv4) ------ip_forward enable routing creating/attaching to sysvipc objects ipfrag_{high,low}_thresh, ipfrag_time ip fragmentation ------ip_default_ttl ip_local_port_range int {shm,sem,msg}get(key_t key, .... , int flags)
others: returns object identifier, -1 on error ip_autoconfig, ip_conntrack_max, ip_dynaddr, ip_nonlocal_bind, ip_no_pmtu_disc key: - commonly known text string tcp layer (man tcp(7), /proc/sys/net/ipv4) - IPC_PRIVATE (generate unique key)
many..... common flags: ../rhd256.bowez_notes.2006-07-20.txt Thu Jul 20 15:31:29 2006 37 IPC_CREAT create new object if doesn’t exist struct shmid_ds { IPC_EXCL fail if the object does already exist struct ipc_perm shm_perm; /* operation perms */ IPC_NOWAIT don’t block int shm_segsz; /* size of segment (bytes) */ mode flags assign permissions to new object time_t shm_atime; /* last attach time */ time_t shm_dtime; /* last detach time */ (note sysvipc objects use same u/g/o rw permissions as files) time_t shm_ctime; /* last change time */ unsigned short shm_cpid; /* pid of creator */ unsigned short shm_lpid; /* pid of last operator */ two ways of handling key management: short shm_nattch; /* no. of current attaches */ }; 1) agree on common string, and deal w/ multiple instantiation
2) generate a private key, and use alternate IPC ------mechanism to deliver key to other processes getting the resource ------common permission data structure ------int shmget(key_t key, int size, int shmflg);
struct ipcperm { key_t key; /* ipc key */ - returns shmid or -1 uid_t uid; /* owner */ gid_t gid; - size is only relevent if creating new segment uid_t cuid; /* creator */ gid_t cgid; - key == IPC_PRIVATE mode_t mode; - key != IPC_PRIVATE, IPC_CREAT set, and key unkonwn to system unsigned short seq; /* ??? */ } - size is rounded up to next multple of PAGE_SIZE (usually 4k)
------modify sysvipc objects ------attaching/detaching to/from the resource ------int {shm,sem,msg}ctl(int id, int cmd, (ds) *buf, ....)
common commands: void *shmat(int shmid, const void *shmaddr, int shmflg ) int shmdt (const void *shmaddr) IPC_STAT fill in instance details IPC_SET set owner uid, gid, and mode IPC_RMID mark as destroyed (only if owner or creator) shmaddr: page aligned suggested address, or 0
------shmflg: SHM_RDONLY
shared memory segments returns attached address, -1 on error.
------attach preserved across fork() - detach across exec() and exit() (but *not* destroyed)
how to use shared memory segments: ------shmctl() 1) get (shmget()) the resource (possible allocating it) ------2) attach (shmat()) the resource into addresss space 3) use it... int shmctl(int shmid, int cmd, struct shmid_ds *buf); 4) detach (shdt()) the resource from address space
- standard IPC_STAT, IPC_SET - note that detaching does _not_ remove the resource - someone must take "cleanup" responsibility - IPC_RMID marks segment for removal
- will be destroyed after all detach ------representative data structure - only available to owner/creator ------root can "lock" segment into physical memory: SHM_{,UN}LOCK ../rhd256.bowez_notes.2006-07-20.txt Thu Jul 20 15:31:29 2006 38
msgflag: IPC_NOWAIT ------
message queues - multiplexed communication channel when receiving:
------msgtype:
0 receive first message (from any channel) >0 receive first message from channel msgtype ------<0 receive first message from channel |msgtype| or lower representative data structure ------msgsz = size allocated for receiving text
struct msqid_ds { msgflag: struct ipc_perm msg_perm; /* operation perms */ time_t msg_stime; /* last send time */ MSG_NOERROR truncate oversized msg (dflt: return -1/E2BIG) time_t msg_rtime; /* last recv time */ MSG_EXCEPT receive first message from channel !msgtype time_t msg_ctime; /* last change time */ msgnum_t msg_qnum; /* num mesgs in queue */ msglen_t msg_qbytes; /* max bytes allowd in queue */ ------/* defaults to MSGNMB */ msgctl() unsigned short msg_lspid; /* pid of last sender */ ------unsigned short msg_lrpid; /* pid of last receiver */ }; int msgctl(int msgid, int cmd, struct msgid_ds *buf);
------getting the resource - standard IPC_STAT, IPC_SET, and IPC_RMID ------
int msgget(key_t key, int msgflg);
------returns msgid or -1 semaphores - mutex and synchronization primitives - new resource created when ------key == IPC_PRIVATE - key != IPC_PRIVATE, IPC_CREAT set, and key unkonwn to system ------representative data structure ------sending/receiving messages struct msqid_ds { ------struct ipc_perm sem_perm; /* operation perms */ struct ipc_perm nsems; /* size of semaphore set */ int msgsnd(int msqid, struct msgbuf *msgp, size_t msgsz, int msgflg) time_t sem_otime; /* last semop time */ time_t sem_ctime; /* last semctl() time */ ssize_t msgrcv(int msqid, struct msgbuf *msgp, size_t msgsz, }; long msgtyp, int msgflg )
------struct msgbuf { getting the resource long mtype; /* message type, must be > 0 */ ------char mtext[1]; /* msg data (note array must be sized correctly) */ }; int semget(key_t key, int nsems, int semflg);
when sending: - upon creation, nsems = size of semaphore set
msgtyp defines "channel" for message - returns semid or -1
msgsz = size of mtext array (NOT struct msgbuf) ------../rhd256.bowez_notes.2006-07-20.txt Thu Jul 20 15:31:29 2006 39 initialization, querying, and ctl ------initialize to number of threads
union semun { each participant: int val; /* value for SETVAL */ - semop -1,0 (waits until semaphore == 0) struct semid_ds *buf; /* buffer for IPC_STAT, IPC_SET */ unsigned short int *array; /* array for GETALL, SETALL */ other protocols as well... struct seminfo *__buf; /* buffer for IPC_INFO */ };
int semctl(int semid, int semnum, int cmd, union semun arg)
commands:
standard IPC_STAT, IPC_SET, IPC_RMID (uses union buf)
{G,S}ETALL (uses union array)
{G,S}ETVAL (uses sennum and union val)
GET{N,Z}CNT
------performing semops ------
int semop(int semid, struct sembuf *sops, unsigned nsops)
struct sembuf { ... short sem_num; /* semaphore number: 0 = first */ short sem_op; /* semaphore operation */ short sem_flg; /* operation flags */ ... }
semantics: either all operations will be performed, or none.
sem_flg: SEM_UNDO undo (cleanup) operation when process exits
sem_op (on semaphore sem_num):
"signal (unlock)" >0 increment semaphore (by sem_op)
"post (lock)" <0 semaphore >= |sem_op|: decrement by |sem_op| semaphore < |sem_op|: block until semaphore >= |sem_op|
"barrier" 0 semaphore == 0 fall through semaphore > 0: block until semaphore == 0
mutex protocol:
- initialize semphore to 1 - lock: semop = -1 - unlock: semop = 1
barrier protocol: