Discovering the magic of the command line in Prague By Mark Krenz of @climagic What is climagic? ● Twitter account I started in 2009 ● Over 10,000 posts (~50% are unique commands) ● Over 138,000 followers and counting ● Meant to inspire beginners and advanced users ● Based on bash running on Linux ● Tweets often driven by my own jobs, needs and ideas. ● Typical format of a tweet: <command> # <description> ● Youtube channel: http://www.youtube.com/user/climagic ● Also check out the IRC channel #climagic on irc.freenode.org Why “magic”? Quite simply because it’s not “magic” at all. "Any sufficiently advanced technology is indistinguishable from magic.” -- Arthur C. Clarke’s 3rd law "Witchcraft to the ignorant, … simple science to the learned" -- Leigh Brackett, 1942 Who am I? Because I guess I’ve never really mentioned it. ● I’m Mark Krenz, I created and run climagic ● Live in the great town of Bloomington, Indiana, USA ● Security Analyst for Indiana University’s Center for Applied Cybersecurity Research ( https://cacr.iu.edu/ ) ● Have been using Linux since 1997 ● Wrote num-utils programs for CLI numeric operations ● Wrote the top rated SSH tutorial (needs updating) Who are you? According to the 2012 climagic survey with over 1000 responses and Twitter Analytics ● Beginners to advanced users and even non-believers ● People from 63+ different countries ● Various Unicies, Mac OS X plus over 26 distros of Linux ● Users of bash, zsh, tcsh, ksh, sh, dash, csh, ash, fish, pdksh and yes even…… powershell. ● People from a variety of different professions with different needs. (IT folk, Scientists, Engineers, Broadcasting, Law, Medicine, Writer, Train technician and even a Pastor) ● People who have a high tolerance for corny geek jokes. The command line? You mean that black window with text inside? No graphics? n the beginning… ( approx. 1.5 billion seconds ago ) there was a file. And Kenneth and Dennis said it is good and easy to work with, so they made everything* act like a file and called it TM Unix . And the system started to grow. Make each program do one thing well. cd - Change your directory ls - What files are here cat - Show the contents of a file ed - Edit a file cp - Copy a file rm - remove a file mv - move a file mkdir - Create a directory wc - Count lines, words, characters su - Become the superuser or another user man - Learn about the system and commands However, the real power was in the plumbing. Make each program act as a filter. (stdin/stdout) thers came and basked in Unix’s glory and exclaimed: “Let us make many programs for all to use with this system.. The command is the word. RTFM.” The terminal window is a portal into the heart of your computer. You are the interface and it can be as good as you can be. How to get help? (Because you’ll probably need it) ● Run ‘man ls’ ● Run ‘man -k foobar’ (must run makewhatis first) ● Run ‘help cd’ ● Run ‘tmux -h’ ● http://mywiki.wooledge.org/BashGuide ● Google it ● Ask online ● Pray! Lonely is the user who searches the Internet for an error message and only finds the source code that generates it. Moving around ● cd Pictures (Now you are in subdirectory Pictures) ● cd /home (Now you are in /home) ● cd (Now you are in your home directory ~/ ) ● cd - (Now you are in /home) ● cd /media/user/Flashcard ● cd . (cd into the same symbolic path. Useful if remounted/recreated) Chuck Norris once ran cd .. in / and it worked. Useful Keystrokes (mostly readline/Emacs) ● [Ctrl]-r (then type what to search for), press repeatedly, <Esc> when done ● [Ctrl]-a (move to beginning of line) ● [Ctrl]-e (move to the end of the line) ● [Ctrl]-w (Yank (like delete) the last word) ● [Meta]-. (Insert the last argument from the last line) ● SSH keystrokes: ○ [RET]~C ○ [RET]~. Job Control ● ./render ● [Ctrl]-z ● jobs ● fg ● bg ● kill %1 "Computers in the future may weigh no more than 1.5 tons." -- Popular Mechanics article from 1949 Variables ● MYWORD="aimlessness" ● export MYWORD (export means make the variable available to subshells, programs, etc) ● TZ="Europe/Prague" date (one time export to command) ● LANG=C sort ● env & set "f u cn rd ths, u cn gt a gd jb n cmptr prgrmmng." -fortune Variables ● ALPHABET=$(echo {a..z} | tr -d ' ') ● echo ${ALPHABET:0:3} ● echo ${ALPHABET^^} ● echo ${ALPHABET/abc/Easy as abc 123 } Our parents made machines that took people to the moon. We created programs that make it look like we are vomiting rainbows. Special Variables ● echo $(($RANDOM % 100)) ● echo $SHLVL ● echo $COLUMNS $LINES ● ps auxww | grep " $$ " ● pgrep -aP $PPID ● mv photo.jpg $OLDPWD/ What do keyboard users and reclusive astronauts have in common? They both think that space bars are too noisy. Functions ● uploadphotos(){ scp -rp "$@" you@webserver:www/photos/; } Usage: uploadphotos IMG_{3201..3250}.JPG ● function todo(){ ${EDITOR:-/usr/bin/vim} ~/$(date +todolist-%Y%m%d); } ● function cd(){ [[ "$1" == "..." ]] && builtin cd ../.. || builtin cd $@ ; } fliptable(){ echo "（╯°□°）╯ ┻━┻"; } Usage: fsck -y /dev/sda || fliptable Functions ● sus(){ sort | uniq -c | sort $@; } ● function acronym(){ elinks -no-numbering -dump \ http://acronymfinder.com/$1.html | sed '/*/!d';} ● function box() { t="$1xxxx";c=${2:-=}; echo ${t//?/$c}; \ echo "$c $1 $c"; echo ${t//?/$c}; } 37, the age at which Grace Hopper joined the Navy, learned computer logic and later invented the compiler and COBOL. It's never too late to get started. Functions (pranks) ● function ls() { /bin/ls $@ | /usr/bin/tac ; } ":-P" -- Albert Einstein Misc tricks ● du -sh */ ● tail -n+2 data.csv ● join -o 1.1,2.2 -1 1 -2 1 -t: passwd shadow ● lsof +D /tmp ● lsof -i TCP:80 ● renice -p 2235 When confronted with new things, I first scoff at it. Then I scoff again. Then I try it. Then I use it. Then I defend it as a way of life. Brace Expansion ● convert photo.{jpg,png} -> convert photo.jpg photo.png ● tar zcvf sourcecode{.tar.gz,} -> tar zcvf sourcecode.tar.gz sourcecode ● echo {0..10} -> 0 1 2 3 4 5 6 7 8 9 10 ● echo a{0..5} -> a0 a1 a2 a3 a4 a5 ● echo {a..z} -> a b c d e f g h i j k l m n o p q r s t u v w x y z # Bash only Did you know? Brian J. Fox's (creator of bash) grandfather Dan Fox, created the Monopoly® Man character. Now the $ prompt makes more sense. Output redirection ● ls -l > lsoutput.txt ● ls -l filethatdoesntexit 2> out.err ● t stream search trump > /dev/sda STDERR is like the fire escape for data. Process Substitution ● comm -12 <(ls -1 dir1 | sort) <(ls -1 dir2 | sort) Unix was invented the same month and year that ZZ Top was formed (June 1969). Coincidence? I think not. awk & sed ● awk '{print $1}' access_log ● awk -F, '{print $2 " " $3}' data.csv ● awk '$7~/\.png/ {array[$7]+=$10} END {for (i in array) \ { print array[i] " " i; } }' access_log | sort -nr Computers are supposed to do the work for us. If you're doing most of If you're doing most of the work for the computer, then you've lost your way. sed ● sed -n '29p' mycode.pl ● sed -s -i.bak 's/Copyright 2016/Copyright 2017/' *.html ● sed -n '12 {p; Q}' file.txt Sometimes regular expressions can get messy. That's what she sed. More awk & sed ● ps aux | awk '$8=="Z"' | wc -l ● elinks -dump http://bit.ly/qutWZK|sed -n -e 's/,//g' \ -e '/^1960/,/^2010/p' | awk '{p=$6/$2*100; print $1 " " p}' How many unix users does it take to change a lightbulb? Well that depends, is it a System V, BSD, GNU or Mac lightbulb? Pipelines ● ls -l | less ● mount | column -t ● du -sh */ | sort -rh ZZ Top: A band formed in June 1969 that is known for its beards. Unix: same commands, same dates, same reputation. Pipelines ● awk {'print $1'} access_log | sort | uniq -c | sort -nr To anyone who ever asks for an example of a program that is the opposite of "do one thing well". Try a word processor. Pipelines ● elinks http://www.$( look . | egrep “^[a-z]{2,6}$” | shuf | head -n 1 ).com It took the power of 3 Commodore 64s to go to the moon, but you need something 100,000 times more powerful to run Firefox. Pipelines ● elinks enwp.org/$(date +%B_%d) | sed -n -e 's/\[[0-9]\+\]//g' \ -e '/^\[.* Ev/,/^\[/p' | tr -d '\n' | tr '*' '\n' | \ grep -v '\[ed' | sort -R | head -1 Try running 'cal 9 1752' Loops ● for i in 1 2 3 ; do echo $i ; done ● for i in *.jpg ; do convert $i ${i%jpg}.png ; done Me: Ok Siri, copy the pictures from number 187 through 250 to the server. Siri: I'm sorry but I can't do that. Loops ● while true ; do uptime ; sleep 5 ; done ● while :;do iwlist wlan0 scan |awk -F\" '/ESSID/{print $2}' |espeak;done Windows = Fast Food Mac = Expensive restaurant dinner iOS/Android = Vending machine snacks Linux = Home cooked meal Loops Figure out which years Nov 21st was on a Friday. for y in {1970..2017} ; do [[ $( date -d $y-11-21 +%u ) -eq 5 ]] && echo $y ; done Bad winters bring good software, and more programmers. Loops Send output of one loop down the pipeline into another loop. for grass in field* ; do mow $grass; done | grep -v weeds \ | while read -r hay; do bale $hay; done Distrust all claims for "one true way". -- "The Art of Unix Programming" by Eric S. Raymond xargs ● find . -name .htaccess | xargs egrep -l "(deny|allow)" ● look . | egrep "^[a-z]{1,4}$" | xargs -P 40 -I{} \ curl -L -sI -o www.{}.com http://www.{}.com ● xargs < file.txt Big commands only looks complex when you look at the whole command at once; Break it down; Its not a picture.