Discovering the magic of the command line in Prague
By Mark Krenz of @climagic What is climagic?
● Twitter account I started in 2009 ● Over 10,000 posts (~50% are unique commands) ● Over 138,000 followers and counting ● Meant to inspire beginners and advanced users ● Based on bash running on Linux ● Tweets often driven by my own jobs, needs and ideas. ● Typical format of a tweet:
Quite simply because it’s not “magic” at all.
"Any sufficiently advanced technology is indistinguishable from magic.” -- Arthur C. Clarke’s 3rd law
"Witchcraft to the ignorant, … simple science to the learned" -- Leigh Brackett, 1942 Who am I?
Because I guess I’ve never really mentioned it.
● I’m Mark Krenz, I created and run climagic ● Live in the great town of Bloomington, Indiana, USA ● Security Analyst for Indiana University’s Center for Applied Cybersecurity Research ( https://cacr.iu.edu/ ) ● Have been using Linux since 1997 ● Wrote num-utils programs for CLI numeric operations ● Wrote the top rated SSH tutorial (needs updating) Who are you? According to the 2012 climagic survey with over 1000 responses and Twitter Analytics
● Beginners to advanced users and even non-believers ● People from 63+ different countries ● Various Unicies, Mac OS X plus over 26 distros of Linux ● Users of bash, zsh, tcsh, ksh, sh, dash, csh, ash, fish, pdksh and yes even…… powershell. ● People from a variety of different professions with different needs. (IT folk, Scientists, Engineers, Broadcasting, Law, Medicine, Writer, Train technician and even a Pastor) ● People who have a high tolerance for corny geek jokes. The command line? You mean that black window with text inside? No graphics? n the beginning… ( approx. 1.5 billion seconds ago ) there was a file.
And Kenneth and Dennis said it is good and easy to work with, so they made everything* act like a file and called it TM Unix . And the system started to grow. Make each program do one thing well.
cd - Change your directory ls - What files are here cat - Show the contents of a file ed - Edit a file cp - Copy a file rm - remove a file mv - move a file mkdir - Create a directory wc - Count lines, words, characters su - Become the superuser or another user man - Learn about the system and commands However, the real power was in the plumbing. Make each program act as a filter. (stdin/stdout)
thers came and basked in Unix’s glory and exclaimed: “Let us make many programs for all to use with this system.. The command is the word. RTFM.”
The terminal window is a portal into the heart of your computer. You are the interface and it can be as good as you can be. How to get help? (Because you’ll probably need it)
● Run ‘man ls’ ● Run ‘man -k foobar’ (must run makewhatis first) ● Run ‘help cd’ ● Run ‘tmux -h’ ● http://mywiki.wooledge.org/BashGuide ● Google it ● Ask online ● Pray!
Lonely is the user who searches the Internet for an error message and only finds the source code that generates it. Moving around ● cd Pictures (Now you are in subdirectory Pictures) ● cd /home (Now you are in /home) ● cd (Now you are in your home directory ~/ ) ● cd - (Now you are in /home) ● cd /media/user/Flashcard ● cd . (cd into the same symbolic path. Useful if remounted/recreated)
Chuck Norris once ran cd .. in / and it worked. Useful Keystrokes (mostly readline/Emacs) ● [Ctrl]-r (then type what to search for), press repeatedly,
"Computers in the future may weigh no more than 1.5 tons." -- Popular Mechanics article from 1949 Variables ● MYWORD="aimlessness" ● export MYWORD (export means make the variable available to subshells, programs, etc) ● TZ="Europe/Prague" date (one time export to command) ● LANG=C sort ● env & set
"f u cn rd ths, u cn gt a gd jb n cmptr prgrmmng." -fortune Variables ● ALPHABET=$(echo {a..z} | tr -d ' ') ● echo ${ALPHABET:0:3} ● echo ${ALPHABET^^} ● echo ${ALPHABET/abc/Easy as abc 123 }
Our parents made machines that took people to the moon. We created programs that make it look like we are vomiting rainbows. Special Variables ● echo $(($RANDOM % 100)) ● echo $SHLVL ● echo $COLUMNS $LINES ● ps auxww | grep " $$ " ● pgrep -aP $PPID ● mv photo.jpg $OLDPWD/
What do keyboard users and reclusive astronauts have in common? They both think that space bars are too noisy. Functions ● uploadphotos(){ scp -rp "$@" you@webserver:www/photos/; } Usage: uploadphotos IMG_{3201..3250}.JPG
● function todo(){ ${EDITOR:-/usr/bin/vim} ~/$(date +todolist-%Y%m%d); }
● function cd(){ [[ "$1" == "..." ]] && builtin cd ../.. || builtin cd $@ ; }
fliptable(){ echo "（╯°□°）╯ ┻━┻"; }
Usage: fsck -y /dev/sda || fliptable Functions ● sus(){ sort | uniq -c | sort $@; }
● function acronym(){ elinks -no-numbering -dump \ http://acronymfinder.com/$1.html | sed '/*/!d';}
● function box() { t="$1xxxx";c=${2:-=}; echo ${t//?/$c}; \ echo "$c $1 $c"; echo ${t//?/$c}; }
37, the age at which Grace Hopper joined the Navy, learned computer logic and later invented the compiler and COBOL. It's never too late to get started. Functions (pranks) ● function ls() { /bin/ls $@ | /usr/bin/tac ; }
":-P" -- Albert Einstein Misc tricks ● du -sh */ ● tail -n+2 data.csv ● join -o 1.1,2.2 -1 1 -2 1 -t: passwd shadow ● lsof +D /tmp ● lsof -i TCP:80 ● renice -p 2235
When confronted with new things, I first scoff at it. Then I scoff again. Then I try it. Then I use it. Then I defend it as a way of life. Brace Expansion ● convert photo.{jpg,png} -> convert photo.jpg photo.png ● tar zcvf sourcecode{.tar.gz,} -> tar zcvf sourcecode.tar.gz sourcecode ● echo {0..10} -> 0 1 2 3 4 5 6 7 8 9 10 ● echo a{0..5} -> a0 a1 a2 a3 a4 a5 ● echo {a..z} -> a b c d e f g h i j k l m n o p q r s t u v w x y z # Bash only
Did you know? Brian J. Fox's (creator of bash) grandfather Dan Fox, created the Monopoly® Man character. Now the $ prompt makes more sense. Output redirection ● ls -l > lsoutput.txt ● ls -l filethatdoesntexit 2> out.err ● t stream search trump > /dev/sda
STDERR is like the fire escape for data. Process Substitution ● comm -12 <(ls -1 dir1 | sort) <(ls -1 dir2 | sort)
Unix was invented the same month and year that ZZ Top was formed (June 1969). Coincidence? I think not. awk & sed ● awk '{print $1}' access_log ● awk -F, '{print $2 " " $3}' data.csv ● awk '$7~/\.png/ {array[$7]+=$10} END {for (i in array) \ { print array[i] " " i; } }' access_log | sort -nr
Computers are supposed to do the work for us. If you're doing most of If you're doing most of the work for the computer, then you've lost your way. sed ● sed -n '29p' mycode.pl ● sed -s -i.bak 's/Copyright 2016/Copyright 2017/' *.html ● sed -n '12 {p; Q}' file.txt
Sometimes regular expressions can get messy. That's what she sed. More awk & sed ● ps aux | awk '$8=="Z"' | wc -l ● elinks -dump http://bit.ly/qutWZK|sed -n -e 's/,//g' \ -e '/^1960/,/^2010/p' | awk '{p=$6/$2*100; print $1 " " p}'
How many unix users does it take to change a lightbulb? Well that depends, is it a System V, BSD, GNU or Mac lightbulb? Pipelines ● ls -l | less ● mount | column -t ● du -sh */ | sort -rh
ZZ Top: A band formed in June 1969 that is known for its beards. Unix: same commands, same dates, same reputation. Pipelines ● awk {'print $1'} access_log | sort | uniq -c | sort -nr
To anyone who ever asks for an example of a program that is the opposite of "do one thing well". Try a word processor. Pipelines ● elinks http://www.$( look . | egrep “^[a-z]{2,6}$” | shuf | head -n 1 ).com
It took the power of 3 Commodore 64s to go to the moon, but you need something 100,000 times more powerful to run Firefox. Pipelines ● elinks enwp.org/$(date +%B_%d) | sed -n -e 's/\[[0-9]\+\]//g' \ -e '/^\[.* Ev/,/^\[/p' | tr -d '\n' | tr '*' '\n' | \ grep -v '\[ed' | sort -R | head -1
Try running 'cal 9 1752' Loops ● for i in 1 2 3 ; do echo $i ; done ● for i in *.jpg ; do convert $i ${i%jpg}.png ; done
Me: Ok Siri, copy the pictures from number 187 through 250 to the server. Siri: I'm sorry but I can't do that. Loops ● while true ; do uptime ; sleep 5 ; done ● while :;do iwlist wlan0 scan |awk -F\" '/ESSID/{print $2}' |espeak;done
Windows = Fast Food Mac = Expensive restaurant dinner iOS/Android = Vending machine snacks Linux = Home cooked meal Loops Figure out which years Nov 21st was on a Friday.
for y in {1970..2017} ; do [[ $( date -d $y-11-21 +%u ) -eq 5 ]] && echo $y ; done
Bad winters bring good software, and more programmers. Loops Send output of one loop down the pipeline into another loop.
for grass in field* ; do mow $grass; done | grep -v weeds \ | while read -r hay; do bale $hay; done
Distrust all claims for "one true way". -- "The Art of Unix Programming" by Eric S. Raymond xargs ● find . -name .htaccess | xargs egrep -l "(deny|allow)" ● look . | egrep "^[a-z]{1,4}$" | xargs -P 40 -I{} \ curl -L -sI -o www.{}.com http://www.{}.com ● xargs < file.txt
Big commands only looks complex when you look at the whole command at once; Break it down; Its not a picture. Putting it all together ● echo 192.168.0.{1..254}| xargs -n1 -d' ' -P254 -I{} \ ping -w1 -c1 {} 2>/dev/null| awk '/ttl/{print $4}'
wait %1 ; cd /pub && more beer Other useful commands (incomplete) ● tmux or screen ● script -t ~/myshell.script 2>~/myshell.time ● netcat and socat ● zcat, zless, zgrep and their cousins bzless, xzless, etc. ● sponge ● pee ● grep 67.205.74.88 messages |sponge |pee "head -1" "tail -1" ● ngrep -d eth0 -i 'select' port 3306 ● parcellite (clipboard/selection sync)
"I couldn't figure out how to quit vi so I rebooted the system." --[name censored to protect the innocent] Thank you for coming
● First of all special thanks to my family for the support. ● Petr Hodač and others from FIT for helping to host and organize this event. ● Czech Technical University for hosting this event. ● The many dedicated people who make open source great.
And thanks for reading this section too.