Extending Fuser
Total Page：16
File Type：pdf, Size：1020Kb
Master Foo and the Ten Thousand Lines Master Foo once said to a visiting programmer: «There is more Unix-nature in one line of shell script than there is in ten thousand lines of C.» The programmer, who was very proud of his mastery of C, said: «How can this be? C is the language in which the very kernel of Unix is implemented!» Master Foo replied: «That is so. Nevertheless, there is more Unix-nature in one line of shell script than there is in ten thousand lines of C.» The programmer grew distressed. «But through the C language we experience the enlightenment of the Patri- arch Ritchie! We become as one with the operating system and the machine, reaping matchless performance!» Master Foo replied: «All that you say is true. But there is still more Unix-nature in one line of shell script than there is in ten thousand lines of C.» The programmer scoffed at Master Foo and rose to depart. But Master Foo nodded to his student Nubi, who wrote a line of shell script on a nearby whiteboard, and said: «Master programmer, consider this pipeline. Implemented in pure C, would it not span ten thousand lines?» The programmer muttered through his beard, contemplating what Nubi had written. Finally he agreed that it was so. «And how many hours would you require to implement and debug that C program?» asked Nubi. «Many,» admitted the visiting programmer. «But only a fool would spend the time to do that when so many more worthy tasks await him.» «And who better understands the Unix-nature?» Master Foo asked. «Is it he who writes the ten thousand lines, or he who, perceiving the emptiness of the task, gains merit by not coding?» Upon hearing this, the programmer was enlightened. 1 File System Users: Extending fuser Example 1: File System Users Beyond fuser Background: • you need to know who is doing what with some file system object (file, directory, whole file system. ) • the fuser command can tell the PIDs of involved process(es) • you want more: – process owner – complete command line Beyond fuser Interactive solution Who is using the /tmp file system? fuser -m /tmp /tmp: 531 3332 3809 11222 16745 32364 Then we need to know what each PID is: ps -p 531 PID TTY TIME CMD 531 ? 00:00:00 screen So it is a screen process, it has no attached tty, and has used no time. Beyond fuser Interactive solution /2 . or even better ps -p 531 -f UID PID PPID C STIME TTY TIME CMD john 531 1 0 Oct19 ? 00:00:00 SCREEN (complete commandline is available adding the -l switch, but output is too long to fit on a slide) We would then need to repeat for each of the PIDs listed in the output of fuser. Beyond fuser Step by Step Shell Function /2 1. first explode the output into tokens: fuser -m /tmp | tr -s ” ” ”nn” /tmp: 531 3332c ... tr simply converts each occurrence of a given character into a different one: here we convert spaces in newlines; the -s option collapses repeated characters. There can be a few letters attached to PIDs: here c means «current directory». Then you can have f open files e executable being run m mmaped file or shared library r root directory Beyond fuser Step by Step Shell Function /3 2. then take only PIDs out of the list: fuser -m /tmp | tr -s ” ” ”nn” |n egrep -o ’[0-9]+’ 531 3332 ... egrep selects only digits ([0-9]), optionally repeated (+), then outputs only the matching string (-o). Note that this implementation if far from perfection: if there is any digit in the file/directory name, it is treated as a PID.s Some versions of fuser are smarter than others, and output PID on STDOUT and any other additional information on STDERR, so filtering could be much easier (and effective). Beyond fuser Step by Step Shell Function /4 3. finally pass the resulting list to ps with any appropriate option: fuser -m /tmp | tr -s ” ” ”nn” |n egrep -o ’[0-9]+’ | xargs ps -f -p UID PID PPID C STIME TTY STAT TIME CMD john 531 1 0 Oct19 ? Ss 0:00 SCREEN eve 3332 1 0 Sep29 ? Ss 0:00 /usr/bin/lamd -H 127.0.0.1 -P 56154 -n 0 -o 0 xargs takes its STDIN and appends it to a given command line (here ps -f -p). Beyond fuser Step by Step Shell Function /5 As a last step we wrap everything in a shell function: function fuser2 () { fuser $* |\ tr -s " " "\n" |\ egrep -o ’[0-9]+’ |\ xargs ps -fl -p } where $* is used so that any command line parameter given to our little function is passed to the inner fuser command. 2 2 Find Users on a Node Example 2: Node Users Background: • HPC cluster with the Maui job scheduler • we need to know the user list for a given node – nodes are not job-exclusive – there can be jobs from different users sharing a node • available commands: – checknode reports information on a given node, including the list of job assigned to the node – checkjob reports information on a given job, including the list of assigned nodes and the job owner Node Users Interactive Solution Step by Step /1 [root@cerbero ] checknode a206 checking node a206 State: Running (general information on node status omitted) Reservations: Job ’115713’(x1) -1:18:00:36 -> 2:05:58:24 (3:23:59:00) Job ’115848’(x1) -22:07:38 -> 3:01:51:22 (3:23:59:00) JobList: 115713,115848 Jobs 115713 and 115848 are running on node a206. Node Users Interactive Solution Step by Step /2 [root@cerbero ] checkjob 115713 checking job 115713 State: Running Creds: user:joe group:phys class:smp2 qos:slow WallTime: 1:18:39:39 of 4:00:00:00 SubmitTime: Tue Oct 21 15:43:36 (more information omitted) Job 115713 owner is user joe; job has run for 1 day, 18 hours and 39 minutes out of 4 days requested. Now we would need to repeat for job 115848. If we wanted to know current users of a dozen more nodes, the whole process is not exactly exciting. Node Users Step by Step Shell Function /1 We are going to define a shell function node_user that is going to accept a single argument (a node name) and output the list of users on a node, along with their remaining time. 1. Let’s start by extracting the list of running jobs on the node: checknode a206 | grep JobList JobList: 115713,115848 Node Users Step by Step Shell Function /2 2. then explode the list to single components: checknode a206 | grep JobList | egrep -o ’[0-9]+’ 115971 115713 The regular expression ’[0-9]+’ means «any digit, repeated one or more times»; the -o option tells egrep to output only the matching strings (not whole lines). 3 Node Users Step by Step Shell Function /3 3. then pass each of the list elements to the checkjob command: checknode a206 | grep JobList | egrep -o ’[0-9]+’ | xargs -i checkjob {} (very long output, not shown) • xargs takes its standard input and uses it to build a new command line • the -i option tells xargs to iterate over input elements and build a command line for each one • {} is a placeholder in the command line where xargs will put each list element Node Users Step by Step Shell Function /4 4. finally filter the insanely long output to get only user and WallTime lines: checknode a206 | grep JobList | egrep -o ’[0-9]+’ | xargs -i checkjob {} | egrep -o ’user:[ˆ ]+|WallTime: .+’ Another regular expression where • | is logical «OR» • [ˆ ] means «everything that is not a space» Node Users Step by Step Shell Function /5 The final magic: function node_user () { [ $# -ne 1 ] && { echo "Usage: $FUNCNAME nodename" 1>&2; return 1 }; checknode $1 | grep JobList |\ egrep -o ’[0-9]+’ | xargs -i checkjob {} |\ egrep -o ’user:[^ ]+|WallTime: .+’ } 3 Check for Dead Services and Restart When Needed Example 3: Restart Dead Services Background: • you have some slightly unstable service that sometimes crashes and needs to be restarted; or maybe it is sometimes killed due to resource exaustion (fairly typical on computing nodes if jobs are allowed to use all available memory) • you don’t want to deploy a full-featured monitoring system and/or need a quick-and-dirty solution • you want a basic level of configurability, e.g. you want to be able to shut down a service and not have it restarted • monitored services are important enough that you want them restarted without human intervention, but not so critical you need them to be restarted «now» 4 Restart Dead Services Interactive Solution Step by Step At some point node m001 starts looking funny. 1. go to node m001: ssh m001 2. check for service availability: service ntpd status ntpd is stopped 3. you known for sure that ntpd should be there, so you restart it: service ntpd restart ntpd: Synchronizing with time server: [OK] Starting ntpd: [OK] Restart Dead Services Step by Step Shell Script /1 Version 0 script could be #!/bin/bash service ntpd status || \ service ntpd restart The first part of the pipeline reports the service status both on STDOUT and in the exit code – we are only interested in the exit code actually. Restart Dead Services Step by Step Shell Script /2 A more advanced version of our script would check if ntpd is supposed to be there before deciding it is thime to (re)start it. This can be done with chkconfig: chkconfig --list ntpd ntpd 0:off 1:off 2:off 3:on 4:on 5:on 6:off But what runlevel are we currently in? runlevel N 3 So this will tell us what we really need to know: RL=$( runlevel | cut -d’ ’ -f2 ) chkconfig --list ntpd | grep -q ${RL}:on echo $? 0 Restart Dead Services Step by Step Shell Script /3 We can make a shell function with our little «check if service should be there» piece of script: function is_active() { [ "x$1" = "x" ] && return 1 local SERVICE=$1 local RL=$( runlevel | cut -d’ ’ -f2 ) chkconfig --list $SERVICE | \ grep -q "${RL}:on" return $? } 5 Restart Dead Services Step by Step Shell Script /4 The new script will then be: #!/bin/bash (here goes the definition of is_active) is_active ntpd && \ { service ntpd status >/dev/null ||\ service ntpd restart ; } Restart Dead Services Step by Step Shell Script /5 As a final step the script needs to be executed periodically.