CPSC 424/624 Quiz #4 Solutions Closed book/computers Spring 2017 Last update: 3/3/2017

Name:

1 Create a simple Bash line involving the utility to assemble a set of files and then have this result piped to to the number of lines . You should use xargs in your pipeline. Explain what xargs does, using your example to illustrate. Include in your summary what would happen if your pipeline did not use xargs. For example, what is the problem if you were to run your pipeline but without using xargs. Based on this exercise (along with any additional reading), please explain the benefits of xargs.

Solution: xargs [option ...] [command [initial-arguments]] xargs is another way to call commands or programs. The term xargs is synonymous with 'combine arguments'. It builds and executes command lines by gathering together arguments from standard in ( as we will see are many times supplied by find)

A simple example….in a directory, issue the following: #; ls | wc; ls | xargs; ls | xargs | wc;

From the bashScripts/monitors directory, the command line result is:

1 ls; 2 monitorScriptRunzhen.py monitorScriptRunzhen.sh mySysMon.sh paul.py paul.sh tecmint_monitor.sh xin.py xin.sh 3 ls | wc; 4 8 8 109 5 ls | xargs; 6 monitorScriptRunzhen.py monitorScriptRunzhen.sh mySysMon.sh paul.py paul.sh tecmint_monitor.sh xin.py xin.sh 7 ls | xargs | wc 8 1 8 109 9 ls | xargs wc 10 72 208 1885 monitorScriptRunzhen.py 11 57 184 1787 monitorScriptRunzhen.sh 12 63 206 1456 mySysMon.sh 13 82 219 1866 paul.py 14 64 201 1444 paul.sh 15 117 491 4015 tecmint_monitor.sh 16 75 252 2329 xin.py 17 59 124 1167 xin.sh 18 589 1885 15949 total

Lines 1 and 2 display the output of ‘ls’- a list of filenames with each separated by a ‘\n’. Lines 3-4 counts the number of results. Line 5-6 shows that xarg’s will gather the results from the previous command and places them in a single line.

Note that is useful….. try ls | od –c and then ls | xargs –c

If a program is specified to xargs (wc in our example), xargs calls the program passing the data from the pipe as parameters. If xargs is used without a p, total of program, it defaults to issuing an of the parameters.

Lines 7 and 8 (ls | xargs | wc) pipes to wc effectively a string (1 line, 8 words…) Line 9- 18 - (ls | xargs wc ) causes wc to be invoked with 8 parameters….wc interprets each as a name, and provides the wc output for each file on a separate line.

Xargs adds flexibility to command lines – it allows pipelines that would require a complicated script to achieve without xargs.

GNU’s parallels can be viewed as a replacement of xargs, adding support for concurrency.

2 On your VM, issue the following commands: 5; alert;

Poke around on your VM to learn about ‘alert’. What is the effective method for learning about something you see used in a command line? Once you have found alert, explain how it works. So…if it is a program, deduce how it works. If it is a script, decompose it to learn and then explain how it works.

Solution: The command is usually the best way as it can give you the information needed to know roughly what to look for. Using type, we find that ‘alert’ is an . With a bit of digging you would find the alias in ./bashrc. type alert alert is aliased to `notify-send --urgency=low -i "$([ $? = 0 ] && echo terminal || echo error)" "$(| -n1| -e 's/^\s*[0-9]\+\s*//;s/[;&|]\s*alert$//')"'

`notify-send --urgency=low -i "$([ $? = 0 ] && echo terminal || echo error)" "$(history|tail -n1|sed -e 's/^\s*[0-9]\+\s*//;s/[;&|]\s*alert$//')"'

Decomposing the alias: Notify-send is a program allowing a script to send desktop notifications. The –i allows the caller to specify a graphics icon that will be displayed with the message.

Executing the following: `notify-send --urgency=low -i "$([ $? = 0 ] && echo terminal || echo error)" "$(history|tail -n1|sed -e 's/^\s*[0-9]\+\s*//;s/[;&|]\s*alert$//')"`

Does display a msg (upper left of desktop) but the txt message is the actual shell code between the backquotes. Why?

It is doing the right thing….it is displaying the last command….in this case the notify- send. When called by the alias, the last command is delay 5

3 The following script (paul.sh) is available https://people.cs.clemson.edu/~jmarty/courses/Spring 2017/CPSC424/code/bashScripts/monitors/.

This following is an excerpt from the monitor script. If invoked as ‘./paul.sh 5 enp0s17’ it will send the following information to standard out every 5 seconds. The second param is the network interface name (use to get this) we want to monitor. Please answer the following questions:  On lines 24-25, explain exactly what a ‘5 minute load CPU utilization ‘ is and second, exactly how this code obtains that information  On lines 27-31, explain what exactly the code produces and explain exactly how the awk code actually works.

Interval (sec): 5 Inteface: enp0s17 Number of Cores: 1 5 Minute Avg CPU Utilization (%): 8.0 Outbound BW (bits): 0 Inbound BW (bits): 0

1 #!/bin/bash 2 # System Health Monitor 3 # monitorScript.sh 4 interval=$1 5 interface=$2 6 if [ "$#" -ne 2 ] 7 then 8 echo 9 echo "Usage: $0 " 10 echo 11 1 12 fi 13 declare -i tempOut=0 14 declare -i tempIn=0 15 while true; do 16 17 echo "Interval (sec): "$'\t'"$interval" 18 echo "Inteface: "$'\t'"$interface" 19 echo -n "Number of Cores: "$'\t' 20 cores=$( pro /proc/cpuinfo -c) 21 echo $cores

22 # - CPU Utilization 23 echo -n "5 Minute Avg CPU Utilization (%): "$'\t' 24 util=$( /proc/loadavg | awk '{print $1}') 25 echo "scale=1; (($util*100)/$cores)/1" | bc # >> cpuUtilBash.txt 26 # - Bandwidth Consumption 27 echo -n "Outbound BW (bits): "$'\t' 28 outbound1=$(awk -v interface="$interface" -F'[: \t]+' \ 29 '{ sub(/^ */,""); 30 if ($1 == interface) print $10; 31 }' /proc/net/dev)

Solution:

Note that this implementation has problems. First, the original script used the 1 minute avg field in /proc/loadavg … it should use the second field (5 minute avg). Second, we need to be precise in describing what we actually want to monitor. The loadavg assesses the demand for not only the CPU but also for I/O by the count of both types of processes waiting on system resources. In general, this should be divided by the number of total CPU cores to get an average across all CPUs.

This is different from a CPU utilization which counts the relative amount of the CPU is busy (versus idle). This measure is more of a result of the OS scheduling – assessing the % of time the CPUs are busy. The measure is the ratio of the number of scheduling intervals that were allocated to a process to the total number of intervals in the measurement period. If a CPU in an interval is not allocated, it is ‘idle’. The measure can be per CPU/Core or it can be the avg for all cores.

The ‘5 minute load CPU utilization ‘ is based a piece of information available in the /proc/load. A ‘cat /proc/load /proc/stat’ on my VM shows: /proc/load 0.08 0.04 0.00 1/413 4071 /proc/stat: cpu 14468 150 17920 5610958 594 0 268 0 0 0 cpu0 5913 135 7698 2806934 183 0 121 0 0 0 cpu1 8555 15 10222 2804024 410 0 146 0 0 0 intr 4782837 36 8466 0 0 0 0 0 0 0 0 0 0 1004 0 0 28013 14238 19425 2 0 46538 25509 34 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ctxt 14984512 btime 1488540624 processes 4070 procs_running 2 procs_blocked 0 softirq 1113111 3 419335 7264 34027 37660 0 8486 419336 0 187000

A ‘man proc’ explains /proc/loadavg as follows:

The first three fields in this file are load average figures giving the number of jobs in the run queue (state R) or waiting for disk I/O (state D) averaged over 1, 5, and 15 minutes. They are the same as the load average numbers given by (1) and other programs. The fourth field consists of two numbers separated by a slash (/). The first of these is the number of currently runnable kernel scheduling entities (processes, threads). The value after the slash is the number of kernel scheduling entities that currently exist on the system. The fifth field is the PID of the process that was most recently created on the system.

A very good summary of loadavg and a time-based measure of the percentage of CPU being consumed is given in the LinuxJournal Article : http://www.linuxjournal.com/article/9001

See the Bash script cpumon.sh as an example of how to monitor the CPU utilization…. https://people.cs.clemson.edu/~jmarty/courses/Spring- 2017/CPSC424/code/bashScripts/monitors/cpumon.sh

On lines 27-31, explain what exactly the awk code produces and explain exactly how the awk code actually works.

1 echo -n "Outbound BW (bits): "$'\t' 2 outbound1=$(awk -v interface="$interface" -F'[: \t]+' \ 3 '{ sub(/^ */,""); 4 if ($1 == interface) print $10; 5 }' /proc/net/dev)

Break it down from the …. Echo –n “ …. “ prints whatever is generated in the quotes.

Line 27 starts the echo “ “ : echo -n "Outbound BW (bits): "$'\t'

Sets the variable outbound1 to the result of the $()

The expre is awk that processes /proc/net/dev as input. It sets interface as a variable that will be used to find the right line in /proc/dev/stat. The –F sets the FS to one or more : and tab characters. For each line in the data file, all characters are deleted with the sub string …. By matching all with /^*/ - which matches all lines –and substitutes with “” blanks. outbound1=$(awk -v interface="$interface" -F'[: \t]+' \ '{ sub(/^ */,"");

For the line with the interface of interest, print field 10 which is the transmitted byte if ($1 == interface) print $10; }' /proc/net/dev)

FYI…..: cat /proc/net/dev Inter-| Receive | Transmit face |bytes packets errs drop fifo frame compressed multicast|bytes packets errs drop fifo colls carrier compressed enp0s17: 3377811 3414 0 0 0 0 0 0 226911 2426 0 0 0 0 0 0 enp0s8: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 lo: 36100 458 0 0 0 0 0 0 36100 458 0 0 0 0 0 0 jjm@jjm-VirtualBox:~/tmp$

$(awk -v interface=enp0s17 -F'[: \t]+' '{sub(/^ */,"")

The –F’[:\t]+’ sets the field separator to one or more colon and tabs. Awk processes the input from /proc/dev/net line by line….when the interface name matches, it prints the first numeric field which is the #bytes received.