LDM­7 Deployment Guidance Steve Emmerson, University Corporation for Atmospheric Research ([email protected]) ​ ​ Malathi Veeraraghavan, University of Virginia ([email protected]) ​ ​

Multicast network test

To install LDM­7, you will need a Linux/Unix machine and a copy of the LDM­7 software package. Also, you need to make sure the network that LDM­7 is going to run on supports and has enabled multicast.

To test whether multicast is supported, please follow the link https://github.com/shawnsschen/CC­NIE­Toolbox/tree/master/generic/mcast_test and obtain the two C files. One is for multicast sender and the other for receiver. If there are more than one network interfaces in the sender’s system, you have to manually update the routing table with one new entry: route add 224.0.0.1 dev eth1. Then use route ­n to check the updated routing ​ ​ ​ table. You will be able to see something like:

Note that the entry we added is only matching a particular multicast address 224.0.0.1, not a subnet. If the multicast address changes, you need to manually change the configuration or use a subnet address.

Now you have a multicast network (though not sure if it works), next step is to upload the mcast_send.c onto the sender system and mcast_recv.c onto all the receiver systems (more than one in a multicast network). Use the following command to compile them, respectively: gcc ­o send mcast_send.c gcc ­o recv mcast_recv.c send is a multicast sender­side test program, it sends out a “hello multicast world” message every second to multicast group 224.0.0.1:5173. recv is a multicast receiver­side program, ​ ​ ​ which receives the “hello multicast world” message from the multicast group 224.0.0.1:5173. ​ ​ Both send and recv take one input argument, the interface IP address, to inform the OS which interface they would like to use for multicast. For example, we usually use eth1 in GENI because it is the data­plane interface. Only on the same plane will this test program work. send listening on eth0 and recv listening on eth1 will not work since the management plane and data plane are physically disconnected. You can do a ping to test the reachability between interfaces.

Once everything has settled, execute ./send on the sender system, ​ and execute ./recv on the receiver system. If working well, you will ​ be able to see “hello multicast world” on all systems.

LDM­7 Installation

As the multicast network itself is working well, we can proceed and start installing LDM­7 on all the nodes in this network. According to UCAR, here are the platform requirements before installing: https://www.unidata.ucar.edu/software/ldm/ldm­6.12.14/basics/platform.html. ​ ​

In addition, pax, sudo, libyaml­dev(el), sysstat, ntp, gcc 4.8, g++ 4.8 (gcc­c++ for CentOS), make, autoconf, m4, and automake are needed for compiling. Ubuntu: sudo apt­get install libxml2­dev libpng­dev zlib1g­dev sudo apt­get install pax libyaml­dev sysstat ntp autoconf m4 automake (Check gnuplot separately, vmstat, netstat, top, etc. should already be installed) Centos: yum install libxml2­devel yum install libpng­devel yum install zlib­devel

Note that the development packages will be found in your package managers as ­dev in Ubuntu/Debian and ­devel in CentOS. Special Notes: zlib package can be found as zlib­dev(el) in package manager.

For CentOS 6, the default gcc/g++ version provided by the system is too outdated. You have to manually upgrade it to 4.8. To do this: $ wget http://people.centos.org/tru/devtools­2/devtools­2.repo ­O /etc/yum.repos.d/devtools­2.repo $ yum install devtoolset­2­gcc devtoolset­2­binutils devtoolset­2­gcc­c++ $ mv /usr/bin/gcc /usr/bin/gcc_old $ mv /usr/bin/g++ /usr/bin/g++_old $ ln ­s /opt/rh/devtoolset­2/root/usr/bin/gcc /usr/bin/gcc $ ln ­s /opt/rh/devtoolset­2/root/usr/bin/g++ /usr/bin/g++ $ gcc ­v You will see gcc is upgraded to 4.8.

For CentOS: To give sudo access to users in the wheel group, execute visudo and search for the ​ following line: # %wheel ALL=(ALL) ALL

For Ubuntu: Look for the following line (it may not be commented) # %sudo ALL=(ALL) ALL

Uncomment it by removing the hash mark if commented. If not commented do nothing. And save and quit. Now, upload LDM­7 package onto the nodes. Next, create an ldm user if it does not exist. To ​ create the user, execute: Centos: adduser ldm ​ Ubuntu: adduser ldm

Then use passwd ldm to set a password. We also add ldm into the sudo list by executing the ​ ​ following command. Centos: usermod ­aG wheel ldm ​ Ubuntu: usermod ­aG sudo ldm

(Can check if the following command was successful with: $ groups ldm) ​ ​

After the account is created, all the following operations will be done as user ldm. We switch to ​ ​ ldm home and proceed: su ­ ldm. ​ ​

Move your uploaded LDM­7 package to ldm home directory, you should expect something like ldm­6.12.15.42.tar.gz. We unpack it with: gunzip ­c ldm­6.12.15.42.tar.gz | pax ­r '­s:/:/src/:' ​ ​ Then we will compile it. Go into the source folder: cd /home/ldm/ldm­6.12.15.42/src. Use the ​ ​ following commands to compile (Note that if make distclean returns “No rule to make target…”, ​ it is fine, just proceed to next step): Centos: $ sudo /etc/init.d/rsyslog start (CentOS 6) $ sudo systemctl start rsyslog.service (CentOS 7) Ubuntu: $ sudo start rsyslog (restart if already running) $ make distclean $ find ­exec touch \{\} \; $ ./configure ­­with­debug ­­with­multicast ­­disable­root­actions CFLAGS=­g CXXFLAGS=­g $ make install $ sudo make root­actions

If no error is reported, you will have a fresh installed LDM­7. We are going to configure it. For the sender side (upstream LDM­7), find the configuration file: vi $HOME/etc/ldmd.conf. Delete ​ ​ everything inside and add the following lines: MULTICAST ANY 224.0.0.1:38800 1 10.10.1.1 ALLOW ANY ^.*$ EXEC "insert.sh" EXEC "cpu_mon.sh" EXEC "tc_mon.sh" 10.10.1.1 is the data­plane IP address for the sender, you can replace it with the actual IP. 224.0.0.1 is the multicast address we are going to use, you should make sure it is the same as what you set in the routing table. For the receivers, open the same configuration file and delete everything. Add the following line: RECEIVE ANY 10.10.1.1 10.10.1.x 10.10.1.1 is the IP of the sender, 10.10.1.x is the IP of the receiver’s data­plane interface. ​ ​ ​

We need to configure the LDM­7 product queue settings as well. For LDM­7 sender, execute regutil ­s 5G /queue/size in ldm home. For receivers, execute regutil ­s 2G /queue/size ​ instead. Then execute regutil ­s 35000 /queue/slots. Finally, execute regutil ­s 10.10.1.x ​ ​ ​ /hostname, where 10.10.1.x is the corresponding IP on that node. ​ ​ ​

Some scripts for measurement are also needed. Go to https://github.com/shawnsschen/CC­NIE­Toolbox/tree/master/generic/misc and get util folder. ​ Put util onto the sender’s ldm home. Run chmod +x util/* to make sure all scripts are ​ ​ executable. Run chown ­R ldm.ldm util to change the permission. Then open ​ /home/ldm/.bash_profile and add a new line at the bottom: export ​ PATH=$PATH:$HOME/util. Then touch /home/ldm/.bash_profile. Put the github repo ​ ​ ​ containing the test input in ldm home: git clone ​ https://github.com/shawnsschen/LDM6­LDM7­comparison.git. ​

For bandwidth measurement, we need to use sysstat. Configure it to log every minute: sed ­i ­e \'s/*\/10/*\/1/g\' /etc/cron.d/sysstat rm /var/log/sa/* And make sure it is running: service crond start service sysstat start

The last thing we need to do is to configure tc htb on the sender. Run the following commands ​ with root permission (assuming eth1 is the data­plane interface): tc qdisc del dev eth1 root tc qdisc add dev eth1 root handle 1: htb default 2 tc class add dev eth1 parent 1: classid 1:1 htb rate 20mbit ceil 20mbit tc qdisc add dev eth1 parent 1:1 handle 10: bfifo limit 600mb tc class add dev eth1 parent 1: classid 1:2 htb rate 20mbit ceil 20mbit tc qdisc add dev eth1 parent 1:2 handle 11: bfifo limit 600mb tc filter add dev eth1 protocol ip parent 1:0 prio 1 u32 match ip dst 224.0.0.1/32 flowid 1:1 tc filter add dev eth1 protocol ip parent 1:0 prio 1 u32 match ip dst 0/0 flowid 1:2 20 Mbps (per­class sending rate) and 600 Mb (per­class BFIFO size) are the key parameters for LDM­7, as the experimental settings change, you can replace them with new values.

So far, we have finished all the settings and are good to go! But wait a second, open util/insert.sh on the sender. See the sleep xxx line? Change it with the value: m*40, where m ​ ​ ​ ​ is your number of receivers. This is to make sure the sender would not start sending before all receivers are connected. You can have 40 seconds to start a receiving LDM. You can also set it to a larger value to buy yourself more time. Then, on all nodes, execute run_ldm ldmd_test as ​ user ldm in the ldm home.

Post­execution processing

As LDM­7 is running, you can execute less var/logs/ldmd_test.log to view the log file and see ​ how the run is going. After one hour (by default we are replaying the 1­hour NGRID file­stream), the run is finished. Check the log again to make sure if it stops at file 28165. If so, then you can ​ ​ fetch all the logs and measurements. To fetch the log files, you can use scp or directly ftp to move the log file back to your local machine. But given all the files have the same name, I recommend to name the log properly before fetching. Name it like ldmd_test_10.10.1.x.log to ​ avoid confusion. On the sender, there are some measurements to collect. In ldm home, execute: sar ­n DEV | grep eth1 > bandwidth.log. Then you will find a bandwidth.log and ​ ​ ​ cpu_measure.log. These two are the CPU measurement and bandwidth measurement. Be ​ sure to move them to your machine as well.

With the log files retrieved, we need to parse the log files to get throughput and FFDR. Go to https://github.com/shawnsschen/CC­NIE­Toolbox/blob/master/generic/LogParser/ldm_parser.py and get it under the same folder as the log files. Also you will need to put the input file­stream metadata (https://github.com/shawnsschen/LDM6­LDM7­comparison/blob/master/std_data) ​ ​ under the same folder. Then execute python ldm_parser.py 1hr_NGRID.txt 10.10.1.x.log ​ 10.10.1.x.csv, where 10.10.1.x.csv is the generated file associated with the log. ​