Int'l Conf. Software Eng. Research and Practice | SERP'15 | 157

cnetmon: Ncurses-based Network Interface Activity Monitor

Steve Hutchinson1, John Wittkamper1, Jovina Allen1, Robert F. Erbacher2 1ICF International for US Army Research Laboratory, Adelphi, MD 20783 2US Army Research Laboratory, Adelphi, MD 20783

Abstract - This report illustrates the development and use of 2 Motivation a network interface activity monitoring tool named cnetmon. Server farms, cloud computing, compute clusters, and This tool is intended to aid system administrators and grid computing are all examples of a common technique to developers with network-oriented software projects. The main combine multiple computer systems into a cooperative objective for this project was to develop a capability to network of systems. These systems often intercommunicate monitor network activity for all or selected interfaces on a using two or NIs (on each system). Clustered- system simultaneously and continuously. We use a display computers are often rack-mounted for higher density and, as generated by the ncurses that is updated using a a result, often lack a keyboard or monitor; therefore, they are configurable interval. We show added capabilities including frequently managed and configured remotely via ssh or telnet interactive response to window-resizing using SIGWINCH. A over a network connection. During system configuration, novel debug-line display capability is provided to show installation, and testing, it is often difficult to determine dynamic debug messages on a dedicated line of the display. whether network traffic is being sent and received by each interface. Keywords: network traffic monitoring, network interface, systems administration, ncurses In general, such systems are built and configured in a central location and then shipped to remote locations to be added to other servers in a system rack or as a single 1 Introduction distributed sensor. cnetmon allows the installer to observe network traffic from each or all NIs to verify that the system cnetmon1 is a very lightweight command-line tool to seems properly configured for the installed environment. It display network traffic (packet activity) on any or all of the also does not require the use of the system GUI or network interfaces (NIs) on a Linux-based system It uses a Xserver/client because cnetmon will create tabular displays ncurses-library-based display that is compatible with any of all traffic using the LIBCURSES library for display on character-based pseudo terminal, and as such, does not any attached ASCII terminal emulator. cnetmon can be used require the use of the system graphical (GUI) from a remote location, accessed and invoked typically from or Xserver:DISPLAY. a ssh command-line, and can be invoked by any logged-in cnetmon is intended for use in the field for remote user; it does not require root-level access. Many techniques access into devices such as (network) sensors or other to observe or sample traffic from any NI require super-user network-attached Linux systems when an administrator with privileges, but obtaining elevated privileges is often user-level access needs to obtain a dynamic indication of all forbidden, hence a benefit of cnetmon. network traffic entering and leaving that system. Because it In this paper, we describe a few use-cases for cnetmon. does not use the GUI, the complexity and access First, cnetmon can be used on a laptop computer, which often requirements are very minimal. cnetmon can be invoked by has two NIs: wired (eth1) and wireless (wlan), along with the any logged-in user, it does not require sudo access, and it can internal loopback interface. Laptop-users often must operate within a typical secure shell (ssh) or telnet session. transition between networks without rebooting. cnetmon is easily invoked from a command window and will show all NI activities to verify communications to the desired network(s). Second, on a desktop or small server with multiple wired or wireless interfaces, cnetmon can show all network activity for each interface dynamically in this more ______complex network topology. Third, compute-server administration and configuration tasks are often performed 1Throughout this paper, Linux commands are set in an italic using a separate administrative system and command-line font. tools. cnetmon facilitates server configuration and testing and was developed for use in these more complex, multi-network 158 Int'l Conf. Software Eng. Research and Practice | SERP'15 |

environments. We frequently use one cnetmon window per devices, this function call is not intended for repeated server during configuration, development, and testing, to invocation to determine network traffic rates. Modern Linux obtain a real- picture of network inter-communications systems provide a /proc/ system to allow user-level and to verify proper configuration and operation. processes to easily read a wide variety of counts for devices; these values are maintained and updated by the kernel in a 3 Related work: bmon virtual file system, /proc/. The /proc/ file system was originally intended as a way to provide information about In the search for a user-level, multi-NI monitor, we processes in a system. As such, it also was a convenient noticed the “bmon” tool [1], which provides indications of means of exposing kernel information to a structured file network bandwidth utilization from multiple interfaces using system requiring only user-access rights to read this the /proc/ file-system [2] and a curses-interface. We use this information. A corresponding application programming strategy to implement a curses-based multi-interface activity interface (API) is provided for read and access — using tool, cnetmon, providing various command-line and key- sysctl (system control) calls to configure parameters of the press event-driven parameters to control the display and running kernel [3]. This capability was gradually introduced monitoring update interval. into systems starting as early as 1984; the current Although bmon was intended to show network implementation in Linux is as an extended, virtual file system bandwidth utilization, we liked its design paradigm using a contained only in memory and has directories for other ncurses display using periodic updates obtained from kernel information categories such as kernel-modules, file- /proc/net/. Our goal was not to show estimated bandwidth systems, interrupts, and devices including NIs, kernel utilization, but to show concurrent network activity measured messages, drivers, and CPUs. in terms of packet counts and transfer rates per sampling The cnetmon executable periodically examines the interval and accumulated for the session. /proc/net/dev file on the Linux system. These values are sampled on each loop cycle (by default, one second), which 4How it works is configurable on invocation or by pressing a number-key A long-standing problem for understanding network while running. Linux systems also maintain an uptime value, activity between (Linux or *nix) systems has been the the number of seconds since last rebooting. cnetmon saves requirement to obtain root or super-user privileges to access this date-time value launch (fork) time and displays the and configure devices, such as a NI. ifconfig is the Unix or session length time in the screen header section Linux command to display the status of NI devices on a system. Upon executing the ifconfig command, the following Contents of /proc/net/dev: information is produced on the console, shown below in Interface Figure 1. The first 6 lines pertain to the hardware and lo: network address parameters for each interface as well as the bytes 570671 status of the interface. The remaining lines show counts of packets 6267 transmitted and received packets, error counts, and finally the errs 0 interrupt number and buffers memory location. drop 0 user@asc2:~$ifconfig fifo 0 eth0Linkencap:EthernetHWaddr00:24:81:1c:fd:7d frame 0 inetaddr:10.0.0.16Bcast:10.0.0.255Mask:255.255.255.0 compressed 0 inet6addr:2601:a:4680:3e6:5cf:ea3d:eed0:64e0/64 multicast 0 Scope:Global bytes 570671 inet6addr:fe80::224:81ff:fe1c:fd7d/64Scope:Link packets 6267 inet6addr:2601:a:4680:3e6:224:81ff:fe1c:fd7d/64 errs 0 Scope:Global drop 0 UPBROADCASTRUNNINGMULTICASTMTU:1500Metric:1 fifo 0 RXpackets:370errors:0dropped:0overruns:0frame:0 colls 0 TXpackets:120errors:0dropped:0overruns:0carrier:0 carrier 0 collisions:0txqueuelen:1000 compressed 0 RXbytes:46300(46.3KB)TXbytes:20936(20.9KB) Interrupt:19Memory:f0500000Ͳf0520000 eth0: bytes 14797900909 Figure 1. Typical ifconfig output. packets 17797994 errs 0 drop 0 Although it is true that we could issue ifconfig fifo 0 repeatedly to obtain the configuration and counts for network frame 0 Int'l Conf. Software Eng. Research and Practice | SERP'15 | 159

compressed 0 For each interface and at each interval: multicast 3120 bytes 4116686178 SessionPKT Tnow  TP ]0[][ (1) packets 14414011 errs 0 IntervalPK Ts P Tnow  inowP ][][ (2) drop 0 fifo 0 (3) colls 0 SessionRat Be Tnow  TB Li]*1000/])0[][( carrier 0 compressed 0 IntervalRa te P Tnow  TP ]0[][ (4)

Command-line programs used for monitoring often 5 Implementation generate display data output in the form of one-line records and then render them into a scrolling console window. Very The design goals and requirements for cnetmon are to wide, or multi-line records, when scrolled like this, are periodically examine the network device-file in the /proc difficult to understand. Since network interface data is of this directory on a Linux system to: nature, a scrolling display will be difficult to use. Instead, we x Enumerate NIs use a display technique that renders these parameters in strict rows and columns such that the location of each on the x Collect traffic statistics screen does not change. This tabular process makes the changing parameters more obvious. Cell contents can change x Convert traffic counts to display quantities and units with the fixed regularity of the chosen update loop interval. x Allow a variety of command-line arguments Although this is a somewhat primitive display technique compared with GUI implementations, such a capability is We also provide a release /build capability for most easily provided by the Linux, Ncurses library. Ncurses Linux systems (including embedded devices, such as allows development of rather sophisticated tabular displays, Raspberry Pi, etc.) useful in situations in which a GUI display is unavailable (as After initialization during which command-line would be the case for many “headless” server or compute- arguments are parsed, cnetmon enters the main_loop. With clustered environments). each pass through main_loop, it obtains new counts for packets, bytes, errors, drops, collisions, etc., and calculates display values as requested updating the ncurses display at the end of each interval. Display values are calculated from 6 Ncurses library the following: Ncurses [4] stands-for “new” curses—a reimplementation of the “curses” library to use a text-based Li update loop interval, in seconds terminal to emulate a more dynamic interface that has some Tu Linux uptime in seconds (since reboot) attributes of a modern GUI. Curses was originally developed Tnow current Linux system time, epoch time seconds at the University of California at Berkeley for a Berkeley T0 cnetmon invocation start timestamp in epoch time Software Division (BSD) release around 1980. Ncurses seconds contains enhancements to curses and was made available session time length in seconds: (Tnow – T0) starting in the mid-1990s under a “Permissive P[i] packet count parameter from /proc/net/dev, at license” and not the General Public License (GPL) to afford time interval = i wide redistribution and linking to this library. B[i] byte count parameter from /proc/net/dev, at time interval = i 160 Int'l Conf. Software Eng. Research and Practice | SERP'15 |

interface row-sets that will fit in the new window and updates the display generation parameters in ncurses without resetting any of the current packet counts and rates. Figure 2 illustrates the various sub functions within main_loop, showing the generation and response to window size changes. Figure 3 below shows the help message with option switches and their meaning.

Help message: cnetmon -H

cnetmon [aD:ehHi:Lm:n:rtTu:] -a Show errors, data rate & totals (-ert) -D # Debug level (0-15) -e Show error data -H Help message -i name Ignore interface “name” -L List interfaces (with some statistics) -m name Show only interface “name” -n # Show total bytes for system uptime -r Show data rate -t Show data totals -T Show total bytes the “Quick” display -u # Update frequency, seconds (default 1)

Interactive: d/D Scroll down interface list q/Q Quit r/R Reset Session time u/U Scroll up interface list 1-9 Load value into interval time Figure 3. Usage help message.

Figure 2. Resize of window to reveal additional interface row-sets. 7 Usage scenario cnetmon, like many other ncurses programs, obtains An actual usage scenario is shown below. We have an terminal window geometry parameters from the terminal existing Linux server (Ubuntu 14.04 server) that will be used emulator when the program is launched. The combination of to provide various services to three separate networks, shown command-line switches will determine the number of rows in Figure 4 as Internet, MeshNet_1, and MeshNet_2. This needed to describe each interface; by default, the display will server does not have an attached display. We use a 2nd require one row per interface with the addition of three system with a terminal emulator and establish ssh session to header rows. Use of the “-a” switch will result in the display the server. We copy the cnetmon executable onto our of 7 or more rows per interface. cnetmon calculates how /home/user/ directory using scp (secure copy command). much space (height) is needed, and then it only displays as This session is established through the Internet and gateway many interface row-sets as can fit in the current window attached to ‘eth0’. Invoking cnetmon, we easily observe geometry. network activity on eth0 and no activity on eth1 or eth2. Use of up/down (U/D keys) allows the user to scroll up cnetmon does enumerate other interfaces such as the local or down an interface (row-set) at any time. To avoid loopback (lo) and a virtual bridge for use by associated requiring the user to quit, resize the terminal, and re-launch libraries to offer network address translation (NAT). in order to see additional interfaces, we support dynamic It is normal for local loopback to accumulate and show changes in window size using the SIGWINCH signal significant traffic during network traffic sessions as it is used (window change), which is supported by most terminal for process-process communications. We then connect a emulators. When the user changes the window geometry, the second network (MeshNet_1) gateway to eth1. This interface program receives the SIGWINCH signal and obtains new had been configured already to accept a DHCP-issued window geometry. cnetmon recalculates the number of Int'l Conf. Software Eng. Research and Practice | SERP'15 | 161

address. cnetmon clearly showed packets corresponding to auto eth2 DHCP requests and lease responses. After obtaining another user shell (ssh) to the server, we were able to access the web iface eth2 inet dhcp admin service on the gateway—to continue configuration of This change required restarting the networking services: this network. sudo /etc/init.d/networking restart We then connected MeshNet_2 gateway to eth2. cnetmon observed no activity on eth2. This required further cnetmon showed no eth2 activity after restarting the network investigation. /etc/network/interfaces is the configuration file services. Next, we tried a shutdown – reboot which did used by Linux systems to initialize and configure all NIs. reconfigure the interfaces and driver. After rebooting, eth2 had not yet been configured, and it was activated by cnetmon showed activity on all three physical NIs as well as adding the following to /etc/network/interfaces (these must the virtual loopback interface. be done as admin or root access):

Figure 4. Server with connection to 3 networks.

If the server had been pre-configured prior to 8 Compute-server example installation, it is likely that cnetmon would have allowed us to observe and verify each of the network gateway To illustrate additional capabilities of cnetmon, we additions in real-time at power-on. In this case, additional show results from running it on a blade-server with 5 NIs. configuration requiring root-level access was required. We This of server is common today and is used to were able to observe resulting network activity in real time populate the many rack spaces at internet and content using cnetmon in a second session window. hosting facilities. Although this server has 5 NIs as shown in the DEBUG: line, the display window geometry affords 162 Int'l Conf. Software Eng. Research and Practice | SERP'15 |

space for only 2 complete record sets, (lo) and (em1) shown in Figure 5. This server has been up (running) for just over 105 days and cnetmon has been running for 85 seconds, updating at 1-second intervals.

Figure 5. cnetmon –D 1 –r showing counts and rates for the session and for the last main_loop interval. Notice this also shows the –D flag, which adds an additional debug-message line to the display. Here, a cd_printf statement has been included to show the first and total number_interfaces available. Int'l Conf. Software Eng. Research and Practice | SERP'15 | 163

9 Debug print 11 References Coding and debugging an ncurses program can be very [1] Travis Graf. “bmon – bandwidth monitor and rate challenging. To facilitate debugging, we incorporate a debug estimator”, retrieved from https://github.com/tgraf/, June 15, display activated using a command-line switch. The code 2014. snippet below from cnetmon. :main() illustrates how to print messages to the debug message line using the ‘-D 1’ [2] Terry Dawson. “Exploring the /proc/net/ directory,” command-line argument. O’Reilly, retrieved from http://www.onlamp.com/pub/a/linux/2000/11/16/LinuxAdmi // Step through device list n.html, March 26, 2015. for (j = 0, i = first_interface; i <= number_interfaces; i++) { [3] M. Tim Jones. “Access the Linux kernel using the /proc int display = 1; filesystem”, IBM developerWorks Technical Library, 2006, if ( !prog_flags.first_time || prog_flags.match_inface ) retrieved from http://www.ibm.com/developerworks/library/l- { proc/index.html, April 15, 2015. display = 0; [4] . “Announcing ncurses release } 5.9”. Free Software Foundation, 2011, retrieved from } https://www.gnu.org/software/ncurses/, March 26, 2015. ///////////////////////////////////////// cd_printf("first_interface:%d number_interfaces:%d", first_interface, number_interfaces); ///////////////////////////////////////// refresh (); // Update display

10 Conclusions We have shown how cnetmon can provide easy access to the network activity from multiple interfaces, on multiple systems; however, the executable must first be available on each system. Therefore, we intend to provide cnetmon to be available as openSource code, providing the sources, documentation, a makefile, and a pre-compiled, 32-bit binary. Although most systems today are 64-bit architecture, the precompiled 32-bit binary should run on almost any Linux . A sophisticated developer-user can re- compile cnetmon from the sources, possibly adding new features and debugging cd_printf statements to facilitate the application and intended uses. We also will approach major Linux packagers and distribution groups, notably Red Hat, Fedora and Ubuntu, to encourage inclusion of cnetmon in future distribution releases.