Local Service Monitoring Status of Linux Operating Systems

MASARYK UNIVERSITY FACULTY OF INFORMATICS

Local service monitoring status of Linux operating systems

BACHELORTHESIS

Jakub Svoboda

Brno, Spring 2012 Declaration

Hereby I declare, that this paper is my original authorial work, which I have worked out by my own. All sources, references and literature used or excerpted during elaboration of this work are properly cited and listed in complete reference to the due source.

Jakub Svoboda

Advisor: Mgr. Pavel Tuˇcek

ii Acknowledgement

I’d like to thank to my advisor Mgr. Pavel Tuˇcekfor patiency, guidance, invaluable assistance and encouragement. I’d also like to thank to Jan Koneˇcnýfor programming advices in the course of designing the application.

iii Abstract

Theoretical part of the thesis analyzes methods of monitoring Linux operating system and monitoring requirements of the Institute of Computer Science. In the practical part of the thesis, Linux monitoring application is designed and implemented. The application is developed as a part of ICS’ Large Enterprise Monitoring (Lemon) project.

iv Keywords

Linux, monitoring, Mono, Lemon, LinMon

v Contents

1 Introduction ...... 1 2 Operating system monitoring in general ...... 2 2.1 Operating system purpose ...... 2 2.2 Reliability of operating system ...... 2 2.3 Reasons for monitoring ...... 3 2.4 Existing GNU/Linux-compatible solutions ...... 3 2.4.1 SYSSTAT ...... 4 2.4.2 Dstat ...... 4 2.4.3 vmstat ...... 4 2.4.4 Collectd ...... 4 2.4.5 Munin ...... 5 2.4.6 Nagios, Shinken and Icinga ...... 5 2.4.7 PCP ...... 5 2.4.8 Xymon ...... 5 3 System monitoring at the Institute of Computer Science ...... 7 3.1 Lemon project ...... 7 3.1.1 Lemon architecture ...... 7 Event generators ...... 7 Transport system ...... 8 Processing system ...... 8 Web service and presentation application ...... 9 3.2 Monitoring of operating systems ...... 9 3.3 Requirements for monitoring of GNU/Linux machines ...... 9 3.3.1 Report format ...... 9 3.3.2 Scope of monitoring ...... 10 3.3.3 Runtime requirements ...... 11 3.3.4 Packaging requirements ...... 11 3.4 Suitability of existing applications ...... 11 4 Implementation of GNU/Linux monitoring application ...... 12 4.1 Chosen goals ...... 12 4.1.1 Operating system and programming language ...... 12 4.1.2 Configuration ...... 12 4.1.3 Types of reports ...... 13 4.1.4 Monitored areas ...... 13 Disk usage ...... 13 Iptables ...... 13 Network interfaces ...... 17 Users ...... 17 Groups ...... 18 Recent logins (both physical and remote) ...... 18

vi Recent physical logins ...... 18 Recent login attempts over ssh (both successful and unsuccessful) ...... 18 Installed packages ...... 19 Available package updates ...... 19 Recent reboots ...... 19 Time of last reboot ...... 20 Processes using CPU above a set limit ...... 20 Information about operating system ...... 20 System installation date ...... 20 4.2 Application architecture ...... 21 4.2.1 Classes and interfaces ...... 21 4.2.2 Error handling ...... 22 4.2.3 File access and permissions ...... 23 4.2.4 Settings file and alternative settings ...... 23 4.2.5 Report production ...... 24 4.2.6 Report comparison ...... 24 Comparison without a primary key ...... 25 Comparison with a primary key ...... 25 5 Testing of LinMon ...... 26 5.1 Testing environment ...... 26 5.1.1 Personal computers ...... 26 GNU/Linux ...... 26 Other than GNU/Linux ...... 27 5.1.2 ICS server ...... 27 Problems that occurred during testing ...... 27 5.2 LinMon in production environment ...... 28 6 Conclusion ...... 29 Bibliography ...... 29

vii 1 Introduction

Masaryk University operates a large computer network. The need to manage the network effectively resulted in a computer monitoring project called Lemon. Lemon is a system that collects data about computers using agents installed on the computers and processes them on centralized servers. It allows Masaryk University to check computer health, localize problems and to prevent misuse. Lemon is modular and consists of several components. Lemon is capable of monitoring only Windows systems at this time, with planned Linux support. This thesis deals with the Linux monitoring agent. The second chapter of the thesis describes monitoring of operating systems in general and Linux-compatible monitoring applications. The third chapter presents system monitoring at the Institute of Computer Science, requirements for the monitoring application and suitability of existing applications. The fourth chapter presents chosen goals and describes architecture of the developed application. The fifth chapter talks about testing and its results. The last chapter evaluates the developed application and its impact on system monitoring at the Masaryk University. The thesis is accompanied by three Appendices. Appendix A lists all source code, Appendix B contains class diagrams and Appendix C is an Administrator’s Guide which helps administrators with using LinMon.

1 2 Operating system monitoring in general

2.1 Operating system purpose

Computers and the software they run are complex mechanisms with an inherent risk of failure1. Most computers today are run with an operating system so that multiple applications can run at once, use the operating system routines instead of accessing the bare hardware and use safe mechanisms of inter-process communication.2

2.2 Reliability of operating system

The apparent advantage of running an operating system comes with a cost. A failure in the operating system can result in all the applications being unable to run properly or to run at all. (Not that an application running on the bare metal cannot fail, but OS adds additional possible point of failure apart from the application itself.) Microkernel and nanokernel operating systems try to solve this problem by minimizing the amount of code comprising the very core of the operating system with independent modules that can be restarted without affecting the rest of the system in a case of failure3. There are even operating systems capable of taking snapshots—containing state of all running programs, memory and necessary data—and then recovering to the latest snapshot in case of failure, with no need to restart programs or check filesystems, effectively restarting quickly and losing only last few minutes of data. This property is called orthogonal persistence4. However, general-purpose operating systems are not made in such a safe and uninterruptible way. They are made of interdependent components and failure can propagate throughout the system. The majority of workstations and a large share of servers run general-purpose operating systems nowadays5. When a failure occurs, a manual intervention may be required. The system administrator6 must collect all the information they need to find and resolve the problem. This might include: • Errors logged in a system log.

1. Proving the correctness of software is a very hard to nearly impossible task. Bruce Schneier has an informative post with a discussion on his blog [1]. 2. Solutions going against this approach do exist—such as running a single application on bare hardware or using exokernel (such as MIT Exokernel Operating System [2]) merely for multiplexing bare hardware, isolating the applications from each other—but they are limited to microcontrollers and embedded systems, as in the case of a single application, or simply very rare, as in the case of using exokernel. 3. Such an operating system is QNX Neutrino [3]. 4. Such operating systems are KeyKOS [4] and EROS [5], for instance. 5. According to W3Techs statistics from March 25th 2012, “Unix” and Windows web servers together have 100% market share. http://w3techs.com/technologies/overview/operating_system/all 6. I will call anyone who is experienced in maintaining and repairing an operating system a “system administrator” for the purpose of this thesis.

2 2. OPERATINGSYSTEMMONITORINGINGENERAL

• Disk usage information. (Is there a disk that is full that consequently caused the system to fail?)

• Installed software and changes in installed software. (Has a new software or software update caused the failure?)

• Users, groups and changes in users and groups. (Is there a new user who may have run something that caused the failure?)

• Logins to the system. (The system administrator can look who was logged in at the time of the failure and limit the scope of the investigation.)

• Network interfaces. (Was there a change in the network configuration that caused the system to fail?)

• Firewall rules. (Was there a change of the firewall rules or does a rule collide with something?)

• Reboots. (If and when the machine was rebooted.)

• What is the exact version of the operating system (there might be a bug specific to this OS version).

2.3 Reasons for monitoring

The need for frequent and repetitive collection of information makes automation relevant. And not only that. There are cases where it is useful to monitor operating system state. A history of the states can be inspected later. This can be very useful for system administrators, saving time when analyzing system state and providing evidence. Therefore, operating system monitoring—the act of collecting information about system state through the time and storing such information—may be useful for viable system administration. Moreover, storing all the information in a central location can further increase administration efficiency.

2.4 Existing GNU/Linux-compatible solutions

The thesis deals with monitoring of GNU/Linux operating system. Hence only GNU/Linux- compatible monitoring software is mentioned in this section. There are numerous applications ranging from monitoring just the load on the local machine to monitoring various properties across a whole network.

3 2. OPERATINGSYSTEMMONITORINGINGENERAL

2.4.1 SYSSTAT

SYSSTAT is a performance analysis toolkit, monitoring state of the operating system it runs on. It contains several monitoring tools: Sar, sadc, sadf, iostat, nfsiostat, cifsiostat, mpstat, pidstat. Some of the monitored properties are: I/O operations, CPU utilization, interrupt statistics, process statistics, network statistics, NFS activity, system load, TTY device activity. [6] SYSSTAT is implemented in the C language. Homepage of SYSSTAT is http://sebastien.godard.pagesperso-orange. fr/index.html.

2.4.2 Dstat

Dstat is a performance analysis tool, monitoring state of the operating system it runs on. Some of the monitored properties are: Disk read and write operations, CPU utilization, interrupt statistics, process statistics, network statistics, system load. [7] Dstat is implemented in Python. Homepage of Dstat is http://dag.wieers.com/home-made/dstat/.

2.4.3 vmstat

Vmstat is a performance analysis tool, monitoring state of the operating system it runs on. Some of the monitored properties are: Process statistics, memory statistics, swap usage, I/O statistics, interrupt statistics, CPU utilization. [8] Vmstat is implemented in C7. Vmstat doesn’t have its own homepage. Linux manual page can be found at [8].

2.4.4 Collectd

Collectd is a performance analysis tool, monitoring state of the operating system it runs on and optionally monitoring state of networked systems. Collectd collects information using plugins and is extensible that way. Some of the available plugins monitor the following: Apache status, battery charge status, CPU state, DNS traffic statistics, iptables, memory utilization, NFS usage, nginx statistics, OpenVPN statistics, sensors status, swap usage, syslog messages, thermal information, uptime, logged users, UUID. It also supports monitoring via SNMP using a plugin [9]. Networking support allows more “client” collectd instances to send data to a “server” collectd instance using a unicast address as well as sending data to multiple “server” instances using multicast [10]. Collectd is implemented in C. Collectd homepage is http://collectd.org/.

7. Source code is available at http://lxr.linux.no/linux+v3.3.4/mm/vmstat.c, for instance.

4 2. OPERATINGSYSTEMMONITORINGINGENERAL

2.4.5 Munin Munin is a performance analysis tool, monitoring state of networked computers or state of the operating system it runs on. Munin supports collecting information using plugins and is extensible that way [11]. Some of the available plugins are: Apache, apt, asterisk, disk, logins, mysql, network, nfs, nginx, ntp, processes, sensors, system. All the plugins are placed in a github directory with little to no documentation [12, 13]. It also supports monitoring via SNMP using a plugin [14]. Munin is implemented in Perl. Munin homepage is http://munin-monitoring.org/.

2.4.6 Nagios, Shinken and Icinga Nagios is a performance analysis tool, monitoring state of network and state of networked computers running an agent8. Some of the monitored properties are: “Network services (SMTP, POP3, HTTP, NNTP, PING, etc.)” and “host resources (processor load, disk usage, etc.)” [15, 16, 17]. Nagios collects information using plugins and is extensible that way. It can monitor whatever property a plugin supports [18]. Nagios is implemented in C. Nagios homepage is http://www.nagios.org/. Shinken is a tool similar to Nagios and compatible with Nagios plugins [19, 20]. Shinken homepage is http://www.shinken-monitoring.org/. Icinga is a tool similar to Nagios and compatible with Nagios plugins [21]. Icinga homepage is https://www.icinga.org/.

2.4.7 PCP PCP (Performance Co-Pilot) is a performance analysis toolkit, monitoring state of network and networked computers. Some of the monitored properties are: Hardware status, kernel status, applications status, CPU utilization, disk usage, memory statistics, swap usage, network statistics, data about databases, apache, sendmail and more. PCP supports collecting information using plugins and agents [22]. PCP is implemented in C++. PCP homepage is http://oss.sgi.com/projects/pcp/.

2.4.8 Xymon Xymon is a performance analysis toolkit, monitoring state of networked operating systems. Some of the monitored properties are: Network services status, disk utilization, logfiles, processes. Xymon is extensible and operating system status is collected usingan agent [23].

8. Agent is required only for host resources monitoring.

5 2. OPERATINGSYSTEMMONITORINGINGENERAL

Xymon is implemented in C. Xymon homepage is http://xymon.sourceforge.net/.

6 3 System monitoring at the Institute of Computer Science

3.1 Lemon project

Lemon (Large Enterprise MONitoring) is a monitoring system developed at the Masaryk University’s Institute of Computer Science. It consists of event generators, a transport system, a processing system, a database, a web service and a presentation application. Lemon monitors most computers in the property of Masaryk University. The administrators can efficiently see what computers are badly configured, detect intrusions and prevent computer misuse or non-study activities in study rooms, for instance. Each monitored computer is equipped with a software component, called an event generator, that logs events to a local folder and a software component, called a transport system, that collects the logs from the local folder and sends them to a spooler server through network. There are several spooler servers in a tree hierarchy, forwarding the logs to a central spooler server that in turn sends them to an event processing server. Event processing server saves all the logs to a database and creates additional realtime warnings based on the incoming data. A web service reads the database and receives warnings from the event processing server. The web service presents data in a human- readable form using a presentation application. LinMon is one of the event generators.

3.1.1 Lemon architecture

Lemon Shared

WinMon Lemon Lemon Lemon WCF WLET Pool Core Route Zappa Namtar Frank 2

Bacula

planned LinMon DB

The architecture of Lemon

Event generators Lemon event generator is a computer program running as a Windows service or a GNU/Linux daemon on each monitored computer. Currently, there are three event generators —WinMon, Namtar2 and Bacula (with planned LinMon).

7 3. SYSTEM MONITORING AT THE INSTITUTEOF COMPUTER SCIENCE

WinMon is event generator for Microsoft Windows. Among other data sources, it subscribes to Windows Event Log and reads various information from there. WinMon produces full daily reports, containing all monitored information, and periodic differential reports containing only changed information. Namtar2 is another event generator for Microsoft Windows, capable of logging only user log in and log off, system start and shutdown and screen lock. It does not use Windows Event Log. Bacula reports only information about client PC backups made by the Bacula application. LinMon is event generator for GNU/Linux and produces reports similarly to WinMon. It can be deployed on both servers and workstations. All event generators save the logs to a local folder on the client PC.

Transport system Lemon transport system comprises of client and server parts. The client part is a computer program running as a Windows service or a GNU/Linux daemon alongside the event generator(s) on client PCs—called WLETdispatcher—and the server part is another program running on a server, called WLETspooler[24]. WLETdispatcher collects the logs from the local PC and sends them over network to a WLETspooler running on a server. WLETspooler receives and saves the logs. There is not a sole server fetching all the logs; there are multiple servers in a tree hierarchy. Each of them, except the top one, runs both WLETdispatcher and WLETspooler, receiving logs from subordinate machines using WLETspooler and forwarding them to superordinate machines using WLETdispatcher. In the top of the hierarchy is altariel2.ucn.muni.cz server that hands the logs over for storage and further processing. There is a new NamtarRemoting transport system based on Microsoft .NET Remoting for both Windows and GNU/Linux in the works, eliminating the tree hierarchy.

Processing system All the logs are saved to a database and also processed by a system called CEP, or Complex Event Processing. It consists of “Lemon Pool”, “Lemon Core”, “Lemon Shared” and “MQ”, a message queue. Lemon Pool receives logs, saves all the raw (not processed) logs to a database con- nected to Lemon Pool, additionally processes the received logs and optionally sends a message to the Message Queue. Lemon Pool has a modular architecture, which allows for customization of the processing. It can, for instance, issue a warning when certain criteria in the received logs are met (examples: A suspicious person logged in on a computer, new administrator account was created on a computer). Output of such processing is not saved to the database; it is send via the Message Queue to Lemon Core instead. Lemon Core executes scenario logic, which involves complex processing of data and extracting the true meaning. An example: A sole warning about a new user account

8 3. SYSTEM MONITORING AT THE INSTITUTEOF COMPUTER SCIENCE

should raise a high priority alarm as this is probably done by an attacker, while warnings about new user accounts on all computers in a room should raise only a low priority notice, as this is probably a legitimate action done by the IT staff. Lemon Core does not save data, it only processes them in real time. All the Complex Event Processing uses a type library called Lemon Shared. CEP uses strong typing for handling data.

Web service and presentation application A Windows Communication Foundation-based web service, called Zappa, and a presentation application, called Frank2, communicate with CEP through an interface called Lemon Route. Zappa exists so that all the heavy work and data communication is done on server and Frank2 merely displays the results. Zappa can ask CEP for data from database and is able to receive real-time warnings and notifications from Lemon Core. Frank2 is what administrators are supposed to work with from the entire Lemon software. It sports a graphical user interface, presenting data in a human-readable textual and graphical form.

3.2 Monitoring of operating systems

According to the Institute of Computer Science and [24, page 15], it is desired to monitor Windows workstations and servers and about 15 GNU/Linux servers. The Lemon infrastructure used to be capable of monitoring Windows machines only, Bc. Andrea Cíkováˇ [24] replaced Windows-only file dispatcher and spooler “WLET” with multiplatform “NamtarRemoting”, and LinMon adds the capability to monitor GNU/Linux machines, thus satisfying the remaining requirements.

3.3 Requirements for monitoring of GNU/Linux machines

3.3.1 Report format Windows machines are already being monitored using the WinMon application. Win- Mon produces reports in a specific format[25]. The Lemon infrastructure consumes and processes these reports. According to [26, page 50, bullet 1], event generator reports must be in an XML format resembling WinMon reports. The exact schema of WinMon reports is unfit for a GNU/Linux monitoring application. For GNU/Linux monitoring application’s reports, a much looser XML schema was proposed by Mgr. Pavel Tuˇcek,describing only the mandatory metadata vital for classi- fying the reports. The rest was advised to be inspired by WinMon report format with platform-specific issues in mind. Appendix A lists the resulting schemas for allthe produced reports.

9 3. SYSTEM MONITORING AT THE INSTITUTEOF COMPUTER SCIENCE

WinMon was developed by Bc. Petr Smˇelý[27].

3.3.2 Scope of monitoring GNU/Linux monitoring application should be on par with WinMon. WinMon monitors the following information[26, page 33]: • antivirus state

• firewall

• network interfaces

• system updates

• users and groups

• operating system logs (EventLog)

• operating system parameters (uptime, installation date, OS version) After debate with Mgr. Pavel Tuˇcek,the following list of monitored areas under GNU/Linux was established: • disk usage

• iptables

• network interfaces

• users

• groups

• recent logins (both physical and remote)

• recent login attempts over ssh (both successful and unsuccessful)

• installed packages

• available package updates

• list of recent reboots and time of last reboot

• list of processes using CPU above a set limit

• information about operating system

• system installation date More on the reports generated by LinMon is in 4.2.5 on page 24.

10 3. SYSTEM MONITORING AT THE INSTITUTEOF COMPUTER SCIENCE

3.3.3 Runtime requirements Institute of Computer Science requires that the GNU/Linux monitoring application (also known as “event generator” in the Lemon infrastructure) is implemented in the C# programming language and runs under Mono runtime [28]. C# is standardized as ISO/IEC 23270:2006 and Common Language Infrastructure is standardized under three additional standards (“ISO/IEC 23270:2006 (CLI), ISO/IEC TR 23272:2006 (CLI, XML Libraries) and ISO ISO/IEC TR 25438:2006 (CLI, Common Generics)”)[29]. Since most of Lemon is programmed in C#, this requirement makes it possible for existing developers of Lemon to embrace and maintain the GNU/Linux event generator with minimum effort and no new developer tools1.

3.3.4 Packaging requirements There were no installation package requirements placed by the Institute of Computer Science. Since the resulting application is one standalone executable, no installation packaging was chosen.

3.4 Suitability of existing applications

None of the known monitoring applications—listed in 2.4 on page 3—is implemented in C# language. This poses a substantial problem in picking an existing monitoring application. Either the requirements would have to be lifted by the Institute of Computer Science, or a complete rewrite into C# would have to be done, in order to pick and re-use an existing monitoring application. If the programming language problem is ignored, there are other aspects to evaluate. Monitoring of the desired areas is only a minor hurdle. The monitored areas can be added by means of writing appropriate plug-ins for several existing monitoring applications, notably Collectd, Munin and Xymon. However, a substantial effort must have been made to adapt output reports to the desired format, had an existing application been used. The very same problem had been analyzed by the creator of WinMon, Petr Smˇelý,in his bachelor thesis. His chosen solution of the problem was to build a new monitoring application from scratch [27, page 6]. Institute of Computer Science placed such—relatively strict—requirements, so that no existing monitoring application could have been used. Developing a new monitoring application in exact accordance with the requirements turned out to be the most effective approach. LinMon is made in such a modular way that it is possible to create a LinMon module reading data from another monitoring application, although there was no need to do this yet. More on the architecture of LinMon is in the following chapter.

1. The standardized Common Language Infrastructure makes it possible for an executable compiled with, say, Microsoft .Net compiler to run under, say, GNU/Linux with Mono runtime.

11 4 Implementation of GNU/Linux monitoring application

As described in the last chapter 3.3 on page 9, there was a vision how the monitoring application should be made and what it should do. The next logical step was to establish goals—these are described in the following section of this chapter, “Chosen goals”. In the rest of this chapter is described how the goals were met, while designing and developing the application.

4.1 Chosen goals1

There are multiple goals in several categories. Where the application runs, what kind of programming is used, how the application is configured, which areas are monitored and what types of reports are produced.

4.1.1 Operating system and programming language As described in 3.3.3 on the previous page, ICS wanted a C# application running under GNU/Linux using the Mono framework, built from scratch. Considering the reasons mentioned in 3.4 on the preceding page, this hasn’t changed and was set as a goal. It was also set as a goal that the application is modular. It is possible to easily add a new class for monitoring of a new area, without changing anything outside the new class except one line to register the new class. More about this is in 4.2.1 on page 21.

4.1.2 Configuration WinMon can be configured using a Windows group policy or a local settings file. Local settings file is used only if group policy is not set. If neither of them is available, default settings are used. WinMon checks for changes in settings and reflects them. Configuration of WinMon is discussed in [27, page 19]. Since there is no way to use group policies on GNU/Linux machines the way it is used on Windows machines, the GNU/Linux monitoring application is able to use an “alternative” settings file instead of the group policies. If no “alternative” settings fileis found, values from the local one, located in /etc/linmon/, are used. If no local settings file in /etc/linmon/ is found, a new one is created, containing default values. Location of the “alternative” settings file is specified in the local settings2 file . Unless configured

1. In order to refrain from excessive use of “should” word, present simple tense is used in this chapter. Because all the goals were eventually met, the use of present simple tense is not in conflict with reality. 2. This way, an alternative settings file location can be specified once when deploying LinMon—pointing to a network share, for instance. It is then possible to change settings for multiple clients anytime by editing the settings file on the network share. Setting the alternative settings file location permanently in the source code would result in the need to recompile LinMon for different kinds of uses, so the location is configurable in the standard settings file.

12 4. IMPLEMENTATION OF GNU/LINUXMONITORINGAPPLICATION

differently in the “alternative” settings file, everything the monitoring application saves and all settings and state information it reads are located in /etc/linmon/.

4.1.3 Types of reports The GNU/Linux event generator produces two kinds of reports—differential reports and daily full reports. The daily full reports are produced once a day and contain all information from all monitored areas. The differential reports are produced only when a change occurs and generally contain only the changed information.

4.1.4 Monitored areas It has been specified what is monitored and how exactly each area is monitored. The list of monitored areas was first mentioned in 3.3.2 on page 10. The way how exactly iseach area monitored is described in the following subsubsections.

Disk usage df utility is used. The output is structured into items, each describing one mounted filesystem and each containing subitems with information about total size, used space, available space, used space in percents and mountpoint path. Changes in disk usage are reported using only the information that changed, thus not reporting the information that did not change.

Iptables iptables is a GNU/Linux application that facilitates setting up the firewall in Linux kernel. There are three ways to export iptables configuration: 1. iptables -L -n 2. iptables-save, 3. iptables-save|iptables-xml The output generated by each one is quite distinct from the others and suitable for different uses. Sample output of iptables -L -n command:

1 Chain INPUT ( policy DROP) 2 target protoptsource destination 3 ACCEPT tcp −− 194.145.181.170 0.0.0.0/0 tcp flags:!0x17/0x02 4 ACCEPT udp −− 194.145.181.170 0.0.0.0/0 5 ACCEPT tcp −− 194.145.181.158 0.0.0.0/0 tcp flags:!0x17/0x02 6 ACCEPT udp −− 194.145.181.158 0.0.0.0/0 7 ACCEPT a l l −− 0.0.0.0/0 0.0.0.0/0 8 ACCEPT icmp −− 0.0.0.0/0 0.0.0.0/0 limit: avg10/sec burst5 9 DROP a l l −− 0.0.0.0/0 255.255.255.255 10 DROP a l l −− 0.0.0.0/0 10.0.2.255 11 DROP a l l −− 224.0.0.0/8 0.0.0.0/0

13 4. IMPLEMENTATION OF GNU/LINUXMONITORINGAPPLICATION

...

63 Chain OUTBOUND (1 references ) 64 target protoptsource destination 65 ACCEPT icmp −− 0.0.0.0/0 0.0.0.0/0 66 ACCEPT tcp −− 0 . 0 . 0 . 0 / 0 0 . 0 . 0 . 0 / 0 s t a t e RELATED, ESTABLISHED 67 ACCEPT udp −− 0 . 0 . 0 . 0 / 0 0 . 0 . 0 . 0 / 0 s t a t e RELATED, ESTABLISHED 68 ACCEPT a l l −− 0.0.0.0/0 0.0.0.0/0 Sample output of iptables-save command:

1 # Generated by iptables −save v1.4.4 on Thu May 3 15:52:39 2012 2 ∗mangle 3 :PREROUTING ACCEPT [ 0 : 0 ] 4 : INPUT ACCEPT [ 0 : 0 ] 5 :FORWARD ACCEPT [ 0 : 0 ] 6 :OUTPUT ACCEPT [ 0 : 0 ] 7 :POSTROUTING ACCEPT [ 0 : 0 ] 8 COMMIT 9 # Completed on Thu May 3 15:52:39 2012 10 # Generated by iptables −save v1.4.4 on Thu May 3 15:52:39 2012 11 ∗ f i l t e r 12 : INPUT DROP [ 0 : 0 ] 13 :FORWARD DROP [ 0 : 0 ] 14 :OUTPUT DROP [ 0 : 0 ] 15 :INBOUND − [ 0 : 0 ] 16 : LOG_FILTER − [ 0 : 0 ] 17 :LSI − [ 0 : 0 ] 18 : LSO − [ 0 : 0 ] 19 :OUTBOUND − [ 0 : 0 ] 20 −A INPUT −s 194.145.181.170/32 −p tcp −m tcp ! −−tcp−f l a g s FIN , SYN, RST ,ACK SYN −j ACCEPT ...

70 −A OUTBOUND −p udp −m s t a t e −−s t a t e RELATED, ESTABLISHED −j ACCEPT 71 −A OUTBOUND −j ACCEPT 72 COMMIT 73 # Completed on Thu May 3 15:52:39 2012 Sample output of iptables-save|iptables-xml command: (this is the previous command run and piped into iptables-xml application, which formats the data to an XML file)

1 < i p t a b l e s −r u l e s version=" 1 . 0 "> 2 < !−− # Generated by iptables ∗−save v1.4.4 on Thu May 3 15:52:48 2012 −−> 3

4 5 6 7 8 9 10 < !−− # Completed on Thu May 3 15:52:48 2012 −−> 11 < !−− # Generated by iptables ∗−save v1.4.4 on Thu May 3 15:52:48 2012 −−> 12

13 14 15 16 17 ~~194.145.181.170/32~~ 18

tcp

19 20 21 FIN,SYN,RST,ACK SYN 22

14 4. IMPLEMENTATION OF GNU/LINUXMONITORINGAPPLICATION

23 24 < a c t i o n s > 25 26 27

...

604 605 < a c t i o n s > 606 607 608 609 610 611 612 < !−− # Completed on Thu May 3 15:52:48 2012 −−> 613

The iptables -L -n command is probably the most famous way to display iptables configuration3. It looks user-friendly when viewing the output. Translating such a text to a machine configuration file is quite a hard task, however. A skilled system administrator should be able to know its meaning and to translate it to a series of commands configuring iptables. Still, it brings unnecessary complexity in scenarios like reverting to a previous configuration using old iptables -L -n data. One can see that iptables-save displays the configuration in such a way that by prepending “iptables” to each line beginning with a dash and executing them in order on a system with clean iptables, the result would be the very same iptables configuration as on the system which exported the configuration. This alleviates the problem mentioned in the last paragraph. Since the requirements on the monitoring application are such that the output reports must be in XML format, it makes some questions relevant: Should the resulting reports be fully in XML? And if they should, should a new iptables-to-xml con- verter be made? Had the answer to the first question been “Yes, they should,” the iptables-save|iptables-xml command would have come handy as a “No, there already is a solution,” reply to the second question. It would seem straightforward to just use this command and embed the output to the generated report file. Nonetheless, there would still be the problem mentioned for the iptables -L -n command: It is not trivial to convert the XML data back to iptables configuration. Hence, the iptables-save output was chosen as the source of iptables configuration information. The resulting report contains elements “” numbered in an attribute “number”, each element containing one line from iptables-save. Changes in iptables are reported in a way that seems counter-intuitive, but it is really simple. Let me explain using an example. There was a transition from an old configuration to a new configuration:

3. Guess based on personal experience and based on Google results on queries like “display iptables configuration”: www.google.com/search?q=display+iptables+configuration

15 4. IMPLEMENTATION OF GNU/LINUXMONITORINGAPPLICATION

Old configuration New configuration Line number Line contents Line number Line contents 1 A 1 A 2 B 2 X 3 C 3 C 4 D 4 Y 5 E 5 Z 6 E 7 F In iptables, the line order matters. We can see that B in the old configuration was replaced by X in the new one and D in the old one was replaced by Y and Z in the new one. F was added to the end. So far, so good and intuitive. How to describe this transition using an algorithm? This issue was discussed with Mgr. Pavel Tuˇcekand the resulting algorithm follows, described using the example data for the sake of simplicity and being illustrative. The algorithm steps through the old configuration, line by line, starting at the first line. It tries to search for the exactly same line in the new configuration and marks the lines it cannot find as deleted. Old configuration Deleted Line number Line contents Line number Line contents 1 A 2 B 2 B 4 D 3 C 4 D 5 E

Then, it walks through the new configuration the same way, making notes about added lines. New configuration Added Line number Line contents Line number Line contents 1 A 2 X 2 X 4 Y 3 C 5 Z 4 Y 7 F 5 Z 6 E 7 F The resulting difference report contains the following information:

16 4. IMPLEMENTATION OF GNU/LINUXMONITORINGAPPLICATION

Deleted Added Line number Line contents Line number Line contents 2 B 2 X 4 D 4 Y 5 Z 7 F The counterintuitive part is line numbering. The list of deleted lines is numbered using the old configuration’s line numbering and the list of added lines is numbered using the new configuration’s line numbering. We can see that F was added toline7. The old configuration had only 5 lines. In order to be able to tell where Freallywas added without having the new configuration, we need to go through all deletions and all additions in the correct order while changing the old report accordingly. This way of storing data about changes is inconvenient for human beings but convenient for simple algorithms. When a change in iptables occurs, the GNU/Linux event generator produces such a differential report. Lemon Pool receives the report and runs a reverse algorithm using the last full iptables data from Lemon database, thus producing full current iptables data without actually receiving the full data. The event generator sends full iptables data only once a day, when a full daily report is produced.

Network interfaces

Output of the ifconfig command is used to gather information about network interfaces. The output is structured into items, each describing one network interface and each containing subitems with information about link encapsulation, MAC address, IPv4 address and/or IPv6 address (if present). Changes in the network interface configuration are reported using only the information that changed, thus not reporting the information that did not change.

Users

Information about users is obtained by parsing the /etc/passwd file. The output is structured into items, each describing one user and each containing subitems with information about name, UID, GID, user’s additional information, path to home directory, default shell and member groups. The groups the user is member of are obtained using the groups command. Any change is reported using only the changed information, thus not reporting the information that did not change, except for the list of groups. If any relevant membership changes, a full new list for that particular user is reported.

17 4. IMPLEMENTATION OF GNU/LINUXMONITORINGAPPLICATION

Groups Information about groups is obtained by parsing the /etc/groups file. The output is structured into items, each describing one group and each containing subitems with information about group name, GID and list of users that are members of the group. A change in membership is reported using a full new list of members for the changed group. A change in name is reported using only the new name.

Recent logins (both physical and remote) Output of the last command is used to gather information about recent logins. The output is structured into items, each describing one login and each containing subitems with information about username, terminal, display, time of log in and time of log out. Any change is reported only using the information that changed. Most of the time, the information change only when users log in and log off and when the system logs are rotated (on which the last command depends). If something or someone, an attacker for instance, deliberately changes the logfile with the logins, this change will be detected and reported.

Recent physical logins Some systems are meant for remote access only. While the GNU/Linux event generator is not capable of detecting intrusions in server rooms—as this is out of its compe- tence4—information about a physical log in may be an important clue for maintaining security. Therefore a separate report exists, monitoring only physical logins. The means of gathering information and the way of producing the report are the same as for the “recent logins (both physical and remote)” report.

Recent login attempts over ssh (both successful and unsuccessful) Information about incoming login attempts over ssh is obtained by parsing sshd logs from the /var/log/auth.log or authlog5 file. The output is structured into items, each describing one login attempt and each containing subitems with information about the date of the attempt, the time, local hostname, sshd instance PID, action (describing what happened, like “Accepted password” or anything else), username of the login attempt, remote address, remote port and protocol version. Any change is reported only using the information that changed. Most of the time, the information change only when users attempt to log in and when the system logs are rotated. If something or someone, an attacker for instance, deliberately changes the logfile with sshd logs, this change will be detected and reported.

4. The GNU/Linux event generator could detect intrusions on the condition that the motion sensors can be accessed using a computer and a specialized module is written for the event generator. It’s better to use an ordinary anti-theft alarm than this convoluted contraption. 5. Some systems omit the dot in the file name. 18 4. IMPLEMENTATION OF GNU/LINUXMONITORINGAPPLICATION

Installed packages

The GNU/Linux environment is quite heterogeneous in its use of package management systems and package formats. To make the GNU/Linux event generator prepared for a set of package management systems would mean inevitable patching and recompilation in the future, if compatibility with a not-included package management system is needed. A simple, future-proof and configurable solution was chosen instead. The event generator can be configured via a settings file to launch a command that printsall installed packages in the system. The system administrator who deploys the event generator knows6 such a command, so this poses no trouble and enables the event generator to be portable even without the need of recompilation. The event generator reads the command from the settings file, saves it to a7 file , executes the file with bash8, gets the output and treats each line as name of one installed package. The resulting report is structured into items, each being name of one installed package. Any change is reported using only the information that changed whereas a package can be added to or removed from the list.

Available package updates

This report is made exactly the same way as the previous one, “installed packages”, with the same reasoning of the way it’s done.

Recent reboots

Output of the last command is used to gather information about recent reboots, shut- downs and runlevel switches. The output is structured into items, each describing one reboot, shutdown or a runlevel switch and each containing subitems with information about the action, kernel version, start time and end time (this can be used later to determine how long was the system running), that is all the information last provides. Any change is reported only using the information that changed. Most of the time, the information change only when the runlevel changes and when the system shuts down or reboots and when the system logs are rotated (on which the last command depends). If the logfile is deliberately manipulated, this change will be detected and reported.

6. Being a system administrator should imply that knowledge. 7. So that the execution of the command goes with as few problems as possible and to avoid the necessary character escaping when passing the command directly to bash. 8. Bash is used for the execution of the command so that familiar constructs (e.g. a pipe |) or short scripts can be used.

19 4. IMPLEMENTATION OF GNU/LINUXMONITORINGAPPLICATION

Time of last reboot It may be useful to know just the time of the last reboot. A special report containing only that information is produced so that Lemon can obtain it directly, without parsing the “list of recent reboots” report. The first line of output of the who -b command is used to get the time and date of the last system boot. The output is structured into two items—date and time, in the textual format provided by who -b. Any change is reported as a new full (albeit rather small) report containing the new date and time.

Processes using CPU above a set limit Output of the ps utility is used to gather information about processes. The information is filtered based on configurable settings, which includes the option to report onlythe processes running longer than a set time in seconds and to report only the processes that have spent more than a set CPU processing time in seconds. The output is structured into items, each describing one process and containing subitems with information about process UID, process PID, elapsed running time, used CPU time and the command used to run the process. Any change is reported only using the information that changed, thus not reporting the information that did not change.

Information about operating system Output of the uname utility is used to gather information about operating system. The output is structured into items with information about kernel name, node name, kernel release, kernel version, machine, processor, hardware platform and the operating system name, that is all the information uname provides9.

System installation date There is no standardized way of determining the date and time of operating system installation10. It is possible to obtain the date and time of kernel compilation using uname -v but this generally has nothing to do with the actual system installation date11. To refrain from dependence on particular GNU/Linux distributions, a solution similar to the “installed packages” report was found after a debate with Mgr. Pavel Tuˇcek.

9. More information can be found in the uname man page [30]. 10. It is hard to find a credible source like a book or an article for this information because thereis none. It is possible to try to search for a solution, such as http://www.google.com/search?q=find+ installation+date+of+linux+system, and to find lots of people asking the question and getting various little hacks and distribution-specific answers, like reading file attributes that are supposed tonotbe changed since the installation or reading installer log files that themselves vary wildly among GNU/Linux distributions. 11. One can install a system compiled ages ago and one can update an old installation with a fresh kernel.

20 4. IMPLEMENTATION OF GNU/LINUXMONITORINGAPPLICATION

The event generator reads a configurable command from the settings file, executes it and uses the output text as the information for the “system installation date” report, without any other processing. This way, the system administrator can specify a command that returns the wanted information on the deployed GNU/Linux distribution. Since the GNU/Linux event generator has no control over the output of the command, it does not try to format or process the date; it simply saves the command output to the report and it is up to Lemon Pool to process the output correctly.

4.2 Application architecture

4.2.1 Classes and interfaces LinMon consists of several classes covering all aspects of the functionality, one class that instantiates them and a main class that acts as an entry point. The classes covering the functionality follow a bridge pattern[31, page 36]—there is an interface and an implementation for each aspect of LinMon functionality.

Part of the class diagram showing the core of LinMon.

There is an interface for “data gathering modules”. Data gathering module is a class containing the logic for producing reports of one area. Each area described in 4.1.4 on page 13 is monitored using one data gathering module class. Each such class has several methods that describe the name of the module, name of the produced report, names

21 4. IMPLEMENTATION OF GNU/LINUXMONITORINGAPPLICATION

of options that the module wants to add to the settings file, bash command producing text data and a method that converts the text data to XML data that are passed back to the application for further processing and report production. The module class does not contain logic accessing the settings file. Logic for executing system commands is not required, too. LinMon does all the work for its modules so that the modules are lightweight, easy to make and reliable.

«interface» IDataGatheringModule

+ GetReportName() : String + GetXmlDataFromTextData(textData: String) : XmlDocument + GetBashCommand() : String + GetModuleID() : String + GetUsedSettingsOptions() : String[] + SetSettingsOption(optionName: String, optionValue: String) : Void

DataGatheringModuleDiskUsage DataGatheringModuleAvailablePackageUpdates DataGatheringModuleSshd

DataGatheringModuleGroups DataGatheringModuleReboots DataGatheringModuleSystemInstallationDate

DataGatheringModuleInstalledPackages DataGatheringModuleLastLogins DataGatheringModuleUname

DataGatheringModuleSystemLoad DataGatheringModuleIfconfig DataGatheringModuleLastReboot

DataGatheringModuleUsers DataGatheringModuleIPTables DataGatheringModulePhysLogins

Part of the class diagram showing data gathering modules.

The rest of the class diagram can be seen in Appendix B. More about making new data gathering modules is in Appendix C.

4.2.2 Error handling LinMon is able to encounter and process two types of errors—fatal and nonfatal. A fatal error is one that makes LinMon unable to continue. For instance, denying LinMon write privilege to report output directory results in a situation where it’s not possible to run anymore as it doesn’t make sense (why run LinMon if it cannot produce reports?). A nonfatal error is one that makes it impossible to complete one isolated task but does not interfere with vital LinMon’s functions. For instance, one data gathering module failing means that one report cannot be produced but it does not interfere with other data gathering modules nor LinMon itself. Since LinMon must run as long as possible, the ability to recover from errors is important. In case of either error, LinMon tries to inform the system administrator about possible problem. If a fatal error occurs, LinMon tries12 to create an XML error report (in the same

12. There might be scenarios where saving the error report is not possible, a read-only file system for instance.

22 4. IMPLEMENTATION OF GNU/LINUXMONITORINGAPPLICATION

format as normal XML reports) and a text error log with the same information, both containing the reason why LinMon crashed. Fatal error reports and logs are always saved into /etc/linmon directory; alternative settings file is ignored in this case. If a nonfatal error occurs, which can happen when a data gathering module fails, the intended—and failed—daily or differential report is replaced by an error report describing the problem. LinMon then continues as normal. LinMon’s error handling makes it possible to run for prolonged periods and to locate possible problems.

4.2.3 File access and permissions

LinMon must be run as root so that the data gathering modules can read anything they need. This requirement can be resolved using suid binaries for the modules that need extra permissions but it adds complexity and the Institute of Computer Science agrees with running LinMon under root. LinMon uses /etc/linmon directory to store its configuration, state, temporary files and produced reports. If read-write access to the directory cannot be obtained, LinMon produces a fatal error and quits. It is possible to specify path to alternative settings file which can in turn specify alternative location where to save produced reports.

4.2.4 Settings file and alternative settings

In 4.1.2 on page 12 was mentioned that there must be two possible sources of configuration so that LinMon can be configured both as a standalone application and asan application with remotely controlled settings. These two settings files are called “standard settings” and “alternative settings” in LinMon. Standard settings file is located in /etc/linmon/settings.xml. If it does not exist, LinMon creates a new settings file with default values. This file can be used as atemplate for alternative settings file. Alternative settings file can be specified in the standard settings file. LinMon detects the presence of alternative settings file at start andusesitas default if it is available, otherwise reverting to standard settings file for the rest of run. Alternative settings file allows to set any option that is in standard settings fileand the alternative value is used with highest priority. If the alternative settings file does not specify an option but the standard file does, the standard one is used. If neither ofthem specifies the option, an internal default is used. The alternative settings file also makes it possible to specify whether to revert back to the standard settings file if the alternative one becomes unavailable and whether to revert back again to the alternative one if it becomes available again. These two options are described in Appendix C, “Configuration” chapter and are useful if the alternative settings file is located on a network share that becomes unavailable. It may beusefulto revert to standard settings and local output folder for reports as a backup to still monitor the state of the system.

23 4. IMPLEMENTATION OF GNU/LINUXMONITORINGAPPLICATION

4.2.5 Report production

All data gathering modules produce data in XML format containing full information—this is all the modules do, the rest is work of LinMon itself. LinMon calls a module whenever a full or a differential report is produced. If a full report is produced, the data from the module are directly used to produce a full report13. The data are also cached for future reference. If a differential report is produced, the last cached data are compared with the new produced data. The new data are then cached for future reference and the comparison result is used to produce a differential report. Report comparison is described in the following subsection. Monitored areas are described in 4.1.4 on page 13 (the goal was met and everything was implemented).

4.2.6 Report comparison

To make reliable and efficient report comparison possible, the XML data coming from data gathering modules have a defined structure. Each module must conform or an error is produced. The structure is described in Appendix C, “Making a new data gathering module” chapter in a way that is easy to follow. The XML data from data gathering modules should look like this:

The document root name should be the same as the report name. It can contain one or more elements—”items”. An item can contain either text or elements—”subitems”. Subitems can contain whatever. There are two ways of comparing data in LinMon—with and without use of a primary key.

13. This involves only adding headers to the data.

24 4. IMPLEMENTATION OF GNU/LINUXMONITORINGAPPLICATION

Comparison without a primary key The items—that is the child elements of root—of the old and new data are compared against each other using InnerXml14. The old data are walked through and items that cannot be found in the new data are detected as deleted. Then the new data are walked through and items that cannot be found in the old data are detected as added. The slight- est change, even in only one subitem in many, results in the entire item being detected as added or deleted. Changed items are detected both as deleted (from the old data) and added (to the new data). This type of comparison is suitable for data where determining a primary key makes little sense—such as reports “Installed packages”, “Available package updates”, “Information about operating system”, or “System installation date”.

Comparison with a primary key Contrary to the previous examples, there are reports where it makes sense to detect changes in subitems. It is necessary to define a primary key in order to tell apart situation where a single item changed from a situation where an item was deleted and a new similar one was created15. When a data gathering module wants the reports to be compared using a primary key, it sets a “primaryKey” attribute to the document root. Its value is equal to the name of subitem that is treated as a primary key—this subitem tells the items apart. Examples: For the “Users” report, the primary key is “UID”; for the “Groups” report, the primary key is “GID”. Comparison of data with a primary key is quite similar to the simpler comparison already mentioned, with a few dissimilarities. The items are compared based on the primary-key-subitem. The old data are walked through and items that cannot be found in the new data are detected as deleted. The new data are walked through and items that cannot be found in the old data are detected as added. When the item can be found in both old and new data, its subitems are walked through and compared16. Any subitems that differ are reported as changed, together with the primary-key-subitem. The subitems that did not change are not mentioned in the resulting report. So if, for example, the default shell for a user changed, only the “shell” (changed subitem) and “UID” (primary key) would be mentioned for that particular user in the resulting differential report. This comparison type requires that each item has always the same number of subitems in the same order.

14. Appendix A, ReportDiffs.cs, page 29, lines 136 and 155. 15. This question is tackled in a comment in the Appendix A, ReportDiffs.cs, page 28, lines 114-123. 16. Again, using InnerXml. Appendix A, ReportDiffs.cs, page 31, lines 276, 277

25 5 Testing of LinMon

Integral part of software development is testing. LinMon was tested in several operating systems on several computers. The most extensive testing was performed on a server belonging to the Institute of Computer Science.

5.1 Testing environment

5.1.1 Personal computers LinMon was tested under GNU/Linux, GNU/kFreeBSD, OpenBSD, FreeBSD and Win- dows 7 operating systems.

GNU/Linux There were 5 distributions involved in testing:

• Ubuntu 12.04

• Debian 6

• Fedora 16

• CentOS 6

• Mandriva 2011

Machine running Ubuntu 12.04 was used for development, the others just for testing. The only software needed for running LinMon was Mono runtime. CentOS 6 uses long mountpoint names which resulted in broken lines in df -h. This would be a problem in any distribution if such long mountpoint names were used. This quickly helped to fix the problem by using df -h -P which automatically extends the columns. Mandriva had an unidentified bug in Mono which saved garbage to the variable ina code like this: t r y { variable = thisMethodThrowsAnException (); } catch {} The solution (apart from using Mono runtime without the bug) was to reinitialize the variable in the catch statement. Except those two problems, all distributions behaved similarly and LinMon ran equally well.

26 5. TESTINGOF LINMON

Other than GNU/Linux LinMon was tested under other systems to see how it behaves under nonstandard conditions but not necessarily to find and fix platform-specific dependencies in thedata gathering modules. It was found that LinMon is able to run and produce (mostly error) reports on any system that has Mono or .Net framework installed. The data gathering modules produced errors when the environment was sufficiently different from GNU/Linux. Under OpenBSD and FreeBSD, LinMon was able produce valid reports about users, groups, ssh login attempts and partly about system information. Had the proper commands been specified in settings, installed packages, available updates and system installation date would have had worked, too. Under Windows, all reports failed (because of missing bash, among other differences) but LinMon correctly generated error reports and everything except the data gathering modules worked. It created directory in \etc\linmon of the drive it was run on. It may be easy to use LinMon on Windows with Windows-specific data gathering modules and a replacement for bash (cmd.exe, for instance). The test proved that LinMon is well-equipped to withstand lots of errors in data gathering modules and run without a crash. Debian GNU/kFreeBSD was found to work the same as GNU/Linux for LinMon. This may be due to the presence of GNU utilities instead of the BSD utilities.

5.1.2 ICS server The Institute of Computer Science set up a testing server for LinMon with the same operating system and environment as other ICS servers, so that LinMon could be tested in as real conditions as possible. LinMon underwent an extensive testing during which over 3000 reports over 3 weeks were generated.

Problems that occurred during testing Two problems were detected during testing on the ICS server. The first problem caused full daily reports to be generated twice, which wasrectified by designing a more robust timer for daily reports. The second problem was a regression in a data gathering module. The ps -www command should, according to its manual page, make each column as wide as needed. This makes it easy to process the output by a program. However, when UID is longer than 5 digits, ps -www fails to make the UID column wide enough, which results in non- aligned data. The fix was to use the ps command with explicit column widths: ps -eo uid:20,pid:20,etime:20,cputime:20,command:500. It doesn’t seem as elegant as ps -www -eo uid,pid,etime,cputime,command which does not explicitly set column widths but it works better in the end.

27 5. TESTINGOF LINMON

Apart from those two problems, there was no other problem found in LinMon. There was no design change necessary based on the testing experience.

5.2 LinMon in production environment

To make easy use and data gathering module development possible, an Administrator’s Guide was created, which can be found in Appendix C.

28 6 Conclusion

The goal of the thesis was to devise a monitoring application for the GNU/Linux operating system operating under narrow specifications determined by the Lemon monitoring system. The theoretical part of thesis describes methods of operating system monitoring, suitability of such methods in relation to the requirements placed by existing infrastructure and the design of a new monitoring application. The practical part of the thesis encompasses the development of the application and creation of the Appendices. The goals were met and the resulting application is planned to be deployed in the foreseeable future. It will allow Lemon to finally monitor GNU/Linux machines in addi- tion to currently monitored Windows machines. Moreover, the application architecture makes Lemon event generator development for other platforms easy, should such a necessity arise in the future.

29 Bibliography

[1] B. Schneier, Proving a Computer Program’s Correctness, Std., Feb. 2012. [Online]. Available: http://www.schneier.com/blog/archives/2009/10/proving_a_compu. html

[2] MIT Exokernel Operating System, Massachusetts Institute of Technology Std., Feb. 2012. [Online]. Available: http://pdos.csail.mit.edu/exo/

[3] QNX Neutrino Realtime Operating System, QNX Software Systems Std., Feb. 2012. [Online]. Available: www.qnx.com/download/download/8117/qnx_neutrino_ brief_0212.pdf

[4] The KeyKOS System, Department of Computer and Information Science Std., Feb. 2012. [Online]. Available: http://www.cis.upenn.edu/~KeyKOS/

[5] J. Shapiro, EROS: A Novel Combination, Std., Feb. 2012. [Online]. Available: http://www.eros-os.org/project/novelty.html

[6] S. Godard, Features, Std., Mar. 2012. [Online]. Available: http://sebastien.godard. pagesperso-orange.fr/features.html

[7] D. Wieërs, Dstat: Versatile resource statistics tool, Std., Mar. 2012. [Online]. Available: http://dag.wieers.com/home-made/dstat/

[8] H. Ware, vmstat(8) - Linux man page, Std., Mar. 2012. [Online]. Available: http://linux.die.net/man/8/vmstat

[9] F. Forster, Features, Std., Mar. 2012. [Online]. Available: http://collectd.org/features. shtml

[10] ——, Networking introduction, Std., Mar. 2012. [Online]. Available: http: //collectd.org/wiki/index.php/Networking_introduction

[11] Munin - Trac, Std., Mar. 2012. [Online]. Available: http://munin-monitoring.org/

[12] Munin Exchange moved to GitHub, Std., May 2012. [Online]. Available: http: //munin-monitoring.org/wiki/MuninExchangeIsDown

[13] munin-monitoring / contrib, Std., May 2012. [Online]. Available: https://github.com/ munin-monitoring/contrib/

[14] HOWTO Monitor Windows, Std., May 2012. [Online]. Available: http:// munin-monitoring.org/wiki/HowToMonitorWindows

[15] About Nagios Core, Nagios Enterprises, LLC Std., Mar. 2012. [Online]. Available: http://nagios.sourceforge.net/docs/nagioscore/3/en/about.html

30 6. CONCLUSION

[16] Monitoring Windows Machines, Nagios Enterprises, LLC Std., Mar. 2012. [Online]. Available: http://nagios.sourceforge.net/docs/nagioscore/3/ en/monitoring-windows.html

[17] Monitoring Linux/Unix Machines, Nagios Enterprises, LLC Std., Mar. 2012. [Online]. Available: http://nagios.sourceforge.net/docs/nagioscore/3/en/monitoring-linux. html

[18] Nagios Plugins, Nagios Enterprises, LLC Std., Mar. 2012. [Online]. Available: http://nagios.sourceforge.net/docs/nagioscore/3/en/plugins.html

[19] Features, Project Shinken Std., Mar. 2012. [Online]. Available: http://www. shinken-monitoring.org/features/

[20] About Shinken, Project Shinken Std., Mar. 2012. [Online]. Available: http: //www.shinken-monitoring.org/wiki/official/about

[21] Icinga vs. Nagios – What’s the difference?, Icinga: Open Source Monitoring Std., Mar. 2012. [Online]. Available: https://www.icinga.org/nagios/

[22] Performance Co-Pilot :: Features, Silicon Graphics International Corp. Std., Mar. 2012. [Online]. Available: http://oss.sgi.com/projects/pcp/features.html

[23] H. Storner, Xymon Monitor, Std., Mar. 2012. [Online]. Available: http://xymon. sourceforge.net/

[24] A. Cíková,ˇ “Návrh a implementace mechanismu pˇredávání zpráv z klienta na server,” Master’s thesis, Masaryk University, 2012. [Online]. Available: http://is.muni.cz/th/175536/fi_m/

[25] P. Smˇelý, Lokální služba monitorující stav operaˇcních systém˚uMicrosoft Windows, XSD schema, Std., 2011. [Online]. Available: https://is.muni.cz/th/256139/fi_b/ 20750312/machineSettingsSchema.xsd

[26] M. Bosák, “Pokroˇcilé monitorování rozsáhlé doménové infrastruktury a bezpeˇcnostníchsystém ˚u,”Master’s thesis, Masaryk University, 2012. [Online]. Available: http://is.muni.cz/th/207567/fi_m

[27] P. Smˇelý, Lokální služba monitorující stav operaˇcníchsystém˚uMicrosoft Windows, Std., 2011. [Online]. Available: https://is.muni.cz/th/256139/fi_b/

[28] MONO, Std., Feb. 2012. [Online]. Available: http://www.mono-project.com/Main_ Page

[29] ECMA C# and Common Language Infrastructure Standards, Microsoft Std., May 2012. [Online]. Available: http://msdn.microsoft.com/en-us/netframework/aa569283

31 6. CONCLUSION

[30] uname(1) - Linux man page, Free Software Foundation, Inc. Std., May 2010. [Online]. Available: http://linux.die.net/man/1/uname

[31] J. Bishop, C# 3.0 design patterns. Farnham: O’Reilly, 2008.

[32] J. Sharp, Microsoft Visual C# 2008 step by step. Redmond, Wash: Microsoft Press, 2008.

[33] J. Skeet, C# in depth. Greenwich, Conn. London: Manning Pearson Education distributor, 2010.