DOCSLIB.ORG

Ntopng Development in Netbeans IDE

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Ntopng Development in Netbeans IDE ntopng development in NetBeans IDE Version 1.0 Aug 2015 Simone Mainardi, [email protected] Page !1 of !11 Index Preface%.................................................................................................................................2# Download and Installation%....................................................................................................2# Importing ntopng in NetBeans %............................................................................................3# Running ntopng in NetBeans%...............................................................................................4# Debugging ntopng in NetBeans%...........................................................................................7 Preface NetBeans is a powerful, widespread Integrated Development Environment (IDE), that supports multiple languages including C and C++. Features such as syntax highlighting, code completion, integrated versioning and debugging make this IDE a very e"ective tool.# This guide describes how to set up NetBeans to develop and debug ntopng on a Unix operating system such as Linux, and Mac OS X.# Download and Installation NetBeans NetBeans can be downloaded from the o$cial website https://netbeans.org/downloads/. There are many flavors ready to be downloaded. If available disk space is not a concern, you are encouraged to get the ‘All’ technologies bundle that has extra support for HTML. Otherwise, the ‘C/C++’ bundle may su$ce.# Installation is straightforward. Simply click on the download archive and follow the instructions.# Alternatively, you may install NetBeans via your distribution package manager — e.g., apt-get, rpm, brew — but changes are that the package will not contain the latest version.# ntopng ntopng source code is available at https://github.com/ntop/ntopng. You can clone the repository in a local copy using git. # Page !2 of !11 git clone https://github.com/ntop/ntopng.git Once downloaded, a Makefile for the project has to be generated. cd into ntopng folder and run# Simones-MBP:code simone$ cd ntopng/ Simones-MBP:ntopng simone$ ./autogen.sh && ./configure Upon successful completion of both autogen.sh and configure, you will find a Makefile file in the root of the project. This file contains all the necessary information to build ntopng in your system. NetBeans will use this file to compile and build the software.# Importing ntopng in NetBeans ntopng can be imported in NetBeans as C/C++ project with existing sources. Open NetBeans and select File -> New Project -> C/C++ Project with existing sources, then click Next.# You will be prompted to specify the folder containing the sources. Click ‘Browse’ and navigate to the ntopng folder. Leave default settings for ‘Build Host’, ’Tools Collection’ and ‘Configuration Mode’ and confirm.# NetBeans will start to compile the sources. Compilation process and status from are shown on a tabbed window at the bottom of NetBeans. Once completed, a ‘MAKE SUCCESSFUL’ message should be displayed.# Page !3 of !11 Now you are ready to develop ntopng. Navigate the project tree on the left side of the IDE. Open files, edit them, and compile the project by clicking on the top toolbar Hammer button. However, there is still some work to do in order to run ntopng. Simply clicking on the run Play button will not su$ce since ntopng requires administrative privileges to start. The following section will discuss how to grant ntopng the necessary privileges.# Running ntopng in NetBeans Grant ntopng administrative privileges ntopng requires administrative privileges to run, since it has to access network interfaces. From the command line, one can simply run ntopng with sudo and insert the root password when prompted. This is not possible in NetBeans.# Therefore, it is necessary to tell sudo to execute ntopng without prompting for passwords. Open a shell and type sudo visudo to configure super user preferences.# Simones-MBP:ntopng simone$ sudo visudo A text editor will open with a preference file. At the very bottom of the file add the following line# <your_user> ALL=NOPASSWD: <full_path_to_ntopng_executable> Then save and close. Replace <your_user> with the user that runs NetBeans and <full_path_to_ntopng_executable> with the absolute path of the ntopng executable. The executable is called ntopng and can be found in the root directory of the ntopng project folder.# In my case, I added the following line# simone ALL=NOPASSWD: /Users/simone/code/ntopng/ntopng Now ntopng can be executed with administrative privileges without passwords. To verify, open a new shell terminal, go to the ntopng project, and run sudo ntopng. It should start without asking for super user password.The next step consists in telling NetBeans to run ntopng using sudo.& Page !4 of !11 Run sudo ntopng in NetBeans Right click on the ntopng project, and select Set Configuration -> Customize. Then, select the category ‘Run’ from the tree on the left, and add prepend the word sudo to the Run command. On Mac OS X, run defaults to NetBeans variable ${OUTPUT_PATH}. This may not be true for other operating systems. Nevertheless, you should not worry about it. Prepending the word sudo to the default command may su$ce. One done, Click ‘Apply’, and confirm with ‘OK’.# At this point we are ready to run ntopng within NetBeans. Click on the green Play button on the top toolbar. A tabbed window ‘Output’ should pop at the bottom of the IDE.# Page !5 of !11 Hooray, ntopng has successfully started within the IDE. Try to fire up a browser and point it to localhost:3000. ntopng instance should serve your requests and print additional output to the window.# Stop ntopng instance running in NetBeans To stop ntopng, hitting the red square ‘stop’ button may not be enough. Recall that ntopng runs with administrative privileges and hence, NetBeans — that is run by an unprivileged user — may not have su$cient privileges to stop it.# Therefore, to make sure ntopng is successfully stopped, a CTRL-C command has to be injected into the ‘Output’ window. Click somewhere on the window to make it active and then, hit CTRL +C.# ‘Shutting down’ will be displayed and ntopng will start to gracefully shut down. Eventually, a RUN FINISHED message will pop to confirm the software has been terminated.# Page !6 of !11 Debugging ntopng in NetBeans In order to debug ntopng, the GNU gdb debugger has to be installed on the system. Debugger installation falls outside the scope of this guide. For the installation please have a look at tutorials specific for your operating system. The most straightforward way to install gdb is to install it via your package manager, e.g., apt-get install gdb. Once installed, the debugger will be available as an executable in the system. Type `which gdb` in a terminal to see the full path of the executable. # Simones-MBP:tmp simone$ which gdb A path should be output. Typical installation paths are# /opt/local/bin/gdb /bin /usr/bin If you can’t find gdb, try with `which ggdb`. Some package managers such as mac ports install it under the name of ggdb. In my case, I have gdb installed with the name ggdb in /opt/local/bin.# Simones-MBP:tmp simone$ which ggdb /opt/local/bin/ggdb Simones-MBP:tmp simone$ /opt/local/bin/ggdb GNU gdb (GDB) 7.9.1 Copyright (C) 2015 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/ gpl.html> […] (gdb) … Note for Mac OS X users: If you are on Mac OS X, you have to certify the debugger in order for it to take control of other applications. Please refer to this guide: http://ntraft.com/installing-gdb-on- os-x-mavericks/.# Once the full path to the debugger executable is known, a one-line wrapper script for the debugger has to be created. ntopng requires administrative privileges and the debugger must have such privileges to take control of it. Go to the ntopng project folder, cd into tools, and touch a file called gdbsudo# Simones-MBP:code simone$ cd ntopng Simones-MBP:ntopng simone$ cd tools/ Simones-MBP:tools simone$ touch gdbsudo Now edit the file gdbsudo and add the following contents:# #!/bin/bash echo "I'm going to call sudo ggdb with options " $* sudo /opt/local/bin/ggdb $* Page !7 of !11 Make sure to change the full path to the debugger so that it matches the path output by the `which gdb` command as discussed above. This one-line wrapper simply calls the debugger with sudo, and forwards any argument to it.# gdbsudo would prompt for the administrator password when executed. Since we want to call it from NetBeans, we have to tell sudo not to ask for passwords when running the debugger.# In a shell, type sudo visudo (see section ‘Grant ntopng administrative privileges’ above) and, at the very bottom of the file, add# <your_user> ALL=NOPASSWD: <full_path_to_the_debugger> Again, make sure to change <full_path_to_the_debugger> so that it matches the path output by the `which gdb` command as discussed above. Also substitute <your_user> with the user name NetBeans is run with. In my case, I added the following line# simone ALL=NOPASSWD: /opt/local/bin/ggdb Now we are ready to tell NetBeans to use the one-line wrapper as debugger for ntopng. Open NetBeans Preferences, tab C/C++ and specify the absolute path to the one-line wrapper in the ‘Debugger Command’.# Page !8 of !11 Make sure you are editing the right Tool Collection on the left. That is, the tool collection used to build and debug ntopng.# In my case, I created a new collection ‘GNU_Mac’ to work with ntopng. This collection has base directory /opt/local/bin. With the default /usr/bin/ NetBeans was complaining about not finding tools (e.g., wget, pkg-config). I changed to /opt/local/bin since mac ports install tools and utilities there. You don’t have to worry about changing C and C++ compliers. ntopng has its own Makefile that is able to automatically detect them.# Click ‘Apply’ and then ‘OK’.# Now right click on ntopng project, ‘Set Configuration’ -> ‘Customize…’, and select the ‘Debug’ Category on the left.
Load more

Recommended publications
  • Monitoring Network Traffic Using Ntopng
    Monitoring Network Traffic using ntopng Luca Deri <[email protected]> © 2013 - ntop.org Outlook • What are the main activities of ntop.org ? • ntop’s view on network monitoring. • From ntop to ntopng. • ntopng architecture and design. • Using ntopng. • Advanced monitoring with ntopng. • Future roadmap items. "2 © 2013 - ntop.org About ntop.org [1/2] • Private company devoted to development of open source network traffic monitoring applications. • ntop (circa 1998) is the first app we released and it is a web-based network monitoring application. • Today our products range from traffic monitoring, high-speed packet processing, deep-packet inspection, and IDS/IPS acceleration. "3 © 2013 - ntop.org About ntop.org [2/2] • Our software is powering many commercial products... "4 © 2013 - ntop.org ntop Goals • Provide better, yet price effective, traffic monitoring solution by enabling users to have increased traffic visibility. • Go beyond standard metrics and increase traffic visibility by analysing key protocols in detail. • Provide users comprehensive and accurate traffic reports able to offer at a fraction of price what many commercial products do together. • Promote open-source software, while protecting selected IPRs. "5 © 2013 - ntop.org ntop’s Approach to Traffic Monitoring • Ability to capture, process and (optionally) transmit traffic at line rate, any packet size. • Leverage on modern multi-core/NUMA architectures in order to promote scalability. • Use commodity hardware for producing affordable, long-living (no vendor lock), scalable (use new hardware by the time it is becoming available) monitoring solutions. • Use open-source to spread the software, and let the community test it on unchartered places. "6 © 2013 - ntop.org Some History • In 1998, the original ntop has been created.
    [Show full text]
  • Ntopng User's Guide
    ! !!" ! ! ! ! ! ! ! ! ! ! ! ! ntopng User’s Guide" High-Speed Web-based Traffic Analysis and Flow Collection " ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! Version 1.2" August 2014!" ! ! © 1998-14 - ntop.org" ntopng User’s Guide v.1.2 1.Table of Contents" ! 1. What’s New?"......................................................................................3" 2.It’s time for a completely new ntop."............................................................5" 3. Introduction"..............................................................................................6" 3.1. The main design principles"................................................................7" 3.2. What ntopng can do for me?"..............................................................7" 3.3. ntopng Architecture"..........................................................................9" 3.4. Download ntopng"............................................................................10" 4.Using ntopng"...........................................................................................11" 4.1. Compiling ntopng Source Code".......................................................11" 4.2. Installing a Binary ntopng"................................................................11" 4.3. ntopng Command Line Options".......................................................11" 4.4. ntopng on Windows".........................................................................16" 5. API Scripting Lua".....................................................................................18"
    [Show full text]
  • Rethinking Hyper- Converged Infrastructure for Edge Computing
    OpenInfra Days Shanghai 2019 Rethinking Hyper- Converged Infrastructure for Edge Computing Feng Li (李枫) [email protected] Nov 5，2019 Agenda I. Edge Computing Overview Outstanding Platforms Status, Trend, and HCI Summary II. Overall Design Design Goals & Principles Hardware Platform Why is eBPF eBPF Development III. Testbed Development Boards Software Platform Development Model IV. Computing, Networking, Storage HPC Networking ntopng DRedis Rethink Lightweight Storage Solutions V. Containerization, Services, and DevOps Kata Container Lightweight Kubernetes Cilium In-Kernel Services VI. Distributed Framework Ray My Practice VII. Messaging & RPC gRPC & Protobuf Rethink In-Kernel Messaging VIII. Data Processing Apache Arrow Lightweight Solution IX. Artificial Intelligence Trends TVM ARM NN My Practice X. Monitoring, Tuning, and Debugging Extending LISA Extending ntopng My Practice XI. Security Reference Design Hardware-Software Co-designed System Security XII. Miscs New Python Runtime FPGA XIII.eBPF-centric New Design Software Architecture XIV.Wrap-up I. Edge Computing 1) Overview https://en.wikipedia.org/wiki/Edge_computing a distributed computing paradigm which brings computation and data storage closer to the location where it is needed, to improve response times and save bandwidth.… https://www.networkworld.com/article/3224893/what-is-edge- computing-and-how-it-s-changing-the-network.html a way to streamline the flow of traffic from IoT devices and provide real-time local data analysis Intelligent
    [Show full text]
  • Network Traffic and Security Monitoring Using Ntopng and Influxdb
    Network Traffic and Security Monitoring Using ntopng and InfluxDB Luca Deri <[email protected]> @lucaderi © 2018 - ntop.org 1 Part I: Welcome to ntopng © 2018 - ntop.org 2 About Me • (1997) Founder of the ntop.org project with the purpose of creating a simple, and open source web-based traffic monitoring application. • Lecturer at the University of Pisa, Italy. • Author of various open source projects ◦n2n: peer-to-peer layer 2 VPN. ◦nDPI: deep-packet-inspection library. ◦PF_RING: high-speed packet capture and transmission. © 2018 - ntop.org 3 About ntop.org • ntop develops open source network traffic monitoring applications. • ntop (circa 1998) is the first app we released and it is a web-based network monitoring application. • Today our products range from traffic monitoring, high-speed packet processing, deep-packet inspection, and IDS/IPS acceleration (bro and suricata). © 2018 - ntop.org 4 ntop’s Approach to Traffic Monitoring • Ability to capture, process and (optionally) transmit traffic at line rate, any packet size. • Leverage on modern multi-core/NUMA architectures in order to promote scalability. • Use commodity hardware for producing affordable, long-living (no vendor lock), scalable (use new hardware by the time it is becoming available) monitoring solutions. • Use open-source to spread the software, and let the community test it on unchartered places. © 2018 - ntop.org 5 Motivation For Traffic Monitoring If you can’t measure it, you can’t improve it (Lord Kelvin, 1824 – 1907) If you can’t measure it, you can’t manage it (Peter Drucker, 1909 – 2005) © 2018 - ntop.org 6 What Happens in Our Network? • Do we have control over our network? • It’s not possible to imagine a healthy network without a clear understanding of traffic flowing on our network.
    [Show full text]
  • Network Security Toolkit Iso Free Download Network Security Toolkit (NST) 24 7977 Installation Step by Step
    network security toolkit iso free download Network Security Toolkit (NST) 24 7977 Installation Step by Step. Network Security Toolkit, aka NST, is a Linux distribution for security professionals and network admins, using MATE as the default desktop environment and as such it is lightweight and runs fast. The latest stable version NST 24 7977 was released on July 4th, 2016. It’s based on the recently released Fedora 24, with the best-of-breed open source network security applications added such as: WireShark WUI Scapy Multi-Traceroute (MTR) ntopng pcap netflow Netfilter Kismet …… These tool can work with Mercator, Google Maps, Google Earth and WebGL Globe to geolocate IP addresses which you can access by visiting the Web user interface (WUI). Very cool indeed. geolocation and graphic tool matrix. So this tutorial will show you how to install Network Security Toolkit 24. Let’s get started. Step 1: Download Network Security Toolkit 24 7977 ISO Image. The NST ISO image can be booted in live mode for those who like to have a try before installing to hard drive. Go to the official website, http://www.networksecuritytoolkit.org which looks so 2010 and a bit of confusing because the download link is not easy to find. Click the Release button on the upper left menu, then click Download . And you will be taken to SourceForge. Click the iso link to download. Note that NST only support 64 bit CPU. It’s a 2.8 GB file so now you can grab a cup of coffee or tea. Step 2: Create a Live DVD/USB.
    [Show full text]
  • Network Troubleshooting Using Ntopng Luca Deri <[email protected]>
    Network Troubleshooting Using ntopng Luca Deri <[email protected]> Outlook • Part 1: Introduction to ntopng ◦ntopng architecture and design. ◦ntopng as a flow collector. ◦Exploring system activities using ntopng. • Part 2: ntopng+Wireshark Monitoring Use Cases ◦Using ntopng. ◦ntopng and Wireshark. ◦Advanced monitoring with ntopng. ◦Future roadmap items. 2 SharkFest 2015 - Computer History Museum June 22-15, 2015 About ntop.org • ntop develops open source network traffic monitoring applications. • ntop (circa 1998) is the first app we released and it is a web-based network monitoring application. • Today our products range from traffic monitoring, to high-speed packet processing, deep-packet inspection, and IDS/IPS acceleration (snort, Bro and suricata). 3 SharkFest 2015 - Computer History Museum June 22-15, 2015 ntop’s Approach to Traffic Monitoring • Ability to capture, process and (optionally) transmit traffic at line rate, any packet size. • Leverage on modern multi-core/NUMA architectures in order to promote scalability. • Use commodity hardware for producing affordable, long-living (no vendor lock), scalable (use new hardware by the time it is becoming available) monitoring solutions. • Use open-source to spread the software, and let the community test it on unchartered places. 4 SharkFest 2015 - Computer History Museum June 22-15, 2015 Some History • In 1998, the original ntop has been created. • It was a C-based app embedding a web server able to capture traffic and analyse it. • Contrary to many tools available at that time, ntop used a web GUI to report traffic activities. • It is available for Unix and Windows under GPL. 5 SharkFest 2015 - Computer History Museum June 22-15, 2015 ntop Architecture Cisco NetFlow HTTP/HTTPS RRD InMon sFlow 6 SharkFest 2015 - Computer History Museum June 22-15, 2015 Why was ntop obsolete? • Its original LAN-oriented design prevented ntop from handling more than a few hundred Mbit.
    [Show full text]
  • Realtime High-Speed Network Traffic Monitoring Using Ntopng
    Realtime High-Speed Network Traffic Monitoring Using ntopng Luca Deri *†, Maurizio Martinelli*, Alfredo Cardigliano† IIT/CNR* ntop† Pisa, Italy {deri, cardigliano}@ntop.org, {luca.deri, maurizio.martinelli}@iit.cnr.it Abstract—Monitoring network traffic has become monitoring tools that are able to spot bottlenecks and security increasingly challenging in terms of number of hosts, protocol issues while providing accurate information for proliferation and probe placement topologies. Virtualised troubleshooting the cause. This means that while NetFlow/ environments and cloud services shifted the focus from dedicated sFlow can prove a quantitative analysis in terms of traffic hardware monitoring devices to virtual machine based, software traffic monitoring applications. volume and TCP/UDP ports being used, they are unable to This paper covers the design and implementation of ntopng, an report the cause of the problems. For instance, NetFlow/IPFIX open-source traffic monitoring application designed for high- can be used to monitor the bandwidth used by the HTTP speed networks. ntopng’s key features are large networks real- protocol but embedded NetFlow probes are unable to report time analytics and the ability to characterise application that specific URLs are affected by large service time. protocols and user traffic behaviour. ntopng was extensively Today a single application may be based on complex validated in various monitoring environments ranging from cloud-based services comprised of several processes small networks to .it ccTLD traffic analysis. distributed across a LAN. Until a few years ago web Keywords—Traffic monitoring, real-time analytics, open-source applications were constructed using a combination of web software, monitoring of virtual and cloud environments.
    [Show full text]
  • Realtime High-Speed Network Traffic Monitoring Using Ntopng
    Realtime High-Speed Network Traffic Monitoring Using ntopng Luca Deri, IIT/CNR and ntop; Maurizio Martinelli, IIT/CNR; Alfredo Cardigliano, ntop https://www.usenix.org/conference/lisa14/conference-program/presentation/deri-luca This paper is included in the Proceedings of the 28th Large Installation System Administration Conference (LISA14). November 9–14, 2014 • Seattle, WA ISBN 978-1-931971-17-1 Open access to the Proceedings of the 28th Large Installation System Administration Conference (LISA14) is sponsored by USENIX Realtime High-Speed Network Traffic Monitoring Using ntopng ! Luca Deri, IIT/CNR, ntop Maurizio Martinelli, IIT/CNR Alfredo Cardigliano, ntop ! ! ! Abstract facilitating the monitoring of virtual environments. Monitoring network traffic has become increasingly These are only partial solutions because either v5 Net- challenging in terms of number of hosts, protocol pro- Flow (or v9 with basic information elements only) or liferation and probe placement topologies. Virtualised inaccurate, sample-based sFlow are supported. Network environments and cloud services shifted the focus from managers need traffic monitoring tools that are able to dedicated hardware monitoring devices to virtual ma- spot bottlenecks and security issues while providing accurate information for troubleshooting the cause. This chine based, software traffic monitoring applications. This paper covers the design and implementation of means that while NetFlow/sFlow can prove a quantita- ntopng, an open-source traffic monitoring application tive analysis in terms of traffic volume and TCP/UDP designed for high-speed networks. ntopng’s key fea- ports being used, they are unable to report the cause of tures are large networks real-time analytics and the abil- the problems.
    [Show full text]
  • Installing NTOPNG-A Web-Based Network Traffic Analysis Tool
    handson Installing NTOPNG — A Web-Based Network Traffic Analysis Tool If you are looking to monitor different network protocols on your servers then you can go for the free tool that is both easy to install and use - Raj Kumar Maurya ou can use this tool to monitor various protocols, # Specifies the path where the PID (process ID) is saved. traffic variants, and bandwidth across multiple # time frames. It is based on libpcap and it has been -G=/var/tmp/ntopng.pid Ywritten in a portable way in order to virtually run # on every Unix platform, MacOSX and on Win32 as well. # -e|--daemon Here is the step by step installation guide for Ntopng # This parameter causes ntop to become a daemon, i.e. a for Ubuntu 14.04 server. task which runs in the background # without connection to a specific terminal. To use ntop Step 1. First of all add Ntopng repo in your Ubuntu repo other than as a casual monitoring list. Create ntop.list file by running the command: # tool, you probably will want to use this option. # sudo nano/etc/apt/sources.list.d/ntop.list -e= # Then add this line: # -i|--interface # Specifies the network interface or collector endpoint to deb http://www.nmon.net/apt-stable/12.04/ x64/ be used by ntopng for network deb http://www.nmon.net/apt-stable/12.04/ all/ # monitoring. On Unix you can specify both the interface name (e.g. lo) or the numeric Step 2: Run the given command to add the key and then # interface id as shown by ntopng -h.
    [Show full text]
  • Network Traffic and Security Monitoring Using Ntopng and Influxdb
    Network Traffic and Security Monitoring Using ntopng and InfluxDB Luca Deri <[email protected]> @lucaderi © 2018 - ntop.org 1 Part I: Welcome to ntopng © 2018 - ntop.org 2 About ntop.org [1/2] • ntop develops of open source network traffic monitoring applications. • ntop (circa 1998) is the first app we released and it is a web-based network monitoring application. • Today our products range from traffic monitoring, high-speed packet processing, deep-packet inspection, and IDS/IPS acceleration (bro and suricata). © 2018 - ntop.org 3 About ntop.org [2/2] • Our software is powering many commercial products... © 2018 - ntop.org 4 ntop Goals • Provide better, yet price effective, traffic monitoring solution by enabling users to have increased traffic visibility. • Go beyond standard metrics and increase traffic visibility by analysing key protocols in detail. • Promote open-source software, while protecting selected IPRs. • All commercial ntop tools are available at no cost for research and education. © 2018 - ntop.org 5 ntop’s Approach to Traffic Monitoring • Ability to capture, process and (optionally) transmit traffic at line rate, any packet size. • Leverage on modern multi-core/NUMA architectures in order to promote scalability. • Use commodity hardware for producing affordable, long-living (no vendor lock), scalable (use new hardware by the time it is becoming available) monitoring solutions. • Use open-source to spread the software, and let the community test it on unchartered places. © 2018 - ntop.org 6 Motivation For Traffic Monitoring If you can’t measure it, you can’t improve it (Lord Kelvin, 1824 – 1907) If you can’t measure it, you can’t manage it (Peter Drucker, 1909 – 2005) © 2018 - ntop.org 7 What Happens in Our Network? • Do we have control over our network? • It’s not possible to imagine a healthy network without a clear understanding of traffic flowing on our network.
    [Show full text]
  • High-Speed Network Traffic Monitoring Using Ntopng
    High-Speed Network Traffic Monitoring Using ntopng Luca Deri @lucaderi Some History • In 1998, the original ntop has been created. • It was a C-based app embedding a web server able to capture traffic and analyse it. • Contrary to many tools available at that time, ntop used a web GUI to report traffic activities. • It is available for Unix and Windows under GPL. 2 ntop Architecture Cisco NetFlow HTTP/HTTPS RRD InMon sFlow 3 ntopng Architecture • Three different and self-contained components, communicating with clean API calls. HTTP Lua-based Web Reports Lua API Calls Users nDPI-based C++ Data Cache Monitoring Engine (Linux) Kernel PF_RING C API Calls PF_RING Kernel Module Internet and Drivers Traffic 4 ntopng Monitoring Engine • Coded in C++ and based the concept of flow (set of packets with the same 6-tuple). • Flows are inspected with a home-grown DPI- library named nDPI aiming to discover the “real” application protocol (no ports are used). • Information is clustered per: ! (Capture) Network Device ! Flow ! Host ! High-level Aggregations 5 Information Lifecycle • ntopng keeps in memory live information such as flows and hosts statistics. • As the memory cannot be infinite, periodically non-recent information is harvested. • Users can specify preferences for data purge: 6 Packet Processing Journey 1.Packet capture: PF_RING (Linux) or libpcap. 2.Packet decoding: no IP traffic is accounted. 3.IPv4/v6 Traffic only: 1.Map the packet to a 6-tuple flow and increment stats. 2.Identify source/destination hosts and increment stats. 3.Use nDPI to identify the flow application protocol 1.UDP flows are identified in no more than 2 packets.
    [Show full text]
  • Flow-Based Monitoring, Troubleshooting and Security Using Nprobe
    Flow-Based Monitoring, Troubleshooting and Security using nProbe Luca Deri <[email protected]> @lucaderi © 2017 - ntop.org Packets, Flows, Activities [1/3] • For years monitoring tools focused on standards often fostered by vendors: NetFlow vs sFlow vs SNMP, Cisco vs Juniper… • This has plagued the market by creating tools more vendor oriented, than result oriented. • Fortunately recent advances in computing and in particular the big data movement, have pushed companies to overcome the market/vendor fragmentation and produce tools able to produce data on a standard format (often JSON) that could be consumed even by non-monitoring tools (e.g. Hadoop, ElasticSearch). © 2017 - ntop.org 2 Packets, Flows, Activities [2/3] • As data increases and people demand feature rich monitoring tools, it has become necessary to ‘compress’ monitoring data. • Network packets are still important for providing evidence or troubleshooting problems (packets or it didn't happen!) but they are “too raw” and take too much storage space, so limiting them to specific situations is a good idea. • Network flow analysis is a good way to “compress packets” into events: sFlow do it with sampling, NetFlow with stateful connection-based packet classification. © 2017 - ntop.org 3 Packets, Flows, Activities [3/3] • These days, saving flows on a big data system is a common practice but it still plagued by the visibility issue: ◦What flows are “more relevant” than others? ◦Can we use flows for more than just host/protocol/application traffic accounting ? ◦How can a network administrator look for a needle in a haystack when the monitoring platform is emitting tenth of thousand flows/ second? • We need yet another level of abstraction on top of flows able to identify activities on top of flows (e.g.
    [Show full text]