Creation of two modules for the Network Authentication
Cracking Tool Ncrack
Ilias Panagiotis
SID: 3306150004
SCHOOL OF SCIENCE & TECHNOLOGY A thesis submitted for the degree of Master of Science (MSc) in Mobile and Web Computing
OCTOBER 2015-2016 THESSALONIKI – GREECE
Ilias Panagiotis
SID: 3306150004
Supervisor: Prof. Papadopoulos Apostolos Supervising Committee Members:
SCHOOL OF SCIENCE & TECHNOLOGY A thesis submitted for the degree of Master of Science (MSc) in Mobile and Web Computing
OCTOBER 2015-2016 THESSALONIKI – GREECE
Contents
CONTENTS...... 1
INTRODUCTION ...... 4
INFORMATION SECURITY...... 4
NCRACK OVERVIEW ...... 5
PROBLEM ANALYSIS ...... 6
ANALYSIS OF NCRACK ARCHITECTURE...... 7
OPERATING SYSTEM OF CHOICE ...... 11
ARCH LINUX ...... 11
ARCH LINUX ADVANTAGES ...... 12
Simplicity...... 12
Modernity ...... 12
Pragmatism ...... 12
User centrality ...... 12
Versatility ...... 13
KALI LINUX ...... 14
CREATING A MODULE ...... 15
MODULE DEVELOPMENT PROCESS ANALYSIS ...... 15
TOOLS ANALYSIS ...... 16
NMAP ...... 16
NCAT ...... 17
WIRESHARK ...... 19
GNU DEBUGGER ...... 20
GIT – GITHUB ...... 21
IMAP MODULE DEVELOPMENT ...... 24
WHAT IS A IMAP? ...... 24
IMAP: The Basics ...... 25
1 | P a g e Mail Servers, Email Clients and IMAP ...... 25
IMAP versus POP ...... 25
Using IMAP ...... 25
Main Advantages of IMAP ...... 25
IMAP Server Configuration ...... 26
DOVECOT IMAP SERVER ...... 26
Dovecot server Configuration...... 28
Authentication ...... 28
MAILSTORE SERVER ...... 28
How to install...... 29
Logging on as Administrator ...... 29
Install Test License ...... 29
Change Password ...... 29
Create Users ...... 29
Create Users Manually ...... 30
ARCHIVING MANAGEMENT BASED ON PROFILES ...... 31
Overall Concept ...... 31
Archiving Your First Email ...... 31
User Access to the Archive ...... 32
Using MailStore Client ...... 32
Using the MailStore Outlook Add-In ...... 32
Using Other Email Clients ...... 32
Via an Internet Browser (MailStore Web Access) ...... 32
Using Smartphone (iPhone, Android, Windows Phone, etc.) ...... 32
DEVELOPMENT PROCEDURE ...... 33
Ncat Authentication Attempt...... 37
BUILDING A MODULE ...... 38
IMAP protocol packets ...... 40
IMAP module Source-code Analysis: ...... 41
CASSANDRA MODULE DEVELOPMENT ...... 47
APACHE CASSANDRA ...... 47
NOSQLDATABASE ...... 48
FEATURES OF CASSANDRA ...... 48
INSTALLING CASSANDRA ...... 49
PREREQUISITES ...... 49
2 | P a g e INSTALLATION PROCEDURE ...... 49
Configuration...... 49
Main runtime properties ...... 50
Changing the location of directories ...... 50
CQLSH ...... 50
Description ...... 51
Using CQL commands ...... 51
Saving CQL output in a file ...... 52
Using files as input ...... 52
The cqlsh environment variables ...... 52
CQLSH Authentication ...... 53
CQLSH failed authentication ...... 54
Cassandra Wireshark authentication Packet ...... 55
Cassandra Server Reply - Positive ...... 56
Cassandra Server Reply - Negative ...... 57
Cassandra Packet Analysis ...... 58
CASSANDRA MODULE SOURCE-CODE ANALYSIS ...... 59
MODULE FINAL TOUCHES ...... 66
MERGING OF THE TWO NEW MODULES ...... 68
EMAIL TO NMAP’S DEVELOPERS TEAM ...... 71
GENERAL NCRACK SCAN ...... 72
NCRACK SCAN USING THE IMAP MODULE ...... 73
NCRACK SCAN USING THE CASSANDRA MODULE ...... 75
CONCLUSIONS ...... 77
BIBLIOGRAPHY ...... 78
3 | P a g e Introduction
Information security
Information security or cybersecurity is a field that becomes more and more popular. Information security or infosec, is several strategies we use in order to control and check the processes, tools as well as policies that are necessary to prevent, expose, document and counter cyber threats to crucial information. Part of information security’s aim is to use a set of business procedures that will keep the information assets safe without having to do with the information’s presentation.
Information security programs are developed with some core objectives in mind: maintaining the confidentiality, integrity and availability of Information Technology systems and business data. These objectives guarantee that important information can only be accessed to authorized parties (confidentiality), unauthorized change of data (integrity) and data can be accessed by authorized parties when they want to (availability).
Major enterprises usually employ a dedicated security group to apply and support the organization's cybersecurity program. The security group is generally accountable for conducting risk management, a process in which vulnerabilities and other dangers to information assets are constantly assessed, and the corresponding protective measures are decided on and applied. The core of an organization resides within its information and so, its security is crucial for business operations, as well as the level of credibility required earning the trust of clients.
The dangers to sensitive and private information have many different forms, such as malware and phishing attacks, identity theft and ransomware. In order to prevent attackers and reduce the number of vulnerabilities at multiple layers, a number security controls are applied and managed as part of a hardened defense in depth strategy. This can mitigate the impact of an attack. So, in cybersecurity in order to be able to monitor a security breach, security experts who are responsible for the corresponding system always have an incident response plan in place. This allows them to observe and limit the damage, remove the threat and add updated defense controls.
Procedures and policies related to the field of information security mostly require physical and digital security actions in order to keep sensitive information intact from unauthorized access and use, modification or destruction. These measures can be mantraps, encryption key management, network intrusion detection systems, password policies or regulatory compliance. Security audit may be scheduled to evaluate the organization's ability to keep their information assets safe.
4 | P a g e
Ncrack Overview
Penetration testing is a popular subject and part of the information security field. A wide set of tools are used to cover different areas of an information system. A category of these tools has to do with passwords attacks which involve guessing attempts for selected networked services that essentially use a username and a password authentication scheme. This includes services like such as HTTP(s), SSH, FTP, POP3, etcetera. To automate a username-password attack against a specific service, it is crucial to generate authentication requests for the specific protocol that is used by the selected service. Ncrack is a tool that makes this automation possible.
Ncrack is an open-source project as well as a popular network authenticator cracking tool. Ncrack is a tool designed for penetration testing which aims to improve the network’s security.
Ncrack is created based on a modularized architecture and aims to separate the core engine from each service/protocol which is basically an independent module which is also responsible for all the authentication steps.
Ncrack, like most open source projects, retains thorough documentation which assists in the rapid development of additional modules. The documentation consists of ncrack’s guide, the FTP module’s detailed analysis as well as a developer’s guide to help the developer fully understand the structure of Ncrack and how to proceed with the creation of a new module.
5 | P a g e Problem Analysis
As far as telecommunications are concerned, a protocol essentially is a set of rules that are applied to create a system in which, two machines share information. These rules consist of syntax, synchronization, semantics and maybe some error recovery methods.
Ncrack is built in such a way that utilizes different protocols or services by using the corresponding module. A module needs to be designed using the principles and rules of the service or protocol it supports. In order to achieve that, the module essentially needs to be able to transmit the exact same packets that are required for the official authentication of the specific service. This means that the module will need to be able to receive specific packets, in certain phases of the authentication (or even before the authentication) and also to be developed in a way that it will communicate properly with Ncrack itself.
There are a couple of tools available for network bruteforcing attacks. THC Hydra and Medusa are two of the most popular and they also happen to be the most direct competitors to Ncrack itself. The modules that each of these tools possess in their arsenal are the following:
• THC Hydra: Cisco AAA, Cisco auth, Cisco enable, CVS, FTP, HTTP(S)-FORM-GET, HTTP(S)-FORM- POST, HTTP(S)-GET, HTTP(S)-HEAD, HTTP-Proxy, ICQ, IMAP, IRC, LDAP, MS-SQL, MySQL, NNTP, Oracle Listener, Oracle SID, PC-Anywhere, PC-NFS, POP3, PostgreSQL, RDP, Rexec, Rlogin, Rsh, SIP, SMB(NT), SMTP, SMTP Enum, SNMP v1+v2+v3, SOCKS5, SSH (v1 and v2), SSHKEY, Subversion, Teamspeak (TS2), Telnet, VMware-Auth, VNC and XMPP • Medusa: CVS, FTP, HTTP, IMAP, MS-SQL, MySQL, POP3, PostgreSQL, SMTP-AUTH, Telnet and VNC • Ncrack: RDP, SSH, HTTP(S), SMB, POP3(S), VNC, FTP, SIP, Redis, PostgreSQL, MySQL, and Telnet
So, right now, apart from the number of modules where Hydra seems to have the massive advantage, each of these tools offer different types of advantages. Medusa for example uses thread-based parallel testing in order to attempt to login to multiple hosts concurrently which increases the speed significantly. Ncrack on the other hand, can automatically adjust its behaviour according to the network feedback it gets, so as to achieve maximum performance and efficiency without any user intervention.
In any case, each of these tools, needs a lot of modules in order to reach a wider audience. For this dissertation, two modules will be created for the Ncrack tool. One module will be focused on the IMAP email protocol and another
6 | P a g e one to the Cassandra Database Management System service. The IMAP module is already supported for the rest of the tools and thus it will be a great add-on for Ncrack itself. A module for the Cassandra service is yet to be seen and this will offer Ncrack the advantage of being the only one capable of abusing this specific service.
Part of the challenge will be the design and development of the packets that Ncrack needs to send to the target machines. Thorough packet analysis and research will be crucial for the proper module operation and also for the correctness of the packets that the module will be able to utilize during the authentication process.
Analysis of Ncrack Architecture
As mentioned Ncrack uses modules that correspond to a specific service or protocol in order to handle an attack. Ncrack’s architecture focuses on keeping modules, as independent as possible from the core engine which is responsible for the low-level details of timing as well as the management of each connection.
Ncrack makes use of the popular Nsock, a library of Nmap which was originally written by Fyodor [Nmap’s author] and has since been altered and tested thoroughly.
Nsock is a parallel sockets library which internally uses select(2) to choose among the registered socket descriptors and which upon a new network event(either write/read/timeout etc.) corresponds to a preregistered callback handler which is in charge of doing something about that specific event. In conclusion, Ncrack is an event-driven API which consists of its core engine nesting inside the ncrack.cc which includes definitions of the following callback handlers: void ncrack_connect_handler(nsock_pool nsp, nsock_event nse, void *mydata); void ncrack_write_handler(nsock_pool nsp, nsock_event nse, void *mydata); void ncrack_read_handler(nsock_pool nsp, nsock_event nse, void *mydata); void ncrack_timer_handler(nsock_pool nsp, nsock_event nse, void *mydata); void ncrack_module_end(nsock_pool nsp, void *mydata); void ncrack_connection_end(nsock_pool nsp, void *mydata);
After the parsing of the user’s host specifications and options, it keeps all the important information it needs, in an object named ServiceGroup (ServiceGroup.h)
ServiceGroup / Service
7 | P a g e The services that are going to be tested, are stored in the ServiceGroup object and they are Service objects. The Service class includes important variables that deal with timing and statistical information, user specified options that apply to this specific service/host, a pointer to a Target object (extremely similar with the Nmap’s Target class, although some fields are left out because of their uselessness for ncrack itself), some other functions that take care of the authentication iteration (username/password list) and an additional list for the active connections that are utilized at that moment.
Connection
A connection (Connection.h) is an instance that stores information of the particular TCP session between us and the service we attempt to crack. It is a fact that a connection must always belong to the Service class. In most cases, during a single connection, a lot of authentication attempts are going to be carried out, depending on the corresponding service.
Module state machine
The most vital thing about a connection, is the 'state' it is currently in.
The 'state' actually corresponds a number of steps that are needed by the service we are cracking, for the authentication procedure. Thus, the number and names of states are developed and specified in each module separately. For example, the authentication step, in which we wait to read the initial banner of the targeted service at the beginning of the connection, is specified by a particular 'state'. Another example, is the 'state' in which we should write the password on the wire or the 'state' where in a service like telnet we need to make the option negotiation. It is also important that every 'state' performs a micro-action of the authentication process which will usually force a certain nsock event to be registered. Of course, this might not always be the case (like telnet for example).
Ncrack core engine
The main nsock loop is located in the function ncrack() @ ncrack.cc and it is responsible for polling for new events that end up calling back one of the registered handlers mentioned before. The ncrack_probes() is called in the end of every loop iteration and verify if it can start off new connections against(all) the available targets in ServiceGroup.
8 | P a g e In order to understand how ncrack_probes() operates, we initially need to examine the way that ServiceGroup handles its services lists. In the very beginning, every user-specified service is saved inside the 'services_active' list. For every service that resides there, we can automatically start a new connection at will. ServiceGroup keeps much more additional lists which hold services that cannot perform additional actions for now (like initiating a new connection) except for the connections that have already started. For example, 'services_full' holds all services that cannot initiate an additional connection since the total number of active connections taking place at that moment has reached the maximum allowed limit (connection limit). The list 'services_wait' stores all services that need to wait for the time of the 'connection_delay' (usually selected by the user) before they can send another connection probe. The list 'services_finished' saves all services that have been labeled as finished, either because a critical error had occured (for example if we got an RST at the first connection attempt or we didn’t stop getting many timeouts for a prolonged time) or the username/password list iteration finished. The concept of keeping separate lists whose name imply the reason that the elements of the list reside there, is also used by Nmap's service scanning engine.
ncrack_probes()
ncrack_probes() iterates the ServiceGroup 'services_active' list, starting new connections until all services have been moved inside a different ServiceGroup list. Note that it does not wait for any connection to actually finish the 3-way handshake, since nsock uses non-blocking connect(2)s and ncrack only needs to register the event and the callback handler (ncrack_connect_handler).
ncrack_connect_handler()
Upon successful connection ncrack_connect_handler() passes control to the call_module() which summons the service module function corresponding to the particular service this connection is up against. If the connection times out or we get an RST and we are in the first connection attempt, then we mark the service as 'dead’ and it is moved into
'services_finished' list. This is quite useful when the user specifies the targets in a wildmask or a netmask notation, blindly searching for services to crack. It is possible that some hosts will not even have that service listening and thus we will stop trying to crack them. It is also important to note that the first connection probe (boolean 'just_started' @ Service class) also gathers valuable timing information like how many authentication attempts the server allows to make per connection. That is why ncrack avoids opening more than one connection probe against a service before that first timing probe finishes its job (which will provide the maximum allowed authentication attempts during that connection).
ncrack_write_handler()
9 | P a g e The write handler is probably the simplest one. The only thing it must do is check the nsock return status and report in case of error. The case of a write failing is the most unlikely one. It can always happen though in case we write on a closed socket (which won't normally happen since we always check if the socket is currently active or the peer closed on us) or in case where the kernel's socket buffers are full (which can only occur on some very old systems with a small amount of RAM).
ncrack_read_handler()
The read handler is responsible for filling the Connection's auxiliary buffer upon a successful nsock read event. Additionally, we use a trick to check whether or not the peer is currently active or it has sent us a FIN in order to close the connection. Whenever the Boolean 'check_closed' is true, if nsock produces a TIMEOUT instead of an EOF error, then this means that we are still online. This happens because the caller that wants to check the connection state, registers a read event with a very small timeout. This is a hack that allows us to check in a portable and efficient way if we have moved to the CLOSE_WAIT state from ESTABLISHED.
ncrack_module_end()
This function is called by a module only when it knows that it has completely executed an authentication attempt. It also updates statistical variables for the service, like the total number of attempts that have been made and implements part of the dynamic timing engine. Every 500 msecs it checks if the current authentication rate is less than the last calculated one and takes suitable steps to increase it. Since the 'ideal_parallelism' variable which is the dominating connection metric, can increase or decrease, we also examine if we can move our service from 'services_full' to 'services_active' and call ncrack_probes() to potentially start new connections. Finally, if we need to check whether our peer is alive (variable 'peer_alive' is false), we do the aforementioned read timeout we discussed.
ncrack_connection_end()
This is one of the most complex functions. It performs all necessary actions whenever a connection is ended either normally or by an error. Firstly, it checks if we have received a FIN from our peer, in which case one of the following could have happened:
i) The peer might have closed on us 'unexpectedly': this happens in cases like
10 | P a g e like telnet that can close the connection promptly after giving the final
results of the latest authentication attempt. For services like the aforementioned we need to always set the variable 'peer_might_close' inside the module immediately after the state that is in charge of writing the password on the wire and before the state that registers the next read call. If we are on the first 'timing' probe, then we increase the amount of supported authentication attempts per connection for this service.
ii) The peer might have closed the connection normally in which case we don't do anything.
iii) The peer might have closed the connection in the middle of the authentication. This is not a usual occasion and it is an indication of a really strange error, usually due to extreme network conditions.
When the aforementioned or a timeout in the middle of the authentication happens, then we promptly adapt the dynamic timing engine to drop the 'ideal parallelism' limit.
Next, if we are on the first timing probe, depending on the timing template, we
calculate our initial ideal parallelism. Furthermore, we update the authentication rate meters accordingly and in the end of the function we call ncrack_probes() since we might have changed the 'ideal_parallelism'.
Operatin g System of choice
Arch Linux
Arch Linux is the operating system that is used for the servers’ installation and configuration. The choice is made because Arch Linux offers an efficient and clean environment that eliminates difficulties with the installation and further configuration of the tools that are needed. It is a developed, i686/x86-64 GNU/Linux distribution that serves those who want the latest stable versions of every available software by using a rolling-release model. Starting with
11 | P a g e the installation, Arch uses a minimal approach which essentially provides an effective way to let the users install from scratch whatever they need, without having to deal with preinstalled and unnecessary software.
Arch Linux Advantages
Simplicity
Arch Linux offers simplicity by avoiding unnecessary additions or modifications. It comes with software that is designed by the original developers without being modified while also offering the minimal distribution-specific changes. When patches are not accepted by the upstream, they are avoided and the downstream patches consist mainly of bug fixes and other similar changes.
Following a similar approach, the configuration files that are provided by Arch’s upstream, change vital distribution- specific issues such as the system file paths, etc. It does not provide automation features by default such as enabling a service just because the package was installed. For a package to split, it happens only in cases of extreme need and mostly when the change will be beneficial in every possible way, like saving disk space. Configuration for Graphical User Interface is avoided, to guide their users to configure the system from the terminal using simply a text editor.
Modernity
Arch Linux strives to keep the latest stable release versions of its software while also trying to keep their software packages reasonably stable. Essentially it offers the so-called rolling-release system, which offers a one-time installation with constant upgrades.
Arch adds a lot of the newer features that are included in most GNU/Linux distributions such as the systemd init system, RAID, udev support, initcpio as well as the latest kernels
Pragmatism
Arch follows a pragmatic approach rather than an ideological one. The core principles have to do mainly with guidelines and useful methods. Every design decision is chosen after technical analysis based on practicality and effectiveness rather than popularity or politics.
Arch Linux offer an extreme number of packages as well as build scripts through its various repositories and while they offer mostly open source and free software options, they also provide proprietary software packages for those who prefer functionality over ideology.
User centrality
Whereas many GNU/Linux distributions try to be more user-friendly, Arch Linux has always been, and will always focus on user-centric philosophy. The distribution’s main purpose is to cover the needs and demands of the users who contribute and maintain it rather than aiming to be appealing to get more users. The target user for Arch Linux is the
12 | P a g e proficient who prefers to customize the system himself and resolve his own problems using the thorough documentation.
Every user can contribute and help with the maintenance and evolution of the distribution. Reporting and helping fix bugs as well as resolving issues is highly appreciated while patches that improve packages or the core aspects of the system are highly valued. Arch's developers (Archers) can assist with the development of the distribution in various ways. Some of them are improvement of the documentation, issue reports, exchange opinions using the official forums or IRC channel or even translation of the documentation.
Versatility
Arch Linux belongs to the general-purpose GNU/Linux distros. Upon installation, the user has only a terminal in his arsenal. This results in a package-free environment which stays free of unwanted or unneeded software. The user has the luxury to build his own custom experience as the repositories provide a plethora of high quality packages that supports both i686 and x86-64 architectures.
Arch is equipped with pacman a lightweight, simple and fast package manager that can upgrade the entire system using a single command. Furthermore, Arch provides the Arch build system, a system that makes easy the building and installation of packages from source, which can also be configured to be used with one command. So lastly, the distribution offers many community contributed PKGBUILD scripts in order to allow compiling from source while also making possible for a user to keep and maintain his own repository.
13 | P a g e
Kali Linux
Kali Linux is a Debian-based Linux distribution which specializes in advanced Penetration Testing which is also one of the best platforms for using tools like Ncrack. Kali comes with a large number of tools that specialize in tasks of Penetration Testing, Computer Forensics, Security Research, Reverse Engineering and others. It also includes all the basic tools that are going to be used for this dissertation such as Wireshark, Ncrack itself, Nmap, Ncat and others.
Kali Linux includes several advantages such as:
• Hundreds of Tools: Kali Linux include all the tools that were once included in Backtrack and their number is growing every day. • Open Source: Kali Linux, like Ncrack is open-source and completely free. There is no reason to pay for any kind of tool already included. • FHS compliant: Kali follows the Filesystem Hierarchy Standard, allowing Linux users to navigate easily and locate whatever they need with ease. • Wide-ranging wireless device support: A strong point of Linux distribution is the support for a variety of wireless interfaces and Kali Linux tries to enhance this concept and support as many as possible wireless devices as possible. • Developed in a secure environment: Kali Linux focuses on establishing a secure environment so as to let the user trust to commit and interact with his repositories. Since all the Ncrack modules development will be implemented in github, such a feature is always convenient to have.
14 | P a g e • Multi-language support: Even though every tool is written in English, Kali Linux ensures that the operating system supports a large number of languages in order to make the navigation easier for different individuals. • Completely customizable: The design of Kali Linux may not be for everybody’s taste, and thus the operating system offers the flexibility to adjust for the target user.
CREATING A MODULE
Module Development Process Analysis
The process of the development includes the following steps: 1: Server installation and configuration
In this part we install the servers of our choice to the machines of our choice. For IMAP, we used the Dovecot IMAP server as well as the MailStore IMAP server. On the other hand, for Cassandra, we installed the official Cassandra server. The next step was to configure the servers in a way that they could authenticate a remote user.
2: Remote Connection and Authentication
The next step includes an additional utility that will help us connect and authenticate ourselves to the corresponding server.
3: Packet Capture
Next comes, the capture of the connection – authentication packets. 4: Packet Analysis and Research
This is where the structure of the packets will be revealed in order to reconstruct them. 5: Module Development - Testing
This is where the module development and testing take place. The module includes the reconstructed packets which are gradually developed through testing as well.
15 | P a g e
TOOLS ANALYSIS
Nmap
Nmap is a free and open source project which specializes in network scanning. Nmap’s main functionality is to discover what hosts are available on the network, what services are being used, which operating system, what kind of firewalls or filters are included and some other characteristics as well. Nmap is designed mostly for targeting large networks. It is compatible with a number of operating systems including Mac OS, Windows, Linux and BSD. Nmap also provides:
16 | P a g e • Flexibility: Nmap provide a plethora of advanced techniques for mapping out networks which include IP filters, firewalls, routers, and others. • Power: Nmap is designed for scanning a enormous networks of a large number of machines. • Portability: Nmap supports most operating systems like Linux, Windows, FreeBSD, OpenBSD,NetBSD, Sun OS, Amiga, and others. • Ease: Nmap uses a simple command line as well as a GUI version that manages to provide simplicity while also providing a massive set of features • Well Documented: Nmap is a large open source project and manages to build massive and well-constructed documentation that stays up to date and supports a large number of languages.
Ncat
======
Ncat is a networking utility and part of the Nmap’s project. It is essentially an improved implementation of Netcat.
So, in order to understand what purpose, Ncat serves, it would be better to examine in depth Netcat itself. Netcat is a Unix program that is used for reading and writing across the network utilizing TCP or UDP protocols. It is developed to be an efficient and reliable tool that can be used either directly from the user or by separate programs or scripts.
17 | P a g e Netcat offers a variety of network debugging and exploration tools and manages to be compatible with any kind of connection.
Using the simplest command "nc host port" establishes a TCP connection to the target port of the specific host. In the same manner, using Ncat, the command is “ncat host port”. When the user sends something to the host, it is sent as a standard input to the targeted port of the specific remote host while whatever comes back through the connection, is sent to the standard output of the user. This continues repeatedly until the user terminates the connection.
Netcat can also be used as a server. It can be configured to listen for incoming connections on selected ports and then proceed to read and write. There are some limitations, but Netcat, while playing the role either of client or server, takes care of data between the sender and the receiver until there is not any left. It can also be configured to shut down after a fixed time of inactivity on the network side.
All of this can be applied using UDP protocol too, which also replace the telnet (does not support this protocol at all. Also, this gives the opportunity to test UDP-oriented servers).
There are a lot of differences between Netcat and Telnet. The main reason is that Telnet needs calculated delays for the network output to finish. On the other hand, Netcat remains operational until the network side closes. Additionally, Telnet does not have the ability to send arbitrary binary data due to the fact that specific characters are considered Telnet options and thus they get removed from the stream of data. Telnet also differentiate from Netcat at how they the data. On the one hand Telnet lets off some of its diagnostic messages while Netcat never modifies data until the user chooses so. Finally, Telnet cannot listen outgoing connections or even use UDP. Netcat does not have any of these disadvantages and while it is smaller and faster, also offers some other benefits as well.
Some of Netcat's major features are:
18 | P a g e
Wireshark
Wireshark is a maybe the most popular open source packet analyzer. It offers a variety of features mostly focused network analysis, troubleshooting, communications protocols and education.
Wireshark is used in both Unix and Windows operating systems using Qt toolkit for interface implementation for its current releases. It also uses pcap for packet capturing. There is a terminal version which does not use a graphical user interface.
Functionality
Wireshark has a lot of similarities with tcpdump, but wireshark has a graphical interface, while adding some filtering and sorting options.
Wireshark lets the user observe all the traffic independently of the interface’s configured addresses or broadcast traffic. Wireshark also offers:
19 | P a g e • Examination of a large amount of protocols grows over time • Offers offline analysis • Compatible with many operating systems such as Windows, Linux, OS X, Solaris, FreeBSD, NetBSD, and many others • Can be used through its GUI version or the terminal-based TShark • Includes several efficient and effective display filters • Also, offers Voice-over-IP analysis • It is compatible with different types of file formats (for capturing). This includes tcpdump (libpcap), Pcap NG, Catapult DCT2000, Cisco Secure IDS iplog, Microsoft Network Monitor, Sniffer® Pro, and NetXray®, Network Instruments Observer, NetScreen snoop, Novell LANalyzer, RADCOM WAN/LAN Analyzer, Shomiti/Finisar Surveyor, Tektronix K12xx, Visual Networks Visual UpTime, WildPackets EtherPeek/TokenPeek/AiroPeek, and many others • It can easily decompress captured gzip files • I can easily read live data Ethernet, IEEE 802.11, PPP/HDLC, ATM, Bluetooth, USB, Token Ring, Frame Relay, FDDI, etcetera • Includes decryption support for some protocols like IPsec, SSL/TLS, WEP, and WPA/WPA2 • Ability to export the output in different types(like XML or plain text) for enforcing compatibility
GNU debugger
A debugger is a program that essentially runs on top of other utilities in order to examine the results of their running commands. This allows additional control over the errors and bugs of our program and thus, it is quite useful for the process of the development.
20 | P a g e GNU Debugger, or gdb, is one of the most popular debugger for *NIX operating systems for the C and C++ projects.
GNU Debugger provides general information about:
GDB Functionality
GDB essentially runs the program until a certain degree and prints the values until that exact moment. This can be done for even one line at a time providing thorough detail about the operation of each part of our project
Git – Github
Git is one of the most popular version control systems and is used a lot nowadays. Git is a free open-source project which was created by the same person that developed the first kernel for Linux, LinusTorvalds. A large number of projects use Git for its version control system, both commercial and open-source. Git
21 | P a g e also works extremely well on a variety of operating systems and IDEs providing availability for everyone who needs it.
Git avoids storing the whole version history of a project in a single place and uses every programmer’s working mirror as his own repository that saves all his changes. Other version control systems like Subversion do not have a characteristic like this.
Apart from being distributed, Git focuses on performance, security and flexibility.
Performance
Git focuses on optimizing every part of its functionalities providing very strong performance. Characteristics include ‘commit’, ‘merging’, ‘comparing versions’ or ‘branching’.
While some other version control programs are fooled by names of the files, git focuses on file contents. Git is designed this way because files are usually renamed, split or changed.
Security
Git has been developed by keeping a security focused design. Every file as well as its relationship with other files, its version, the tags and commits remain secure by optimized cryptographically hashing algorithms. This prevents damage caused by accidental changes and ensures that the history can always be found.
Flexibility
Another advantage that Git highlights, is flexibility. Git offers support for different kinds of nonlinear development workflows. It is also compatible with a great number of protocols and systems.
Version control with Git
Git is maybe the best choice for nearly every project team nowadays. While everyone needs something, different and has to do analysis in his own way, Git tries to outperform most of the competition by offering advantages like the following.
Git has everything
Git is functional, flexible, provides performance and focuses on security in order to cover most of its user’s needs.
Git is the new standard
22 | P a g e Git is widely known and is used by a large number of teams worldwide.
In addition to the popularity that Git possesses, Git has also made third party software tools and services to be integrated in the Git version control system, like IDEs, a new desktop client called Source Tree, project tracking software like JIRA and code hosting services like Bitbucket.
Git is a quality open source project
Git is an open source project that is massively supported during the product’s lifetime. Git’s maintainers have done a great job updating the project by focusing on meeting the long-term needs with frequent releases that improve stability, usability and functionality. Git’s quality is of critical importance since a lot of businesses rely on it.
Git also has great community support and thus, the documentation and the educational sources are plentiful, well- detailed and always up-to-date. Being open-source, Git gives the opportunity to freelancers to benefit from all its features without paying any fees. So, in the very end, Git is undoubtedly the successor of older open-source version systems like CVS or SVN.
Git Drawback
A lot of people believe that Git is hard to learn. It is true that some of Git’s functionalities are not that easy to use for newcomers or users of different systems. Also, Git uses different terminology than SVN or CVS. Nevertheless, Git is quite powerful and flexible at the same time. If a user spends enough time for adjusting to a program like Git, he will be able to increase the speed of his projects development in an instant.
Some teams that come from a non-distributed VCS, believe that having a central repository is a good feature and thus, they don’t want to lose it. Since Git is distributed version control system you can have both an official repository where all of the changes of the core project take place but also every single developer can use his/her own repository as a separate branch without affecting the performance of the main server. Because of Git’s flexibility, someone can work as he is used to but he could also find some of the additional Git’s features come in handy.
For Ncrack I created a separate repository with my Ncrack mirror in order to be able to keep track of the versions of my project. Git also played a huge role in cooperating with Ncrack’s author so as to proceed successfully with the development.
23 | P a g e IMA
P MOD
ULE DEV
ELO PME NT
What is a IMAP?
It is a fact that everyone who has set up an account in the past, has come to choose between which email protocol to use, POP or IMAP. While this question seems very technically specified, the selection of the protocol will result in a noticeable difference in the experience of sending, receiving and general use of email messages. While POP, or Post Office Protocol, seemed to be the most used in the past years, IMAP - or Internet Message Access Protocol has become the most popular choice for most people today. So how does IMAP function as a protocol and especially against POP and what advantages does it offer?
24 | P a g e IMAP: The Basics
So, as the name suggests, IMAP protocol gives you access to your emails independently of where and when. This is possible using the Internet. So, emails are kept in servers and when someone tries to access the inbox of his account, the corresponding server presents the emails to the email client of the user. Basically, when someone checks a target email, the email is not downloaded on the specific device. Instead it is read from the server and thus, it can be accessible from more than one devices without altering the content.
Mail Servers, Email Clients and IMAP
To understand how IMAP operates, it would be like thinking it play the role of an intermediate entity between the server and the email client. So, email servers are always active when receiving or reading emails. The only way that an email is changed in a server is when there is a request for deletion. So, when a user uses an email client to sign in, he gets a representation of his inbox. The emails appearing are ‘tokens’ and they are not downloaded in the device until they take a request for reading. In other words, unless someone opens an email, the email is not downloaded in a specific device.
IMAP versus POP
IMAP is completely different from POP in terms of functionality. POP is a protocol which operates with a less flexible manner. That’s because when a user want to access his emails, the corresponding device downloads all the emails from the server. After the completion of the download, the server remains empty from emails. This is fine for someone who access his emails from a single device but if the user changes locations and needs to check his emails from different devices, IMAP is a much more suitable choice.
Using IMAP
While POP requires downloading of the emails in order to access you data, IMAP offers the ability to write, read, organize and access your emails while they are still on the server. This results great speed, efficiency and performance. Another advantage is that the server gives the ability to access the emails that the user sent from anywhere. It lastly, does not move emails from the email client to the server, instead it just synchronizes them.
Main Advantages of IMAP
There are a lot of benefits while using IMAP. Firstly, it offers remote access to the emails from different devices. Secondly, it downloads the content of the email only if the user wants to read it. This comes in handy in occasions where the user only needs to check a new email and does not have to download the whole list of emails to be able to
25 | P a g e access them. Furthermore, IMAP does not download attachments automatically providing additional control over the content of the user’s emails. Lastly, IMAP has an offline mode to provide additional functionality.
It is a fact that mobile devices become more and more popular. The combination of mobile device’s popularity with the evolution of cloud computing help IMAP be established as one of the most efficient email protocols. Most users nowadays possess at least a smartphone and a laptop while some also buy a tablet so, having the ability to access you emails effectively is crucial and really makes the difference.
IMAP Server Configuration
The first part of the development’s process is to set up the IMAP environment. For this, the Dovecot IMAP server will be used for the GNU/Linux distribution while the MailStore enterprise server is the server of choice for the Windows 10 operating system.
Dovecot IMAP Server
In order to understand what Dovecot covers as a server it is essential to examine the whole procedure of a classic mail message’s lifespan.
To start with, there is a user who wants to create a mail message using an email application. So, a program that let the user send and receive emails is called Mail User Agent or MUA. Some quite popular examples of MUA is Icedove for GNU/Linux platforms and the Office365 web application. Technically though, even if MUA was utilized the email was created and sent to an email server. In other words, the user does not interact in any way with the server itself like the MUA. Server’s main purpose is to either push the email to its destination or handle email arrival from other computers. The email server is also known as Mail Transfer Agent or MTA. The MTA is responsible of checking a new email and after the processing of the corresponding Domain Name System finds which MTA is the one responsible for handling email for the target recipient. So, when a user sends an email, the email is sent to the MTA of the ‘sender’ to the MTA of the ‘receiver’.
Depending on the target network settings, the sender’s email will most probably end up in another MTA. The MTA at some point will need to deliver the message to its destination. This is the moment where the MTA will deliver the email to the Mail Delivery Agent or MDA. This Agent is responsible for storing the email to the disk. When configured, the MDAs can do things like filtering or delivering emails to folders. Their main purpose though is to store the emails on the server.
26 | P a g e Now, let’s move to the part where the user needs to check his emails. The MUA is initiated and the mail server receives a query which informs him what protocol the user is using (either IMAP or POP3). The server authenticates the user and sends all the emails to the operating MUA. Finally, MUA shows all the available emails, ready for reading.
So, this is the part where the Dovecot server is making the difference.
As a server which supports IMAP and POP3 protocols, Dovecot offers the ability for the MUAs(Mail User Agents) to access the emails. So, for a user to access his emails, he should utilize a specific software and more specifically an IMAP or POP3 server. As an IMAP and POP3 server, Dovecot offers the ability for the MUA to get access to the emails. So, IMAP and POP3 servers receive requests from different Mail User Agents to provide access to the corresponding inbox. Dovecot is software which plays the role of IMAP and POP3 server.
Additionally, Dovecot brings functionality for final message delivery utilizing the Dovecot Local Delivery
Agent(LDA). The LDA’s main functionality is saving emails into the message store. Additionally, MTA, Mail
Delivery Agent or Dovecot LDA take care of the local delivery. The choice is made based on the configuration and the server needs.
It is important to clarify that Dovecot does not receive emails from another server. Instead it is responsible for handling (a) messages that are taken out of the local message store to end up to IMAP or POP3 clients and (b) emails that have arrived and need to be placed in the local message store.
IMAP and POP3 are the two popular protocols used by Mail User Agents to communicate with mail storage servers. POP3 is mostly used when the user do not have a fast internet connection. One of the basic features of POP3 is that it downloads the emails and store them locally while also deleting them from the server. IMAP on the other hand is focused on high speed connections. Every time it communicates with a server to provide the email that the user wants to read. Dovecot is highly optimized for the use of IMAP specifically and that’s why it was one of the top choices for the server of choice for the purpose of the IMAP module development.
Dovecot is not directly taking part in the actual reception of e-mails. That purpose is fulfilled by an MTA like Exim or Postfix. When the email arrives into an MTA, it either gets delivered directly by the MTA, by another MDA or by the LDA that Dovecot provides. The choice depends on the purpose the system is trying to achieve and what kind of customers it serves. Flexibility is a key especially for the software that plays the role of the IMAP server and this is one of the main reason, Dovecot remains so popular among Unix operating systems.
The main options for storing emails on the Unix world is either mbox or Maildir. Mbox stores many messages in a single file. On the other hand, Maildir stores every mail in a different file. Both Maildir and mbox are supported by a variety of MTAs and MDAs and they are also fully supported by Dovecot. In addition, Dovecot also uses its own mail storage formats: sdbox and mdbox.
27 | P a g e Dovecot server Configuration
Dovecot configuration includes mostly mail storage type, mail storage location, user list, and a password list.
Right now, Dovecot supports a fair amount of user and password sources, such as *NIX passwd, shadow, PAM, LDAP, SQL, and vpopmail. It's usually most appropriate to select a source that is being utilized by all the parts of the user’s overall mail solution, including your MTA, MDA, and Dovecot.
Dovecot is expected work better when it delivers emails using its own LDA which needs to be synchronized with the MTA in order to provide the MTA with the ability to deliver emails to the Dovecot LDA.
However, there exists a default configuration for the Dovecot that does not actually use the Dovecot LDA, and thus, it is used solely as an IMAP and/or POP3 server. In such cases, it is essential to make sure that the MDA and Dovecot are configured properly.
To locate the Dovecot configuration file, you use: doveconf -n | head -n1
If the installation was completed from sources, no configuration file is generated. Instead, a README file on the config directory should be sufficient to use as a basic configuration. Thus, we do: cp -r /usr/share/doc/dovecot/example-config/* /etc/dovecot/
The default configuration is usually contained in a file called dovecot.conf, which includes an !include conf.d/*.conf statement to read the rest of the configuration. This split of the configuration files does not have to be used, and does not have any particular impact on which file contains a specific setting, as long as it is not overridden from another file. To check if everything works as expected, the user can use doveconf -n.
Authentication
Dovecot’s default settings are configured to use the system user authentication. If needed, the user could also make additional research, reading PAM or bsdauth to configure it.
The default system user authentication is the authentication method that was used for the packet research although for the testing part, both authentication methods were taken into consideration.
MailStore Server
28 | P a g e How to install
Due to low system requirements, MailStore Server can be installed on a normal personal computer for testing. Testing with a larger number of users should be done in an environment that follows the corresponding System Requirements.
Hence it is essential that the admin account is available for proper modification of the server.
Logging on as Administrator
In order to enter as administrator, the user has to initiate the MailStore Client application and fill the credentials (Usern: admin / Pswd: admin) using the Standard Authentication. Finally, it is essential to confirm that the MailStore server is a trusted host.
Install Test License
The next step requires a product key validation. In this step there is a trial option which lets the user take advantage of the services of the server for a month. After the choice is made, a restart is required and after that, the user is ready to use again the admin credentials to configure the system. For this dissertation I used the free trial.
Change Password
One crucial step is the change of the default ‘admin’ account’s password as a paramount security measure. For the purpose of the testing, the password got changed.
Note: If someone leaves the password set to the default admin, he will not be able to use this account to remotely connect to the server. This setting is applied for security reasons.
Create Users
MailStore server offers the ability to synchronize users using either Active Directory or generic LDAP support. Additionally, MailStore Server allows syncronization from Mdaemon, Kerio Connect and IceWarp servers. The user can also sychronize from his own backend.
Option 1: Synchronize Users
29 | P a g e To configure properly the synchronization with a directory service in MailStore Server, the user needs to do as follows:
Create Users Manually
To create users manually, you should follow the steps below:
This is something important for the purpose of this dissertation. The module must be tested in more than one user in a single server.
30 | P a g e Archiving Management based on Profiles
Overall Concept
In order to archive, Mailstore Server uses archiving profiles which contains the following:
Archiving Your First Email
MailStore users who have privileges by the administrators, can archive their individual mailboxes, their email profiles as well as single emails.
In addition, administrators can archive emails for every user they want. This can come in handy when there is a need for archiving multiple mailboxes. Below, we see how you can archive a single mailbox:
31 | P a g e After archiving is complete, you can immediately access the archived emails using the folder structure provided on the left of the window. The archiving profile you have just created is now displayed in the Saved Profiles list. From here, you can run it whenever it is required.
User Access to the Archive
A User can access his archives in multiple ways. By clicking the Install MailStore Client on other Computers they can access the MailStore Client as well as the MailStore Outlook.
Using MailStore Client
By using MailStore Client, the user can get access to his own personal emails from everywhere inside a specific network. If the user has the corresponding privileges, he can archive his emails as well. This can be done by different computers of the same network.
Using the MailStore Outlook Add-In
The MailStore Outlook add-in utilizes the Microsoft Outlook for easy and efficient access to the emails or archives of a single user.
Using Other Email Clients
MailStore Server also includes an integrated IMAP server for those who want to be able to have access to their archive from everywhere, in every device they own. This feature enables tport 143 for the IMAP protocol to take place and thus, the module can utilize it in order to retrieve passwords.
Via an Internet Browser (MailStore Web Access)
MailStore Web Access provides the ability for the user to access his inbox from the browser of his choice. By doing this, it offers the flexibility, a user needs while sustaining the ease of use by keeping the web application simple enough.
Using a Smartphone (iPhone, Android, Windows Phone, etc.)
MailStore Mobile Web Access provide the convenience of accessing the inbox from different kind of smartphones. This is compatible with either iOS, Android or Windows Mobile. This feature also offers a search engine for easier navigation through the archive.
32 | P a g e
Development Procedure
Ncat is the utility that is chosen for the remote authentication of the IMAP protocol. In order to achieve the authentication, firstly we need to use the following command for initializing the connection.
‘ncat -v 192.168.1.4 143’
-v : is an option that gives verbose output in order to appear more understandable. 192.168.1.4: is the IP of the target system
143: is the port that ncat will use to establish the connection
When the server is, up and running, it replies as depicted above. On the other hand, when the server is offline, port 143 is not properly configured or there is a connection error for any kind of reason, then the network responds in the following way
33 | P a g e
Whenever the network exhibits connectivity issues, Ncat gets an error which describes the situation.
For example “Ncat: Network unreachable.”
So, ultimately, Ncat had to complete the remote authentication. In order to achieve that, we needed to complete the following steps:
1: Ncat Connection Establishment ‘ncat -v ip_address port’ 2: Dovecot Authentication ‘## LOGIN username password’
While we are following this procedure, Wireshark is capturing packets for further analysis. It is important to note that the ‘netcat’ utility could have been used instead of ‘Ncat’.
34 | P a g e
In case the credentials are wrong, the Dovecot server has to reply back to Ncat about the new status of the connection.
In this case it responds with an ‘Authentication failed.’ message.
35 | P a g e
Wireshark response
So, the authentication process is divided in three steps. One packet is sent for each step. In the wireshark packet capture above, the first step is depicted where the server approved the connection.
Dovecot responded with a ‘Dovecot ready. \r\n’.
36 | P a g e MailStore Server responded with a similar response as we can see below
Ncat Authentication Attempt
37 | P a g e The next packet is sent during the authentication process by Ncat. In this example, it is good to recall that Dovecot used the following syntax for the authentication.
## ‘method’ username password or 01 LOGIN barrend zergling539
Essentially this is the packet that should be implemented in our module. In other words,
Ncrack has to send a packet with a plaintext payload of the form ’01 LOGIN username password’ to complete the
Authentication process.
Building a Module
To develop a module there is a number of steps that the developer needs to complete beforehand.
A Macroscopic View The source code for the IMAP module is located in the ncrack/modules/ncrack_imap.cc. The file contains a part of the code, (provided by the developer’s manual) that offers insight regarding some vital parts to be included in our module, as well as, some initial steps that the developer has to follow. In our module we are going to use the following:
[License Header]
[Library Inclusions]
[Function Declarations]
[Auxiliary Functions]
[Main Module Function]
These are the main sections of every module and thus, we will include them in the IMAP and Cassandra modules as well.
-- [License Header] --
This is the Nmap license copyright notice that is included in every Ncrack file. In the first line we also describe, what the module is about. For example, in our IMAP module, we’ll get:
/***************************************************************************
* ncrack_imap.cc -- ncrack module for the IMAP protocol *
38 | P a g e * *
***********************IMPORTANT NMAP LICENSE TERMS************************
* *
...
-- [Library Inclusions] --
These are the files which every module must include:
#include "ncrack.h" #include "nsock.h" #include "NcrackOps.h" #include "Service.h"
#include "modules.h"
Some of the modules include some additional files, although our modules will just use the five default libraries.
-- [Function Declarations] --
This part includes the declared functions by also including an ‘extern’ reference for all of the Ncrack’s core functions that are needed by the module. These are the following: extern void ncrack_read_handler(nsock_pool nsp, nsock_event nse, void *mydata); extern void ncrack_write_handler(nsock_pool nsp, nsock_event nse, void *mydata); extern void ncrack_module_end(nsock_pool nsp, void *mydata);
We also use the enum user-defined data type to define the states of the scan. For the IMAP module, we get: enum states {IMAP_INIT, IMAP_USER};
-- [Auxiliary Functions] --
These are mostly helper functions and usually differ based on the module that needs them. There is often a procedure for reading protocol-specific data. This is quite important, since Nsock is protocol-agnostic. Its main functionality is to transfer network data from the kernel, to Ncrack’s incoming buffers. The rest of the parsing is assigned to the module itself. For example, the imap_loop_read() parses incoming data from Nsock and translates what kind of IMAP answer returned.
-- [ Main Module Function] --
39 | P a g e This is the function that plays the role of the core engine of the module, storing its state machine as well as the main steps that are needed to complete and assess the authentication phase of the corresponding protocol. Basically, it consists of a long switch statement, which maps each case with one of the module’s states. Every state has to end with a call to the core function of Ncrack. This happens because of Nsock’s event-driven design. Therefore, the module essentially reads or writes from the network using the nsock_write or nsock_read or notifies when a phase for this round has ended using the ncrack_module_end.
IMAP protocol packets
40 | P a g e As we saw before (as depicted above), after the connection gets established, the dovecot server (IP: 192.168.1.2) sends a packet that contains the status of the server. In this example, the server is ready to accept an IMAP command. Ncrack needs to only send the LOGIN command, so it crafts the corresponding TCP payload, using the appropriate syntax provided by the dovecot server.
IMAP module Source-code Analysis:
‘enum states { IMAP_INIT, IMAP_USER } ‘
There are only two states that are needed in the IMAP module.
static int
imap_loop_read(nsock_pool nsp, Connection *con)
{
if ((con->inbuf == NULL) || !(memsearch((const char *)con->inbuf- >get_dataptr(),"\r\n",con->inbuf->get_len()))) {
nsock_read(nsp, con->niod, ncrack_read_handler, IMAP_TIMEOUT, con); return -1;
}
if (memsearch((const char *)con->inbuf->get_dataptr(),"NO",con->inbuf- >get_len())) return 1;
return 0;
}
In this part of the code, the module tries to detect the ‘\r\n’(specific for the IMAP packets) pattern in order to verify that the whole server reply has arrived successfully.
‘void ncrack_imap(nsock_pool nsp, Connection *con)
{
int ret;
41 | P a g e nsock_iod nsi = con→niod;
switch(con→state)
{
case IMAP_INIT:
if (!con→login_attempts) {
if ((imap_loop_read(nsp, con)) < 0) { break;
}
}’
The module starts with the IMAP_INIT state in which it tests if there are any login attempts that have taken place in this connection. If this is the first login attempt in the connection, imap_loop_read is called in order to verify that the server has sent the initial status packet. If the packet is accepted (or more than one login attempts took place before), the module proceeds to the next step.
‘if (con→outbuf)
delete con→outbuf; con→outbuf = new Buf(); con→outbuf→snprintf(12 + strlen(con→user) + strlen(con→pass), “01 LOGIN %s %s\r\n”, con→user, con→pass);
nsock_write(nsp, nsi, ncrack_write_handler, IMAP_TIMEOUT, con, (const char *)con→outbuf→get_dataptr(), con→outbuf→get_len()); break;’
In this phase Ncrack sends a packet with length of [12+username+password] characters using the specific syntax of
“01 LOGIN username password\r\n” which is imap-specific.
‘case IMAP_USER:
if ((ret = imap_loop_read(nsp, con)) < 0) break;
42 | P a g e if ((ret ==0)
con→auth_success = true;
con→state = IMAP_INIT;
return ncrack_module_end(nsp, con);
’
In the last part, if the return value of imap_loop_read) is not 0, this means that a) either the packet hasn't been received in full yet (ret = -1) or b) the credentials weren't correct (ret = 1). If ret == 0, it means that Ncrack found a valid password for the given username and proceeds with testing the remaining credentials.
43 | P a g e
44 | P a g e
45 | P a g e
46 | P a g e CASSANDRA MODULE DEVELOPMENT
Apache Cassandra
Cassandra is another open source database management system. It is popular for storing and managing huge amounts of structured data that is distributed around the world. It provides highly available service with no single point of failure.
Listed below there are some notable things about Cassandra:
• It is scalable, fault-tolerant, and consistent. • It is a column-oriented database. • Its distribution design is based on Amazon’s Dynamo and its data model on Google’s Bigtable. • Created at Facebook, it differs sharply from relational database management systems. • Cassandra implements a Dynamo-style replication model with no single point of failure, but adds a more powerful “column family” data model. • Cassandra is being used by some of the biggest companies such as Facebook, Twitter, Cisco, Rackspace, e-Bay, Twitter, Netflix, and more.
47 | P a g e NoSQLDatabase
A NoSQL database is a database that offers the ability to store and manage data using different kind of relations (whereas in the relational databases, only tables are used to store data). These kinds of databases are easy to manage, consistent, can use any form the user chooses and can also save large amount of data.
A NoSQL database aims to provide
• simple and efficient design, • scalability and ease of management, and • availability and effectiveness NoSql databases utilize different data structures in comparison to relational databases. The schema of a NoSQL database tends to adapt with the requirements. NoSQL vs. Relational Database The main differences between a NoSQL and a relational database are presented below.
Relational Database NoSQL Database
Supports powerful query language. Supports very simple query language.
It has a fixed schema. No fixed schema.
Follows ACID (Atomicity, Consistency, Isolation, and It is only “eventually consistent”.
Durability).
Supports transactions. Does not support transactions.
Features of Cassandra
Cassandra is getting more and more popular because of its multiple features. Some of them are the following:
• Elastic scalability - Cassandra supports efficient scalability. It is possible to add additional hardware to serve additional users. • Durable architecture - Cassandra has no single point of failure and it stays endlessly available for critical applications avoiding any possible failure. • Linear increase in performance - Cassandra is scalable is a way that when someone adds a node in the cluster, it automatically increases the throughput promptly.
48 | P a g e • Flexible data storage – Cassandra stores data in different kinds of schemes. This includes, structured, semi- structured or unstructured shapes of databases. • Easy data distribution – Cassandra offers the ability to copy data across the data servers allowing the user to access it from literally everywhere. • Transaction support - Cassandra supports Atomicity, Consistency, Isolation, and Durability due to its ‘No single point of failure’. • Fast writes - Cassandra was designed to run in low-performance hardware and thus it can achieve extreme speeds when writing and reading.
Installing Cassandra
Prerequisites
• Cassandra needs a version of Java, either OpenJDK 7 or 8. To check what version, the system supports, the user needs to use the ‘java -version’ command. • Cassandra also needs Python 2.7 to run the Cassandra query language shell (which will be analyzed later). The command which verifies that the correct Python version is running, is ‘python –version’. Installation procedure • Firstly, we download the latest stable release from the official Apache Cassandra website. • We untar the file in a target directory: tar -xvf apache-cassandra-3.6-bin.tar.gz /home/cassandra • We add apache-cassandra/bin to our path. • In order to start Cassandra, we use the bin/cassandra -f command from the terminal. In order to stop the server, we can ‘CTRL + C’. We can also stop Cassandra by using ‘kill pid’ where pid is the process id of the Cassandra service or pkill -f CassandraDaemon. • In order to check the status of our Cassandra server we can use ‘bin/nodetool status’. • Log and data directories reside in the corresponding logs and data sub-directories.
Configuration
The Cassandra configuration files are located in the conf directory. According to packages, the configuration files will
reside in /etc/cassandra.
49 | P a g e Main runtime properties
Most of configuration options in Cassandra are included inside the cassandra.yaml. Some of the most important
settings are the following:
• cluster_name: the name of the server’s cluster. • seeds: a separated list of the IP addresses of the corresponding cluster seeds. • storage_port: I had to be sure that there are no firewalls blocking this port • listen_address: This essentially is the ip of our node. This provides the ability for other nodes to communicate with this one. I did change that to listen_interface in order to inform Cassandra which interface to use. • native_transport_port: This is the port that allows intermediate communication between Cassandra’s clients. It is important for this port to be open and properly configured. • Thrift Interface: In order to connect remotely with Cassandra I needed to make sure that I enabled the thrift interface which listens port.
Changing the location of directories
The following yaml properties control the location of directories:
CQLSH
Starting cqlsh In order to start the Cassandra Query Language shell, we use the following commands:
cqlsh [options] [host [port]]
python cqlsh.py [options] [host [port]]
50 | P a g e Description
CQL shell is included in the official Cassandra installation package. CQL shell is a python-based command line that executes commands for the client. In order to use the CQL shell on Linux or Windows we just use the command
‘cqlsh’.
Using CQL commands
Upon initialization, cqlsh presents the name of the cluster, its IP address, and the port used for connection to the cqlsh utility. The cqlsh prompt is cqlsh>. For example, a typical cqlsh startup includes the following:
51 | P a g e
Notice that the 9042-port used, means that the connection is local and not a remote one. Our module will target the remote authentication utilizing the Cassandra service.
At the cqlsh prompt, one can type CQL commands. In order to terminate a command the developer can use a semicolon. For example: cqlsh> USE demo_cl; cqlsh:demo_cl> SELECT * FROM demo_table
... WHERE id = 0;
When a command is executed successfully, the results are presented as a standard output.
Saving CQL output in a file
This is a useful ability to have as by using the -e option cqlsh saves a CQL statement in a single file. For example, to save the output of a SELECT statement to output.txt: cqlsh -e "SELECT * FROM mytable" > output.txt
Using files as input
In the same way, we execute CQL commands inside a file while using the -f option as well as the path to the file on the corresponding command line.
The cqlsh environment variables
The default cqlsh host and listen port can be overridden by setting a CQLSH_HOST and CQLSH_PORT environment variables. We set the CQLSH_HOST to our IP address. After that, cqlsh will use these variables instead of the default values of localhost and port 9042. The port for the remote authentication is 9160 and it is the one we are going to use.
52 | P a g e CQLSH Authentication
This is how cqlsh responds when the user has successfully achieved an authentication attempt. As we see, the username is ‘cassandra’ while the password was also ‘cassandra’.
The ip of the target Cassandra server is 192.168.1.4 in this example. As we can see, the port for the remote authentication is 9160. This is a specific feature of the Cassandra service. Cassandra’s client API is listening to 9160 by default and that’s the port, our module will be developed to utilize.
53 | P a g e CQLSH failed authentication
As we can clearly see, in this example, the user tries to connect to the Cassandra server without completing the authentication step with success. Cassandra returned the following response
“AuthenticationException(why=’Username and/or password are incorrect’)
Essentially cqlsh sent an authentication request to the Cassandra server using the Thrift port (9160). The API checked the username-password pair list and didn’t find a match.
54 | P a g e Cassandra Wireshark authentication Packet
Using the tcp.port == 9160 filter to focus on the packets that we are interested in, we get the results depicted above.
This is the structure of the authentication packet or the packet that the Cqlsh (remote computer) sent to our Cassandra server asking for authentication.
55 | P a g e Cassandra Server Reply - Positive
This is the reply of the server when the authentication is approved. The data of the packet, in contrary to the reply (corresponding packet) of the failed authentication attempt (see below) does not contain any ‘String:’, that informs the user about the status of the connection.
This is not needed though because the user is taken to the cqlsh command line immediately.
56 | P a g e Cassandra Server Reply - Negative
This is the packet which informs the user that the server does not approve the authentication. In this case, the server informs the user with a message “Username and/or password are incorrect”
57 | P a g e Cassandra Packet Analysis
During the development, Ncrack while using the Cassandra module should be able to send the same packets Cqlsh sends for the authentication. The reason behind this, is that Ncrack does not essentially use the Cassandra query language shell (cqlsh) to communicate with the Cassandra server. Instead it ‘borrows’ the same packets and tricks the server. This allows Ncrack to verify which are the valid credentials. The packet for the authentication process is the following:
In order to be able to send the packet through the wire, it is important to organize that. The best way to do it, is using structs. So, we get:
So, now that we have our format, we need to create a new format, which is the one that will be passed to send().
Here are some conversion functions that we might find handy. Each of these converts your host format structures to a network format buffer.
The important point is that we have a proper conversion function that writes all of the data, at specified offsets, in a well-defined byte order, to a buffer, and then we pass that buffer to a send() function.
58 | P a g e Cassandra module Source-code Analysis
“
#include "ncrack.h"
#include "nsock.h"
#include "NcrackOps.h"
#include "Service.h"
#include "modules.h"
#define CASS_TIMEOUT 20000 //here
extern NcrackOps o;
extern void ncrack_read_handler(nsock_pool nsp, nsock_event nse, void *mydata);
extern void ncrack_write_handler(nsock_pool nsp, nsock_event nse, void
*mydata); extern void ncrack_module_end(nsock_pool nsp, void *mydata);
static int cass_loop_read(nsock_pool nsp, Connection *con);
static void cass_encode_CALL(Connection *con);
static void cass_encode_data(Connection *con);
“
In this part, we declare the functions that are going to be used to the Cassandra module.
“ enum states { CASS_INIT, CASS_USER };
“
These are the two states that are needed in the Cassandra module.
“ static int
cass_loop_read(nsock_pool nsp, Connection *con)
{
59 | P a g e if ((con->inbuf == NULL) || !(memsearch((const char *)con->inbuf->get_dataptr(),"\r\n",con->inbuf->get_len()))) { nsock_read(nsp, con->niod, ncrack_read_handler, CASS_TIMEOUT, con); return -1; printf("step1");
}
if (memsearch((const char *)con->inbuf->get_dataptr(),"Username and/or password are incorrect",con- >inbuf->get_len())) return 1; printf("step2");
return 0;
} “
In this part of the code, the module tries to detect the ‘\r\n’ (specific for the cassandra packets) pattern in order to verify that the whole server reply has arrived successfully.
After that we need to organize our data. In order to achieve that, we use structs to create a “host format” of the data we want to send.
“ typedef struct cass_CALL
{ uint16_t version[1]; uint8_t zero; uint8_t call_id; uint8_t length[4]; uint16_t sequence_id[2];
};
typedef struct cass_data {
60 | P a g e uint16_t t_struct; uint16_t field_id; uint8_t t_stop; struct { uint8_t t_map; uint16_t field_id; uint8_t t_stop; struct { int t_utf71; int t_utf72; uint16_t nomitems; uint8_t length1[4]; u_char string1[8]; uint8_t length2[4]; uint8_t length3[4]; u_char string3[8]; uint8_t length4[4];
} map; }
Struct;
};
“
After that, we need a different format (network format) in order to make our data capable of transmitted through the wire.
61 | P a g e “ static void cass_encode_CALL(Connection *con) { cass_CALL call;
call.version[0] = 0x0180; //2byte con->outbuf-
>append(&call.version, sizeof(call.version)); call.zero =
0; con->outbuf->append(&call.zero, sizeof(call.zero)); call.call_id = 1; con->outbuf->append(&call.call_id, sizeof(call.call_id)); call.length[0] = 0; call.length[1] = 0; call.length[2] = 0; call.length[3] = 5; con->outbuf->append(&call.length, sizeof(call.length)); con->outbuf->snprintf(5, "login");
//strncpy((char* )&call.method[0], "login", 5); call.sequence_id[0]=0; call.sequence_id[1]=0; call.sequence_id[2]=0; call.sequence_id[3]=0; con->outbuf->append(&call.sequence_id, sizeof(call.sequence_id));
}
static void cass_encode_data(Connection *con) {
62 | P a g e cass_data data;
data.t_struct = 12; //T_STRUCT (12)=1byte con->outbuf-
>append(&data.t_struct, sizeof(data.t_struct)); data.field_id = 1; //
Field Id: 1 =2byte con->outbuf->append(&data.field_id, sizeof(data.field_id)); data.Struct.t_map = 13; // T_MAP (13) =1byte con->outbuf->append(&data.Struct.t_map, sizeof(data.Struct.t_map)); data.Struct.field_id = 1;
con->outbuf->append(&data.Struct.field_id, sizeof(data.Struct.field_id)); data.Struct.map.t_utf71 = 11; con->outbuf->append(&data.Struct.map.t_utf71, sizeof(data.Struct.map.t_utf71)); data.Struct.map.t_utf72 = 11; con->outbuf->append(&data.Struct.map.t_utf72, sizeof(data.Struct.map.t_utf72));
data.Struct.map.nomitems = 2; con->outbuf->append(&data.Struct.map.nomitems, sizeof(data.Struct.map.nomitems)); data.Struct.map.length1[0] = 0; //4byte data.Struct.map.length1[1] = 0; data.Struct.map.length1[2] = 0; data.Struct.map.length1[3] = strlen("username"); con->outbuf->append(&data.Struct.map.length1, sizeof(data.Struct.map.length1)); strncpy((char * )&data.Struct.map.string1[0],"username",8); con->outbuf-
>append(&data.Struct.map.string1, sizeof(data.Struct.map.string1)); data.Struct.map.length2[0] = 0; data.Struct.map.length2[1] = 0; data.Struct.map.length2[2] = 0;
63 | P a g e data.Struct.map.length2[3] = strlen(con->user); con->outbuf-
>append(&data.Struct.map.length2, sizeof(data.Struct.map.length2)); con-
>outbuf->snprintf(strlen(con->user), "%s", con->user); data.Struct.map.length3[0] = 0; //4byte data.Struct.map.length3[1] = 0; data.Struct.map.length3[2] = 0; data.Struct.map.length3[3] = strlen("password"); con->outbuf->append(&data.Struct.map.length3, sizeof(data.Struct.map.length3)); strncpy((char * )&data.Struct.map.string3[0],"password",8); con->outbuf-
>append(&data.Struct.map.string3, sizeof(data.Struct.map.string3)); data.Struct.map.length4[0] = 0; //4byte data.Struct.map.length4[1] = 0; //4byte data.Struct.map.length4[2] = 0; //4byte data.Struct.map.length4[3] = strlen(con->pass); //4byte con->outbuf->append(&data.Struct.map.length4, sizeof(data.Struct.map.length4)); con->outbuf->snprintf(strlen(con->pass), "%s", con->pass); data.Struct.t_stop = 0; //1byte con->outbuf-
>append(&data.Struct.t_stop, sizeof(data.Struct.t_stop)); data.t_stop
= 0; con->outbuf->append(&data.t_stop, sizeof(data.t_stop));
//con->outbuf->append(&data, sizeof(cass_data));
} “
In the next part, the module begins from the first State[CASS_INIT] in which it sends a packet that contains the packets that were developed above. These packets contain the corresponding username and password for this specific attempt.
64 | P a g e “ void ncrack_cassandra(nsock_pool nsp, Connection *con)
{ int ret; nsock_iod nsi = con->niod;
switch(con->state)
{ case CASS_INIT: con- >state = CASS_USER; if
(con->outbuf) delete con->outbuf; con-
>outbuf = new Buf(); cass_encode_CALL(con); cass_encode_data(con); nsock_write(nsp, nsi, ncrack_write_handler, CASS_TIMEOUT, con, (const char *)con->outbuf->get_dataptr(), con- >outbuf->get_len()); break;
“
In the last part, if the return value of cass_loop_read is not 0, this means that a) either the packet hasn't been received in full yet (ret = -1) or b) the credentials weren't correct (ret = 1). If ret == 0, it means that Ncrack found a valid password for the given username and proceeds with testing the remaining credentials.
“
65 | P a g e case CASS_USER:
if ((ret = cass_loop_read(nsp, con)) <
0) break;
if (ret == 0) con-
>auth_success = true;
con->state = CASS_INIT;
return ncrack_module_end(nsp, con);
}
} “
MODULE FINAL TOUCHES
4.3 - Final Steps After the completion of the development of the module, it is essential to follow a number of steps in order to include it in Ncrack’s framework. a) modules.h inclusion ncrack/modules/modules.h must include the declaration of the main module function whose naming format follows the convention of ncrack_ like it is depicted below. void ncrack_imap(nsock_pool nsp, Connection *con); void ncrack_cassandra(nsock_pool nsp, Connection *con);
b) Proper source code split-up
Everything that has to do with the modules should reside in a file under ncrack/modules/ whose name is ncrack_imap.cc and ncrack_cassandra.cc respectively. Every function that was used for the module but was too generic, got placed in the ncrack/utils.cc or in the ncrack/crypto.cc in case they were of crypto related nature. c) ncrack-services
66 | P a g e The file ncrack/ncrack-services contains all the protocols that are currently supported by Ncrack along with their default port numbers. The new modules need to be added as well as shown below:
So, we have: imap 143/tcp cassandra 9160/tcp
Note the case of SSL-supporting protocols there is an ‘s’ added in the end of the name (‘imaps’ for example) which is not the case in these two.
d) ncrack.cc::call_module
The call_module function inside ncrack/ncrack.cc must have an additional 'else-if' case which essentially calls the specific module. For this instance, it is written as:
if (!strcmp(name, "imap")) ncrack_imap(nsp, con);
else if (!strcmp(name, "cassandra")) ncrack_cassandra(nsp, con);
e) configure.ac
Finally, the ncrack/configure.ac should include the new modules at the MODULES_SRCS and MODULES_OBJS statements:
MODULES_SRCS="$MODULES_SRCS ncrack_imap.cc ncrack_cassandra.cc"
MODULES_OBJS="$MODULES_OBJS ncrack_imap.o ncrack_cassandra"
After these changes the autoconf utility need to run to apply the changes in “configure.ac”. The steps are the following
You can now test your new module by recompiling Ncrack using:
$ autoconf
67 | P a g e $ ./configure
$ make
Merging of the two new Modules
When a module is finished, it has to be merged with Ncrack’s main repository. The official Ncrack’s repository is https://github.com/nmap/ncrack.
Below we see how the official git repository looks like, as well as the repository which was used for the IMAP’s and Cassandra’s development.
Official Ncrack Git Repository
68 | P a g e
My personal Ncrack Git Repository
69 | P a g e
In the modules folder reside the two modules under the name of
• ncrack_imap.cc • ncrack_cassandra.cc
70 | P a g e Email to Nmap’s developers Team
In order for the merging to take place the developer needs to send an email to Nmap’s developers team. The email should clarify the subject (bug report, module creation, etcetera) and also to contain the necessary information about what module is created. Below we see the emails that took place between me and Nmap’s team in response to the creation of the IMAP module.
71 | P a g e General Ncrack Scan
A common Ncrack scan looks like this. For someone to execute an attack like this, he needs to know the basic commands of Ncrack. The commands that are going to be used for the development of the modules are the following: -u
--user
-P
-p
CL: Maximum number of connections that are going to be initiated for this test
-d[level]: Set a debugging level in order to be able to check the result of each attempt as well as the general behavior of the module.
72 | P a g e Ncrack Scan using the IMAP module
This is the process of a successful Ncrack scan, using the IMAP module. As we can see, Ncrack tests the credentials given (in this case, username ‘barrend’ paired with a number of passwords individually), targeting the 192.168.1.4 IP while utilizing the IMAP protocol.
Ncrack completed 9 attempts, one for each password, and in the end, prints the results to the user. It also reports the time that it needed for the attack to complete.
73 | P a g e An Ncrack attack using the IMAP protocol looks like this
In this picture, the steps of the authentication process are depicted in detail. At the start, Ncrack establishes a connection with the target machine. After that, Ncrack starts to test credentials. One can tell by observing at the
‘Response’ packets, what the Ncrack itself will return to the user about the specific try.
Ncrack debugging option offers some insight about an attack, but Wireshark is much more thorough when it comes to the timing of the packets, or some additional information about responses of the server, which are crucial to analyze for the development of the module.
74 | P a g e Ncrack Scan using the Cassandra module
When the module is used, it is clearly shown that it scans a password list and returns the valid pairs back. In this case
‘cassandra-cassandra’
When we capture the packets, we clearly the cohesion of the ‘CALL’ – ‘REPLY’ packets during the authentication process. Below we evaluate the packets and see that they are constructed appropriately.
75 | P a g e An Ncrack attack using the Cassandra service looks like this
76 | P a g e Even though the parser did not manage to provide us with the additional fields that already exist in the official package of the CQL shell, if we examine the packet carefully (at the bottom part of wireshark), we can see that the data we need is here. The trigger for the packet to be completed is a small change in the structure or the host network.
Conclusions
Even though it was my first time attempting to achieve something like this, I have to admit that a dissertation of this caliber gave me the opportunity to enhance my knowledge over the networks, to come in touch with a great number of security tools and also to come in contact with true professionals of the field.
I think that by achieving to complete at this degree a project like this, I have faced real problems and challenges. I have managed to study and learn a lot including network protocols, operating systems, computer security, programming in C/C++, specific tools and others.
I would like to offer my gratitude to my Professor Apostolos Papadopoulos for accepting to supervise me. My Professor Marios Gkatzianas who was the one that made me realize what I really wanted and pointed me to the right direction and the Security Researcher Fotis Chantzis who helped in every part of the process and withstood the endless questions that I asked by always helping me understand how to think and how to approach a problem in order to find a solution.
77 | P a g e Bibliography
Learn C the Hard Way by Zed Shaw
st TCP/IP Illustrated: The Implementation, Vol.2 1 Edition by Gary R. Wright (author), W. Richard Stevens http://sock-raw.org/
https://nmap.org/
https://stackoverflow.com/
https://www.wireshark.org/
https://www.sourceware.org/gdb/
https://nmap.org/ncat/
https://www.archlinux.org/
https://www.offensive-security.com/
http://netcat.sourceforge.net/
http://whatismyipaddress.com/imap
https://www.dovecot.org/
https://cassandra.apache.org/
https://bbs.archlinux.org/
78 | P a g e