Phishing detection using machine learning MSc Research Project MSc Cyber Security Suraj Wandhare Student ID: 18157432 School of Computing National College of Ireland Supervisor: Ben Fletcher www.ncirl.ie National College of Ireland Project Submission Sheet School of Computing Student Name: Suraj Wandhare Student ID: 18157432 Programme: MSc Cyber Security Year: 2019 Module: MSc Research Project Supervisor: Ben Fletcher Submission Due Date: 03/02/2020 Project Title: Phishing detection using machine learning Word Count: 3500 Page Count: 11 I hereby certify that the information contained in this (my submission) is information pertaining to research I conducted for this project. All information other than my own contribution will be fully referenced and listed in the relevant bibliography section at the rear of the project. ALL internet material must be referenced in the bibliography section. Students are required to use the Referencing Standard specified in the report template. To use other author's written or electronic work is illegal (plagiarism) and may result in disciplinary action. I agree to an electronic copy of my thesis being made publicly available on NORMA the National College of Ireland's Institutional Repository for consultation. Signature: Date: 3rd February 2020 PLEASE READ THE FOLLOWING INSTRUCTIONS AND CHECKLIST: Attach a completed copy of this sheet to each project (including multiple copies). Attach a Moodle submission receipt of the online project submission, to each project (including multiple copies). You must ensure that you retain a HARD COPY of the project, both for your own reference and in case a project is lost or mislaid. It is not sufficient to keep a copy on computer. Assignments that are submitted to the Programme Coordinator office must be placed into the assignment box located outside the office. Office Use Only Signature: Date: Penalty Applied (if applicable): Phishing detection using machine learning Suraj Wandhare 18157432 Abstract Phishing attacks cause a loss of millions of dollars every year. It involves so- cial engineering which makes it much more effective. There are many proposed solutions to solve the problem of phishing using machine learning. This research is partly a study to compare different supervised machine learning algorithms to find the optimal algorithm for phishing detection using machine learning and partly to address the issue of URL shortening service exploitation. URL shortening is used on a daily basis to help reduce the size of the URL, but attackers use these ser- vices to hide the original URL of the phishing website. In this research we propose an application that takes in a URL, uses multiple feature extraction techniques to determine whether the URL is phishing or not. We propose a URL unshortener which will return the original URL, compliments the machine learning algorithm increasing its accuracy. The initial accuracy is 92.6% and the False Positive rate is 0.4% but when the URL unshortener is added the accuracy is increased to 96.6%. 1 Introduction Over the last few years, the Web has seen a massive growth in the number and kinds of web services that include social networking, forums, blogs, and video sharing sites [1]. The more ways people are able to interact, the more ways to exploit these services. One- way criminals exploit this is by using Phishing attacks. Phishing can be described as an attack in which the attacker disguises as a trustworthy person to obtain information such as login credentials and credit card information. It is a combination of social engineering attack which is used to deceive and manipulate someone to give away their data. Phishing websites are crafted such that they are very similar to the original website which it tries to mimic. It contains at least one login page which captures the confidential information. There are numerous ways to detect phishing sites, one of the most efficient way is to create a blacklist. PhishTank is an open source library which acts like a blacklist for phishing websites, controlled and operated by community members which ranks websites as phishing and not phishing based on votes. The important requirement for a blacklist is timely updates. Another efficient way to detect and prevent phishing is by combining previously built mechanisms to machine learning. Machine learning is the subset of artificial intelligence used by computer systems which uses algorithms, statistical models, patterns and inferences to complete a certain task without human intervention. This study aims to compare the available machine learning algorithms for detecting phishing websites and create a system that include detection and prevention of 'phish- ing using URL shortening services'. The literature review carried out for this research 1 revealed that there are numerous phishing detection techniques built upon machine learn- ing but a only few address the problem with the exploitation of URL shortening services. A URL shortening service is third party service which takes a long URL and converts it into a long case sensitive alphanumeric code. Some of the most popular URL shortening services are • Bit.ly • adf.ly • goo.gl • tinyurl.com This paper mainly focuses of different types of machine learning algorithms that can be used to detect phishing websites combined with the URL unshortening services. The dataset used for this research is taken from PhishTank.com and PhishLabs.com which are the two top reliable community based phish verification systems(more description is section 5). The research question proposed by this research is Can machine learning algorithms be used to prevent phishing by URL shortening techniques. Various algorithms and models have been researched and studied in the literature review to compare the already existing models and chosen considering the complexity of the problem. 2 Related Work Phishing detection and prevention is a major step towards protection of internet and cy- bersecurity. Most phishing incidents occur due to the inability of the user to differentiate between a real and phishing webpage. Machine learning is a way to prevent phishing from reaching the user. One way to achieve a good defence against phishing is if phishing website is blocked before the user accesses it. Phishing detection using machine learning is mainly classified into three categories. 2.1 Blacklisting URLs and IP addresses A phishing blacklist is a list of phishing URLs which have been reported by security experts or community members as harmful or dangerous. PhishTank is a open source community which contains a large database of websites which can be used as a blacklist. Google also provides an API called Google Safe Browsing API which comes built in with the Google Chrome browser. In this research the PhishTank database is used. The study [2] demonstrates how a framework can be developed to create a blacklist for phishing websites. Authors propose a framework called SEAHound which works in four steps to differentiate between a normal and a phishing website. The first feature checks if the website has any bad commands or malicious commands. Second it checks if the webpage contains any message that denote urgency, which is fairly common in phishing attacks. Third is generic greeting and the last is link analysis which uses third party API to check the validity of the URL. The study promises a secure framework but as the use of scripts and frameworks to develop websites increases it becomes more difficult to analyse the HTML content of the webpage. 2 Another study [3] shows how machine learning based on multidimensional features driven by deep learning can help prevent phishing attacks. The author makes a very good argument that 47%-83% of phishing websites are added to blacklists after 12 hours, and 63% of phishing websites have a lifespan of only 2 hours. This makes the blacklisting approach obsolete. The author proposes a model consisting of CNN-LSTM(Convolutional Neural Networks-Long Short-Term Memory), DCDA and multidimensional feature de- tector. CNN is used to extract local features and LSTM is used to extract context dependency. As a counterargument rather than discarding blacklisting approach if it is combined with the new machine learning model produces more efficient and secure system. Whitelisting URLs can also prevent phishing to a certain extent but as the internet increases and with new websites emerging everyday, it is not a feasible option to create a whitelist and update it regularly. This study [3] uses whitelist approach to create a browser extension that uses DNS servers to detect phishing and pharming attacks. 2.2 Domain name based phishing detection The Uniform Resource Locator, also called as a web address consists of different parts from which information about it can be extracted. Consider the url htts://www.example. com/students/?id=1234#page2 The URL mainly consists of: 1 • Protocol: Represents the technology being used to transfer the data. In this case https. • Domain: The registered domain name of the website, www.example.com • Path: The path associated with the page on the web server, /students/ • Hash: Shows the section on the page page2 • Query string: The data being transferred, ?id=1234 Machine learning can be used to extract features from a url. The study [4] demon- strates the feature extraction in depth. • Length of the URL: The number of characters in the domain string. High number of phishing attacks have a long URL. • Frequency of domain name: The number of times the domain name appears on the website. • Page title and domain name match. There are several more features that can be extracted from the URL to determine if the URL is used for phishing attacks. Authors in the study [5] propose phishstorm which is a phishing rating system which extracts URL features and ranks them on the basis of registered domain name. The author also suggests that URL obfuscation techniques are used to prevent phishing to be detected.