Designing an effective network forensic framework for the investigation of botnets in the Internet of Things Nickolaos Koroniotis A thesis submitted in fulfilment of the requirements for the degree of Doctor of Philosophy School of Engineering and Information Technology The University of New South Wales Australia March 2020 COPYRIGHT STATEMENT ‘I hereby grant the University of New South Wales or its agents a non-exclusive licence to archive and to make available (including to members of the public) my thesis or dissertation in whole or part in the University libraries in all forms of media, now or here after known. I acknowledge that I retain all intellectual property rights which subsist in my thesis or dissertation, such as copyright and patent rights, subject to applicable law. I also retain the right to use all or part of my thesis or dissertation in future works (such as articles or books).’ ‘For any substantial portions of copyright material used in this thesis, written permission for use has been obtained, or the copyright material is removed from the final public version of the thesis.’ Signed ……………………………………………........................... Date …………………………………………….............................. AUTHENTICITY STATEMENT ‘I certify that the Library deposit digital copy is a direct equivalent of the final officially approved version of my thesis.’ Signed ……………………………………………........................... Date …………………………………………….............................. 1 Thesis Dissertation Sheet Surname/Family Name : Koroniotis Given Name/s : Nickolaos Abbreviation for degree : PhD as give in the University calendar Faculty : UNSW Canberra at ADFA School : UC Engineering & Info Tech Thesis Title : Designing an effective network forensic framework for the investigation of botnets in the Internet of Things 2 Abstract 350 words maximum: The emergence of the Internet of Things (IoT), has heralded a new attack surface, where attackers exploit the security weaknesses inherent in smart things. Comprised of heterogeneous technologies and protocols, the IoT is a source of high-speed and volume data, rendering pre-existing forensic solutions ineffective. As a result, developing new network forensic solutions for the IoT is imperative. Some of the challenges involved in designing network forensic solutions for the IoT are 1) obtaining realistic data that represent contemporary network behaviour, 2) selecting and optimizing a machine learning model, best suited to deal with such data and 3) identifying and tracing attacks. This thesis provides considerable contribution to the research focusing on building a network forensic framework tasked with investigating botnet activities in IoT networks. The first contribution is the design of a new virtual testbed and the generation of a new network dataset, called Bot-IoT. This new dataset incorporates normal IoT traffic and represents a range of realistic network attacks. The second contribution is the selection of optimal features for the dataset. The process combined two measures, namely Pearson Correlation and Joint Entropy to create a score for the features, allowing for the selection of the 10 least-similar, which helped in removing any redundant information from the dataset. The third contribution is the analysis performed on the Bot-IoT dataset. For this analysis, two other widely used dataset, the UNSW-NB15 and NSL-KDD datasets were selected and seven machine learning models were trained. The fourth contribution is the development of the Particle Deep Framework (PDF) which covers the stages of the digital forensic investigation process. The PDF utilizes Particle Swarm Optimization for the selection of the optimal hyperparameters of a deep learning model, which lies at its core and is trained to detect attack network flows. 3 Declaration relating to disposition of project thesis/dissertation I hereby grant to the University of New South Wales or its agents a non-exclusive licence to archive and to make available (including to members of the public) my thesis or dissertation in whole or in part in the University libraries in all forms of media, now or here after known. I acknowledge that I retain all intellectual property rights which subsist in my thesis or dissertation, such as copyright and patent rights, subject to applicable law. I also retain the right to use all or part of my thesis or dissertation in future works (such as articles or books). ................................................ ................................................ Signature Date The University recognises that there may be exceptional circumstances requiring restrictions on copying or conditions on use. Requests for restriction for a period of up to 2 years can be made when submitting the final copies of your thesis to the UNSW Library. Requests for a longer period of restriction may be considered in exceptional circumstances and require the approval of the Dean of Graduate Research. 4 Originality Statement I hereby declare that this submission is my own work and to the best of my knowledge and belief, it contains no material previously published or written by another person, nor material which to a substantial extent has been accepted for the award of any other degree or diploma at UNSW or any other educational institution, except where due acknowledgment is made in the thesis. Any contribution made to the research by colleagues, with whom I have worked at UNSW or elsewhere, during my candidature, is fully acknowledged. I also declare that the intellectual content of this thesis is the product of my own work, except to the extent that assistance from others in the project’s design and conception or in style, presentation and linguistic expression is acknowledged. .......................................... .................................................. Signature Date 5 Inclusion of Publicaitons Statement UNSW is supportive of candidates publishing their research results during their candi- dature as detailed in the UNSW Thesis Examination Procedure. Publications can be used in their thesis in lieu of a Chapter if: • The candidate contributed greater than 50% of the content in the publication and is the “primary author”, ie. the candidate was responsible primarily for the planning, execution and preparation of the work for publication. • The candidate has approval to include the publication in their thesis in lieu of a Chapter from their supervisor and Postgraduate Coordinator. • The publication is not subject to any obligations or contractual agreements with a third party that would constrain its inclusion in the thesis Please indicate whether this thesis contains published material or not: This thesis contains no publications, either published or submitted for publication. Some of the work described in this thesis has been published and it has been documented in the relevant Chapters with acknowledgement. This thesis has publications (either published or submitted for publication) incor- porated into it in lieu of a chapter and the details are presented below. 6 CANDIDATE’S DECLARATION I declare that: • I have complied with the UNSW Thesis Examination Procedure • where I have used a publication in lieu of a Chapter, the listed publication(s) below meet(s) the requirements to be included in the thesis. Candidate’s Name Signature Date (dd/mm/yy) POSTGRADUATE COORDINATOR’S DECLARATION I declare that: • the information below is accurate • where listed publication(s) have been used in lieu of Chapter(s), their use complies with the UNSW Thesis Examination Procedure • the minimum requirements for the format of the thesis have been met. PGC’s Name PGC’s Signature Date (dd/mm/yy) 7 Details of Publicaiton #1: Full title: Forensics and deep learning mechanisms for botnets in Internet of Things: A survey of challenges and solutions Authors: Koroniotis, Nickolaos and Moustafa, Nour and Sitnikova, Elena Journal or book name: IEEE Access Volume/page numbers: 7/61764–61785 Date accepted/published: 14 May 2019 Status Published Accepted In progress and in press (Submitted) The Candidate’s Contribution to the Work The candidate designed, implemented and is the primary author of this work. Location of the work in the thesis and/or how the work is incorporated in the thesis: The publication was used in lieu of Chapter 2, as the Literature Review. PRIMARY SUPERVISOR’S DECLARATION I declare that: • the information above is accurate • this has been discussed with the PGC and it is agreed that this publication can be included in this thesis in lieu of a Chapter • All of the co-authors of the publication have reviewed the above information and have agreed to its veracity by signing a ‘Co-Author Authorisation’ form. Primary Supervisor’s Primary Supervisor’s Date (dd/mm/yy) name signature 8 Details of Publicaiton #2: Full title: Towards the development of realistic botnet dataset in the internet of things for network forensic analytics: Bot-iot dataset Authors: Koroniotis, Nickolaos and Moustafa, Nour and Sitnikova, Elena and Turnbull, Benjamin Journal or book name: Future Generation Computer Systems Volume/page numbers: 100/779–796 Date accepted/published: 15 May 2019 Status Published Accepted In progress and in press (Submitted) The Candidate’s Contribution to the Work The candidate designed, implemented and is the primary author of this work. Location of the work in the thesis and/or how the work is incorporated in the thesis: The publication was used in lieu of Chapter 3, to describe the design, generation and refinement of the Bot-IoT dataset.