Air Force Institute of Technology AFIT Scholar Theses and Dissertations Student Graduate Works 3-22-2012 Analysis of the Impact of Data Normalization on Cyber Event Correlation Query Performance Smile T. Ludovice Follow this and additional works at: https://scholar.afit.edu/etd Part of the Computer Sciences Commons, and the Systems Engineering Commons Recommended Citation Ludovice, Smile T., "Analysis of the Impact of Data Normalization on Cyber Event Correlation Query Performance" (2012). Theses and Dissertations. 1275. https://scholar.afit.edu/etd/1275 This Thesis is brought to you for free and open access by the Student Graduate Works at AFIT Scholar. It has been accepted for inclusion in Theses and Dissertations by an authorized administrator of AFIT Scholar. For more information, please contact [email protected]. ANALYSIS OF THE IMPACT OF DATA NORMALIZATION ON CYBER EVENT CORRELATION QUERY PERFORMANCE THESIS Smile T. Ludovice, Master Sergeant, USAF AFIT/GIR/ENV/12-M03 DEPARTMENT OF THE AIR FORCE AIR UNIVERSITY AIR FORCE INSTITUTE OF TECHNOLOGY Wright-Patterson Air Force Base, Ohio APPROVED FOR PUBLIC RELEASE; DISTRIBUTION UNLIMITED. The views expressed in this thesis are those of the author and do not reflect the official policy or position of the United States Air Force, Department of Defense, or the United States Government. This material is declared a work of the United States Government and is not subject to copyright protection in the United States. AFIT/GIR/ENV/12-M03 ANALYSIS OF THE IMPACT OF DATA NORMALIZATION ON CYBER EVENT CORRELATION QUERY PERFORMANCE THESIS Presented to the Faculty Department of Systems and Engineering Management Graduate School of Engineering and Management Air Force Institute of Technology Air University Air Education and Training Command In Partial Fulfillment of the Requirements for the Degree of Master of Science in Information Resource Management Smile T. Ludovice, BS Master Sergeant, USAF March 2012 APPROVED FOR PUBLIC RELEASE; DISTRIBUTION UNLIMITED. AFIT/GIR/ENV/12-M03 ANALYSIS OF THE IMPACT OF DATA NORMALIZATION ON CYBER EVENT CORRELATION QUERY PERFORMANCE Smile T. Ludovice, BS Master Sergeant, USAF Approved: S I G N E D -- 8 Mar 12 __________________________________________ ______________ Michael R. Grimaila, PhD, CISM, CISSP (Chairman) Date S I G N E D -- 8 Mar 12 __________________________________________ ______________ Robert F. Mills, PhD (Member) Date S I G N E D -- 8 Mar 12 __________________________________________ ______________ Brent T. Langhals, LtCol, PhD (Member) Date AFIT/GIR/ENV/12-M03 Abstract A critical capability required in the operation of cyberspace is the ability to maintain situational awareness of the status of the infrastructure elements that comprise cyberspace. Event logs from cyber devices can yield significant information, and when properly utilized can provide timely situational awareness about the state of the cyber infrastructure. In addition, proper Information Assurance requires the validation and verification of the integrity of results generated by a commercial log analysis tool. Event log analysis can be performed using relational databases. To enhance database query performance, previous literatures affirm denormalization of databases; yet, database normalization can also increase query performance. Database normalization improved majority of the queries performed using very large data sets of router events; however, performance is also dependent on the type of query executed. Queries performed faster on normalized table if all the necessary data are contained in the normalized tables. Furthermore, database normalization improves table organization and maintains better data consistency than non-normalized. Nonetheless, there are some tradeoffs when normalizing a database such as additional preprocessing time and extra storage requirements though minimal in this experiment. Overall, normalization improved query performance and must be considered as an option when analyzing event logs using relational databases. iv Acknowledgments I would like to say many thanks to my thesis advisor, Dr. Michael Grimaila, for his unwavering support, mentorship, and patience throughout many months school and research effort. I would also like to thank Dr. Robert Mills and LtCol Brent Langhals for their guidance and support. I would also like to send my appreciation to the Information Resource Management program faculty and staff. Additionally, I want to acknowledge my fellow students and classmates for their fellowship and camaraderie. Thanks for the support from my friends and family. Most importantly, I would like to say thank you to my wife for her love, support, and understanding all throughout this process. She gave me all the strength and encouragement to successfully graduate while taking care of our newborn son. It has been an awesome and unforgettable experience. Smile T. Ludovice v Table of Contents Page Abstract .......................................................................................................................................... iv Acknowledgments............................................................................................................................v Table of Contents ........................................................................................................................... vi List of Figures ................................................................................................................................ ix List of Tables ...................................................................................................................................x List of Equations ........................................................................................................................... xii I. Introduction .................................................................................................................................1 1.1 Research Motivation ..............................................................................................................1 1.2 Problem Statement .................................................................................................................2 1.3 Research Goals ......................................................................................................................3 1.4 Research Questions ................................................................................................................3 1.5 Scope, Assumptions, and Limitations ...................................................................................4 1.6 Thesis Overview ....................................................................................................................4 II. Literature Review .......................................................................................................................5 2.1 Chapter Overview ..................................................................................................................5 2.2 Event Logs .............................................................................................................................5 2.2.1 Types of Event Logs and Event Entries ......................................................................... 6 2.3 Current Standardization Effort ..............................................................................................7 2.3.1 Syslog Protocol .............................................................................................................. 7 2.3.2 Common Event Expression (CEE) ................................................................................ 9 2.4 Purpose of Event Logs .........................................................................................................11 2.5 Log Analysis of Security Incidents and Complexity ...........................................................13 2.6 Enhancing Situational Awareness, Information Assurance, and Mission Assurance .........15 2.6.1 Situational Awareness .................................................................................................. 15 2.6.2 Information Assurance ................................................................................................. 16 2.6.3 Mission Assurance ....................................................................................................... 18 2.7 Learning from Event Log Data ............................................................................................21 2.8 Event Correlation .................................................................................................................22 2.9 Database Normalization.......................................................................................................23 2.9.1 Steps in Normalization ................................................................................................. 24 2.9.2 Linking Normalized Tables ......................................................................................... 26 2.10 Data Warehousing .............................................................................................................27 2.11 Summary ............................................................................................................................29 III. Methodology ...........................................................................................................................30 vi Page 3.1 Overview..............................................................................................................................30