East Tennessee State University Digital Commons @ East Tennessee State University Electronic Theses and Dissertations Student Works 12-2002 Data Mining with Newton's Method. James Dale Cloyd East Tennessee State University Follow this and additional works at: https://dc.etsu.edu/etd Part of the Computer Sciences Commons Recommended Citation Cloyd, James Dale, "Data Mining with Newton's Method." (2002). Electronic Theses and Dissertations. Paper 714. https://dc.etsu.edu/ etd/714 This Thesis - Open Access is brought to you for free and open access by the Student Works at Digital Commons @ East Tennessee State University. It has been accepted for inclusion in Electronic Theses and Dissertations by an authorized administrator of Digital Commons @ East Tennessee State University. For more information, please contact [email protected]. Data Mining with Newton’s Method A thesis presented to the faculty of the Department of Computer and Information Sciences East Tennessee State University In partial fulfillment of the requirements for the degree Masters of Science in Computer Science by James Dale Cloyd December 2002 Donald Sanderson, Chair Laurie Moffit Kellie Price Keywords: Genetic algorithms, evolutionary operations, neural networks, simplex evop, maximum likelihood estimation, non-linear regression, robust regression, knowledge discovery in databases. ABSTRACT Data Mining with Newton’s Method by James D. Cloyd Capable and well-organized data mining algorithms are essential and fundamental to helpful, useful, and successful knowledge discovery in databases. We discuss several data mining algorithms including genetic algorithms (GAs). In addition, we propose a modified multivariate Newton’s method (NM) approach to data mining of technical data. Several strategies are employed to stabilize Newton’s method to pathological function behavior. NM is compared to GAs and to the simplex evolutionary operation algorithm (EVOP). We find that GAs, NM, and EVOP all perform efficiently for well-behaved global optimization functions with NM providing an exponential improvement in convergence rate. For local optimization problems, we find that GAs and EVOP do not provide the desired convergence rate, accuracy, or precision compared to NM for technical data. We find that GAs are favored for their simplicity while NM would be favored for its performance. 2 Copyright © 2002 by James D. Cloyd All rights reserved 3 ACKNOWLEDGMENTS The student is grateful to Dr. Terry Countermine and the faculty and staff of the Computer and Information Sciences Department of East Tennessee State University who took the time to schedule and teach evening classes for folks who work full time and would otherwise be unable to attend day classes. The financial support of Eastman Chemical Company through its employee tuition refund program is gratefully acknowledged. The support and assistance of the chair members, Drs. Laurie Moffit and Kellie Price, are greatly appreciated. The guidance, leadership, support, and assistance of the thesis chair, Dr. Donald Sanderson, are gratefully acknowledged. The student would like to acknowledge the late Professor Jerry Sayers whose teaching skills, scientific viewpoints, and leadership were very influential and he will be missed both as a friend and a teacher. 4 CONTENTS Page ABSTRACT..................................................................................................................... 002 ACKNOWLEDGEMENTS ............................................................................................. 004 LIST OF TABLES ........................................................................................................... 008 LIST OF FIGURES.......................................................................................................... 009 Chapter 1. INTRODUCTION........................................................................................... 012 2. DATA MINING LITERATURE SURVEY ................................................... 014 Data Mining and Knowledge Discovery.................................................. 015 Examples of Data Mining Applications................................................... 016 Data Mining Techniques .......................................................................... 018 Evolutionary Operations (EVOP) ................................................ 019 The Basic Simplex Algorithm.......................................... 019 The Variable Size Simplex Algorithm............................. 020 Initial Simplex Points...................................................... 022 Summary of Simplex EVOP Algorithm .......................... 024 Genetic Algorithms ...................................................................... 024 Newton's Method ......................................................................... 026 Support Vector Machine .............................................................. 028 Association Rule Mining.............................................................. 029 Neural Networks .......................................................................... 030 Other Techniques Used in Data Mining................................................... 035 Datasets for Experimentation................................................................... 036 Data Mining Literature Survey Summary................................................ 036 3. NEWTON'S METHOD .................................................................................. 038 Non-Linear Equation Iteration Algorithms.............................................. 039 Newton's Method in Higher Dimensions ................................................. 041 5 Chapter Page Convergence Behavior of the Newton-Raphson Algorithm .................... 042 Fixed Point Iteration..................................................................... 043 Quadratic Convergence of Newton's Method .............................. 044 Number of Iterations Required..................................................... 045 Conditions for Convergence of Newton's Method....................... 045 Example Iteration Functions .................................................................... 046 Mandlebrot Set ............................................................................. 046 Modified Mandlebrot Set ............................................................. 046 Square Root of 2........................................................................... 047 Euler's Formula: exp(pi) + 1 = 0 ................................................. 047 Fourth Roots of One..................................................................... 047 Verhulst Population Growth Model ............................................. 048 Krieger-Dougherty Equation........................................................ 048 Convergence Behavior for x in R: Quadratic Convergence.................... 048 Newton's Method in the Complex Plane.................................................. 049 Mandlebrot Set ............................................................................. 052 Square Root of 2........................................................................... 054 Euler's Equation ........................................................................... 056 One Fourth Roots of One ............................................................. 058 Modified Mandlebrot Function .................................................... 060 Krieger-Dougherty Equation........................................................ 062 Comparison of Convergence Behaviors................................................... 065 4. MODELING AND NEWTON'S METHOD .................................................. 066 Local Newton's Method ........................................................................... 066 Global Newton's Method.......................................................................... 067 Maximum Likelihood Estimation ............................................................ 068 Case 1: Linear Least Squares ...................................................... 069 Case 2: Linear Model with Uncertainty in Independent Variables.......................................... 070 6 Chapter Page Case 3: Linear Model with Non-Normal Residuals.................... 072 Case 4: Non-linear Robust Regression........................................ 073 Converting Local to Global Newton's Method ........................................ 074 Summary of Modeling with Newton's Method........................................ 075 5. MATRIX ALGEBRA ..................................................................................... 076 Gauss Elimination .................................................................................... 077 Gauss-Jordan Elimination ........................................................................ 079 LU Decomposition ................................................................................... 079 Cholesky Factorization............................................................................. 080 QR Factorization ...................................................................................... 080 Singular Value Decomposition(SVD)...................................................... 081 Newton's Method with SVD .................................................................... 081 Local Newton's Method by Maximum Likelihood Estimates.................. 084 6. RESULTS AND DISCUSSION ....................................................................