DOCSLIB.ORG

Automated Feature Engineering for Deep Neural Networks with Genetic Programming Jeff .T Heaton Nova Southeastern University, [email protected]

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Automated Feature Engineering for Deep Neural Networks with Genetic Programming Jeff .T Heaton Nova Southeastern University, Jeffheaton@Acm.Org Nova Southeastern University NSUWorks CEC Theses and Dissertations College of Engineering and Computing 2017 Automated Feature Engineering for Deep Neural Networks with Genetic Programming Jeff .T Heaton Nova Southeastern University, [email protected] This document is a product of extensive research conducted at the Nova Southeastern University College of Engineering and Computing. For more information on research and degree programs at the NSU College of Engineering and Computing, please click here. Follow this and additional works at: https://nsuworks.nova.edu/gscis_etd Part of the Computer Sciences Commons All rights reserved. This publication is intended for use solely by faculty, students, and staff of oN va Southeastern University. No part of this publication may be reproduced, distributed, or transmitted in any form or by any means, now known or later developed, including but not limited to photocopying, recording, or other electronic or mechanical methods, without the prior written permission of the author or the publisher. NSUWorks Citation Jeff .T Heaton. 2017. Automated Feature Engineering for Deep Neural Networks with Genetic Programming. Doctoral dissertation. Nova Southeastern University. Retrieved from NSUWorks, College of Engineering and Computing. (994) https://nsuworks.nova.edu/gscis_etd/994. This Dissertation is brought to you by the College of Engineering and Computing at NSUWorks. It has been accepted for inclusion in CEC Theses and Dissertations by an authorized administrator of NSUWorks. For more information, please contact [email protected]. Automated Feature Engineering for Deep Neural Networks with Genetic Programming by Jeff Heaton A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Computer Science College of Engineering and Computing Nova Southeastern University 2017 We hereby certify that this dissertation, submitted by Jeff Heaton, conforms to acceptable standards and is fully adequate in scope and quality to fulfill the dissertation requirements for the degree of Doctor of Philosophy. _____________________________________________ ________________ James D. Cannady, Ph.D. Date Chairperson of Dissertation Committee _____________________________________________ ________________ Sumitra Mukherjee, Ph.D. Date Dissertation Committee Member _____________________________________________ ________________ Paul Cerkez, Ph.D. Date Dissertation Committee Member Approved: _____________________________________________ ________________ Yong X. Tao, Ph.D., P.E., FASME Date Dean, College of Engineering and Computing College of Engineering and Computing Nova Southeastern University 2017 An Abstract of a Dissertation Submitted to Nova Southeastern University in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy Automated Feature Engineering for Deep Neural Networks with Genetic Programming by Jeff Heaton 2017 Feature engineering is a process that augments the feature vector of a machine learning model with calculated values that are designed to enhance the accuracy of a model’s predictions. Research has shown that the accuracy of models such as deep neural networks, support vector machines, and tree/forest-based algorithms sometimes benefit from feature engineering. Expressions that combine one or more of the original features usually create these engineered features. The choice of the exact structure of an engineered feature is dependent on the type of machine learning model in use. Previous research demonstrated that various model families benefit from different types of engineered feature. Random forests, gradient-boosting machines, or other tree-based models might not see the same accuracy gain that an engineered feature allowed neural networks, generalized linear models, or other dot-product based models to achieve on the same data set. This dissertation presents a genetic programming-based algorithm that automatically engineers features that increase the accuracy of deep neural networks for some data sets. For a genetic programming algorithm to be effective, it must prioritize the search space and efficiently evaluate what it finds. This dissertation algorithm faced a potential search space composed of all possible mathematical combinations of the original feature vector. Five experiments were designed to guide the search process to efficiently evolve good engineered features. The result of this dissertation is an automated feature engineering (AFE) algorithm that is computationally efficient, even though a neural network is used to evaluate each candidate feature. This approach gave the algorithm a greater opportunity to specifically target deep neural networks in its search for engineered features that improve accuracy. Finally, a sixth experiment empirically demonstrated the degree to which this algorithm improved the accuracy of neural networks on data sets augmented by the algorithm’s engineered features. Acknowledgements There are several people who supported and encouraged me through my dissertation journey at Nova Southeastern University. First, I would like to extend my thanks to Dr. James Cannady for serving as my dissertation committee chair. He provided guidance and advice at every stage of the dissertation process. He also gave me several additional research opportunities during my time at NSU. I would also like to thank my committee members Dr. Sumitra Mukherjee and Dr. Paul Cerkez. Their advice and guidance through this process were invaluable. My sincere gratitude and thanks to Tracy, my wife and graduate school partner. Her help editing this dissertation was invaluable. We took a multi-year graduate journey as she completed her Master’s degree in Spanish at Saint Louis University and I worked on my Ph.D. It was great that she could accompany me to Florida for each of the cluster meetings and I could join her in Madrid for summer classes. The GECCO conference that I attended in Madrid was instrumental in the development of this research. Her love and encouragement made this project all possible. Thank you to Dave Snell for encouraging me to pursue a Ph.D. and for guidance at key points in my career. I would like to thank my parents. From the time that they bought me my first Commodore 64 and through all my educational journeys, they have always supported me. Table of Contents Approval ii Acknowledgements iv List of Tables vii List of Figures viii 1. Introduction 1 Problem Statement 6 Dissertation Goal 7 Relevance and Significance 8 Barriers and Issues 9 Definitions of Terms 12 List of Acronyms 22 Summary 23 2. Literature Review 25 Feature Engineering 25 Neural Networks 30 Deep Learning 39 Evolutionary Programming 48 Summary 59 3. Methodology 60 Introduction 60 Algorithm Contract and Specification 61 Algorithm Design Scope 65 Experimental Design 68 Measures 68 Experiment 1: Limiting the Search Space 70 Experiment 2: Establishing Baseline 73 Experiment 3: Genetic Ensembles 74 Experiment 4: Population Analysis 76 Experiment 5: Objective Function Design 79 Experiment 6: Automated Feature Engineering 81 Real World Data sets 85 Synthetic Data sets 86 Resources 87 Summary 88 4. Results 90 Introduction 90 Experiment 1 Results 90 Experiment 2 Results 94 Experiment 3 Results 96 v Experiment 4 Results 97 Experiment 5 Results 101 AFE Algorithm Design 103 Experiment 6 Results 116 Summary 123 5. Conclusions, Implications, Recommendations, and Summary 124 Conclusions 124 Implications 125 Recommendations 126 Summary 129 References 182 Appendixes A. Algorithms 138 B. Hyperparameters 144 C. Third Party Libraries 152 D. Data Set Descriptions 156 E. Synthetic Data Set Generation 163 F. Engineered Features for the Data Sets 167 G. Detailed Experiment Results 173 H. Dissertation Source Code Availability 179 vi List of Tables Tables 1. Common neural network transfer functions 41 2. Experiment 1 results format, neural network and genetic programming 72 3. Experiment 2 result format, neural network baseline 74 4. Experiment 3-result format, neural network genetic program ensemble 76 5. Experiment 4 results format, patterns in genetic programs 79 6. Experiment 5 results format, evaluating feature ranking 80 7. Experiment 6 results format, engineered feature effectiveness 84 8. Experiment 1 neural network results for select expressions 92 9. Experiment 1 genetic program results for select expressions 93 10. Experiment 2 baselines for data sets (RMSE) 95 11. Experiment 3, Ensemble, GP & neural network results 97 12. Experiment 4, patterns from genetic programs 100 13. Experiment 6, feature engineered vs non-augmented accuracy 120 14. Experiment 6, T-Test with data sets that showed improvement 122 15. Data sets used by the dissertation experiments 156 vii List of Figures Figures 1. Regression and classification network (original features) 3 2. Neural network engineered features 5 3. Elman neural network 35 4. Jordan neural network 35 5. Long short-term memory (LSTM) 37 6. Dropout layer in a neural network 44 7. Expression tree for genetic programming 50 8. Point crossover 52 9. Subtree mutation 53 10. Feature engineering to linearly separate two classes 58 11. Algorithm high level design and contract 64 12. Ensemble of genetic programs for a neural network 75 13. Generate candidate solutions 77 14. Branches with common structures 78 15. High-level overview of AFE algorithm 82 16. Dissertation algorithm evaluation 83 17. Overview of AFE algorithm 105 18. Input neuron weight connections 109 19. Data flow/mapping for the AFE algorithm
Load more

Recommended publications
  • Incorporating Automated Feature Engineering Routines Into Automated Machine Learning Pipelines by Wesley Runnels
    Incorporating Automated Feature Engineering Routines into Automated Machine Learning Pipelines by Wesley Runnels Submitted to the Department of Electrical Engineering and Computer Science in partial fulfillment of the requirements for the degree of Master of Engineering in Electrical Engineering and Computer Science at the MASSACHUSETTS INSTITUTE OF TECHNOLOGY May 2020 c Massachusetts Institute of Technology 2020. All rights reserved. Author . Wesley Runnels Department of Electrical Engineering and Computer Science May 12th, 2020 Certified by . Tim Kraska Department of Electrical Engineering and Computer Science Thesis Supervisor Accepted by . Katrina LaCurts Chair, Master of Engineering Thesis Committee 2 Incorporating Automated Feature Engineering Routines into Automated Machine Learning Pipelines by Wesley Runnels Submitted to the Department of Electrical Engineering and Computer Science on May 12, 2020, in partial fulfillment of the requirements for the degree of Master of Engineering in Electrical Engineering and Computer Science Abstract Automating the construction of consistently high-performing machine learning pipelines has remained difficult for researchers, especially given the domain knowl- edge and expertise often necessary for achieving optimal performance on a given dataset. In particular, the task of feature engineering, a key step in achieving high performance for machine learning tasks, is still mostly performed manually by ex- perienced data scientists. In this thesis, building upon the results of prior work in this domain, we present a tool, rl feature eng, which automatically generates promising features for an arbitrary dataset. In particular, this tool is specifically adapted to the requirements of augmenting a more general auto-ML framework. We discuss the performance of this tool in a series of experiments highlighting the various options available for use, and finally discuss its performance when used in conjunction with Alpine Meadow, a general auto-ML package.
    [Show full text]
  • Learning from Few Subjects with Large Amounts of Voice Monitoring Data
    Learning from Few Subjects with Large Amounts of Voice Monitoring Data Jose Javier Gonzalez Ortiz John V. Guttag Robert E. Hillman Daryush D. Mehta Jarrad H. Van Stan Marzyeh Ghassemi Challenges of Many Medical Time Series • Few subjects and large amounts of data → Overfitting to subjects • No obvious mapping from signal to features → Feature engineering is labor intensive • Subject-level labels → In many cases, no good way of getting sample specific annotations 1 Jose Javier Gonzalez Ortiz Challenges of Many Medical Time Series • Few subjects and large amounts of data → Overfitting to subjects Unsupervised feature • No obvious mapping from signal to features extraction → Feature engineering is labor intensive • Subject-level labels Multiple → In many cases, no good way of getting sample Instance specific annotations Learning 2 Jose Javier Gonzalez Ortiz Learning Features • Segment signal into windows • Compute time-frequency representation • Unsupervised feature extraction Conv + BatchNorm + ReLU 128 x 64 Pooling Dense 128 x 64 Upsampling Sigmoid 3 Jose Javier Gonzalez Ortiz Classification Using Multiple Instance Learning • Logistic regression on learned features with subject labels RawRaw LogisticLogistic WaveformWaveform SpectrogramSpectrogram EncoderEncoder RegressionRegression PredictionPrediction •• %% Positive Positive Ours PerPer Window Window PerPer Subject Subject • Aggregate prediction using % positive windows per subject 4 Jose Javier Gonzalez Ortiz Application: Voice Monitoring Data • Voice disorders affect 7% of the US population • Data collected through neck placed accelerometer 1 week = ~4 billion samples ~100 5 Jose Javier Gonzalez Ortiz Results Previous work relied on expert designed features[1] AUC Accuracy Comparable performance Train 0.70 ± 0.05 0.71 ± 0.04 Expert LR without Test 0.68 ± 0.05 0.69 ± 0.04 task-specific feature Train 0.73 ± 0.06 0.72 ± 0.04 engineering! Ours Test 0.69 ± 0.07 0.70 ± 0.05 [1] Marzyeh Ghassemi et al.
    [Show full text]
  • Expert Feature-Engineering Vs. Deep Neural Networks: Which Is Better for Sensor-Free Affect Detection?
    Expert Feature-Engineering vs. Deep Neural Networks: Which is Better for Sensor-Free Affect Detection? Yang Jiang1, Nigel Bosch2, Ryan S. Baker3, Luc Paquette2, Jaclyn Ocumpaugh3, Juliana Ma. Alexandra L. Andres3, Allison L. Moore4, Gautam Biswas4 1 Teachers College, Columbia University, New York, NY, United States [email protected] 2 University of Illinois at Urbana-Champaign, Champaign, IL, United States {pnb, lpaq}@illinois.edu 3 University of Pennsylvania, Philadelphia, PA, United States {rybaker, ojaclyn}@upenn.edu [email protected] 4 Vanderbilt University, Nashville, TN, United States {allison.l.moore, gautam.biswas}@vanderbilt.edu Abstract. The past few years have seen a surge of interest in deep neural net- works. The wide application of deep learning in other domains such as image classification has driven considerable recent interest and efforts in applying these methods in educational domains. However, there is still limited research compar- ing the predictive power of the deep learning approach with the traditional feature engineering approach for common student modeling problems such as sensor- free affect detection. This paper aims to address this gap by presenting a thorough comparison of several deep neural network approaches with a traditional feature engineering approach in the context of affect and behavior modeling. We built detectors of student affective states and behaviors as middle school students learned science in an open-ended learning environment called Betty’s Brain, us- ing both approaches. Overall, we observed a tradeoff where the feature engineer- ing models were better when considering a single optimized threshold (for inter- vention), whereas the deep learning models were better when taking model con- fidence fully into account (for discovery with models analyses).
    [Show full text]
  • Identification of Activities of Daily Living Through Data Fusion on Motion and Magnetic Sensors Embedded on Mobile Devices
    Identification of Activities of Daily Living through Data Fusion on Motion and Magnetic Sensors embedded on Mobile Devices Ivan Miguel Pires1,2,3, Nuno M. Garcia1, Nuno Pombo1, Francisco Flórez-Revuelta4, Susanna Spinsante5 and Maria Canavarro Teixeira6,7 1Instituto de Telecomunicações, Universidade da Beira Interior, Covilhã, Portugal 2Altranportugal, Lisbon, Portugal 3ALLab - Assisted Living Computing and Telecommunications Laboratory, Computer Science Department, Universidade da Beira Interior, Covilhã, Portugal 4Department of Computer Technology, Universidad de Alicante, Spain 5Università Politecnica delle Marche, Ancona, Italy 6UTC de Recursos Naturais e Desenvolvimento Sustentável, Polytechnique Institute of Castelo Branco, Castelo Branco, Portugal 7CERNAS - Research Centre for Natural Resources, Environment and Society, Polytechnique Institute of Castelo Branco, Castelo Branco, Portugal [email protected], [email protected], [email protected], [email protected], [email protected], [email protected] Abstract Several types of sensors have been available in off-the-shelf mobile devices, including motion, magnetic, vision, acoustic, and location sensors. This paper focuses on the fusion of the data acquired from motion and magnetic sensors, i.e., accelerometer, gyroscope and magnetometer sensors, for the recognition of Activities of Daily Living (ADL). Based on pattern recognition techniques, the system developed in this study includes data acquisition, data processing, data fusion, and classification methods like Artificial Neural Networks (ANN). Multiple settings of the ANN were implemented and evaluated in which the best accuracy obtained, with Deep Neural Networks (DNN), was 89.51%. This novel approach applies L2 regularization and normalization techniques on the sensors’ data proved it suitability and reliability for the ADL recognition. Keywords: Mobile devices sensors; sensor data fusion; artificial neural networks; identification of activities of daily living.
    [Show full text]
  • Artificial Neural Networks
    ARTIFICIAL NEURAL NETWORKS: A REVIEW OF TRAINING TOOLS Darío Baptista, Fernando Morgado-Dias Madeira Interactive Technologies Institute and Centro de Competências de Ciências Exactas e da Engenharia, Universidade da Madeira Campus da Penteada, 9000-039 Funchal, Madeira, Portugal. Tel:+351 291-705150/1, Fax: +351 291-705199 Abstract: Artificial Neural Networks became a common solution for a wide variety of problems in many fields. The most frequent solution for its implementation consists of building and training the Artificial Neural Network within a computer. For implementing a network in an efficient way, the user can access a large choice of software solutions either commercial or prototypes. Choosing the most convenient solution for the application according to the network architecture, training algorithm, operating system and price can be a complex task. This paper helps the Artificial Neural Network user by providing a large list of solution available and explaining their characteristics and terms of use. The paper is confined to reporting the software products that have been developed for Artificial Neural Networks. The features considered important for this kind of software to have in order to accommodate its users effectively are specified. The development of software that implements Artificial Neural is a rapidly growing field driven by strong research interests as well as urgent practical, economical and social needs. Copyright CONTROLO2012 Keywords: Artificial Neural Networks, Training Tools, Training Algorithms, Software. 1. INTRODUCTION commercialization of new ANN tools. With the purpose to inform which tools are available at present Nowadays, in different areas, it is important to and make the choice of which tool to use, this paper analyse nonlinear data to do prediction, classification contains the description of software that has been or to build models.
    [Show full text]
  • Programming Neural Networks with Encog3 in Java
    Programming Neural Networks with Encog3 in Java Programming Neural Networks with Encog3 in Java Jeff Heaton Heaton Research, Inc. St. Louis, MO, USA v Publisher: Heaton Research, Inc Programming Neural Networks with Encog 3 in Java First Printing October, 2011 Author: Jeff Heaton Editor: WordsRU.com Cover Art: Carrie Spear ISBN’s for all Editions: 978-1-60439-021-6, Softcover 978-1-60439-022-3, PDF 978-1-60439-023-0, LIT 978-1-60439-024-7, Nook 978-1-60439-025-4, Kindle Copyright ©2011 by Heaton Research Inc., 1734 Clarkson Rd. #107, Chester- field, MO 63017-4976. World rights reserved. The author(s) created reusable code in this publication expressly for reuse by readers. Heaton Research, Inc. grants readers permission to reuse the code found in this publication or down- loaded from our website so long as (author(s)) are attributed in any application containing the reusable code and the source code itself is never redistributed, posted online by electronic transmission, sold or commercially exploited as a stand-alone product. Aside from this specific exception concerning reusable code, no part of this publication may be stored in a retrieval system, trans- mitted, or reproduced in any way, including, but not limited to photo copy, photograph, magnetic, or other record, without prior agreement and written permission of the publisher. Heaton Research, Encog, the Encog Logo and the Heaton Research logo are all trademarks of Heaton Research, Inc., in the United States and/or other countries. TRADEMARKS: Heaton Research has attempted throughout this book to distinguish proprietary trademarks from descriptive terms by following the capitalization style used by the manufacturer.
    [Show full text]
  • A Thesis Entitled a Supervised Machine Learning Approach Using
    A Thesis entitled A Supervised Machine Learning Approach Using Object-Oriented Programming Principles by Merl J. Creps Jr Submitted to the Graduate Faculty as partial fulfillment of the requirements for the Masters of Science Degree in Computer Science and Engineering Dr. Jared Oluoch, Committee Chair Dr. Weiqing Sun, Committee Member Dr. Henry Ledgard, Committee Member Dr. Amanda C. Bryant-Friedrich, Dean College of Graduate Studies The University of Toledo May 2018 Copyright 2018, Merl J. Creps Jr This document is copyrighted material. Under copyright law, no parts of this document may be reproduced without the expressed permission of the author. An Abstract of A Supervised Machine Learning Approach Using Object-Oriented Programming Principles by Merl J. Creps Jr Submitted to the Graduate Faculty as partial fulfillment of the requirements for the Masters of Science Degree in Computer Science and Engineering The University of Toledo May 2018 Artificial Neural Networks (ANNs) can be defined as a collection of interconnected layers, neurons and weighted connections. Each layer is connected to the previous layer by weighted connections that couple each neuron in a layer to the next neuron in the adjacent layer. An ANN resembles a human brain with multiple interconnected synapses which allow for the signal to easily flow from one neuron to the next. There are two distinct constructs which an ANN can assume; they can be thought of as Supervised and Unsupervised networks. Supervised neural networks have known outcomes while Unsupervised neural networks have unknown outcomes. This thesis will primarily focus on Supervised neural networks. The contributions of this thesis are two-fold.
    [Show full text]
  • Optimization of Neural Network Architecture for Biomechanic Classification Tasks with Electromyogram Inputs
    International Journal of Artificial Intelligence and Applications (IJAIA), Vol. 7, No. 5, September 2016 OPTIMIZATION OF NEURAL NETWORK ARCHITECTURE FOR BIOMECHANIC CLASSIFICATION TASKS WITH ELECTROMYOGRAM INPUTS Alayna Kennedy 1 and Rory Lewis 2 1 The Pennsylvania State University, University Park, PA, USA 2 Department of Computer Science, University of Colorado at Colorado Springs, USA ABSTRACT Electromyogram signals (EMGs) contain valuable information that can be used in man-machine interfacing between human users and myoelectric prosthetic devices. However, EMG signals are complicated and prove difficult to analyze due to physiological noise and other issues. Computational intelligence and machine learning techniques, such as artificial neural networks (ANNs), serve as powerful tools for analyzing EMG signals and creating optimal myoelectric control schemes for prostheses. This research examines the performance of four different neural network architectures (feedforward, recurrent, counter propagation, and self organizing map) that were tasked with classifying walking speed when given EMG inputs from 14 different leg muscles. Experiments conducted on the data set suggest that self organizing map neural networks are capable of classifying walking speed with greater than 99% accuracy. KEYWORDS Electromyogram, Prostheses, Neural Networks, Biomechanical Analysis, Machine Learning, Myoelectric Control Schemes, Self Organizing Maps, Pattern Recognition Algorithms 1. INTRODUCTION Assistive prosthetic devices commonly use a myoelectric control scheme, which adjusts the function of a prosthetic device given electromyogram (EMG) inputs, and which has been proven an effective method of man-machine interfacing between user and prosthetic [1]-[4]. However, despite the significant development of the prosthetic industry over the past decade, high-accuracy commercial prostheses remain too expensive for the average middle-class amputee to afford [5], [6].
    [Show full text]
  • Deep Learning Feature Extraction Approach for Hematopoietic Cancer Subtype Classification
    International Journal of Environmental Research and Public Health Article Deep Learning Feature Extraction Approach for Hematopoietic Cancer Subtype Classification Kwang Ho Park 1 , Erdenebileg Batbaatar 1 , Yongjun Piao 2, Nipon Theera-Umpon 3,4,* and Keun Ho Ryu 4,5,6,* 1 Database and Bioinformatics Laboratory, College of Electrical and Computer Engineering, Chungbuk National University, Cheongju 28644, Korea; [email protected] (K.H.P.); [email protected] (E.B.) 2 School of Medicine, Nankai University, Tianjin 300071, China; [email protected] 3 Department of Electrical Engineering, Faculty of Engineering, Chiang Mai University, Chiang Mai 50200, Thailand 4 Biomedical Engineering Institute, Chiang Mai University, Chiang Mai 50200, Thailand 5 Data Science Laboratory, Faculty of Information Technology, Ton Duc Thang University, Ho Chi Minh 700000, Vietnam 6 Department of Computer Science, College of Electrical and Computer Engineering, Chungbuk National University, Cheongju 28644, Korea * Correspondence: [email protected] (N.T.-U.); [email protected] or [email protected] (K.H.R.) Abstract: Hematopoietic cancer is a malignant transformation in immune system cells. Hematopoi- etic cancer is characterized by the cells that are expressed, so it is usually difficult to distinguish its heterogeneities in the hematopoiesis process. Traditional approaches for cancer subtyping use statistical techniques. Furthermore, due to the overfitting problem of small samples, in case of a minor cancer, it does not have enough sample material for building a classification model. Therefore, Citation: Park, K.H.; Batbaatar, E.; we propose not only to build a classification model for five major subtypes using two kinds of losses, Piao, Y.; Theera-Umpon, N.; Ryu, K.H.
    [Show full text]
  • Feature Engineering for Predictive Modeling Using Reinforcement Learning
    The Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18) Feature Engineering for Predictive Modeling Using Reinforcement Learning Udayan Khurana Horst Samulowitz Deepak Turaga [email protected] [email protected] [email protected] IBM Research AI IBM Research AI IBM Research AI Abstract Feature engineering is a crucial step in the process of pre- dictive modeling. It involves the transformation of given fea- ture space, typically using mathematical functions, with the objective of reducing the modeling error for a given target. However, there is no well-defined basis for performing effec- tive feature engineering. It involves domain knowledge, in- tuition, and most of all, a lengthy process of trial and error. The human attention involved in overseeing this process sig- nificantly influences the cost of model generation. We present (a) Original data (b) Engineered data. a new framework to automate feature engineering. It is based on performance driven exploration of a transformation graph, Figure 1: Illustration of different representation choices. which systematically and compactly captures the space of given options. A highly efficient exploration strategy is de- rived through reinforcement learning on past examples. rental demand (kaggle.com/c/bike-sharing-demand) in Fig- ure 2(a). Deriving several features (Figure 2(b)) dramatically Introduction reduces the modeling error. For instance, extracting the hour of the day from the given timestamp feature helps to capture Predictive analytics are widely used in support for decision certain trends such as peak versus non-peak demand. Note making across a variety of domains including fraud detec- that certain valuable features are derived through a compo- tion, marketing, drug discovery, advertising, risk manage- sition of multiple simpler functions.
    [Show full text]
  • Detecting Simulated Attacks in Computer Networks Using Resilient Propagation Artificial Neural Networks
    ISSN 2395-8618 Detecting Simulated Attacks in Computer Networks Using Resilient Propagation Artificial Neural Networks Mario A. Garcia and Tung Trinh Abstract—In a large network, it is extremely difficult for an (Unsupervised Neural Network can detect attacks without administrator or security personnel to detect which computers training.) There are several training algorithms to train neural are being attacked and from where intrusions come. Intrusion networks such as back propagation, the Manhattan update detection systems using neural networks have been deemed a rule, Quick propagation, or Resilient propagation. promising solution to detect such attacks. The reason is that This research describes a solution of applying resilient neural networks have some advantages such as learning from training and being able to categorize data. Many studies have propagation artificial neural networks to detect simulated been done on applying neural networks in intrusion detection attacks in computer networks. The resilient propagation is a systems. This work presents a study of applying resilient supervised training algorithm. The term “supervised” propagation neural networks to detect simulated attacks. The indicates that the neural networks are trained with expected approach includes two main components: the Data Pre- output. The resilient propagation algorithm is considered an processing module and the Neural Network. The Data Pre- efficient training algorithm because it does not require any processing module performs normalizing data function while the parameter setting before being used [14]. In other words, Neural Network processes and categorizes each connection to learning rates or update constants do not need to be computed. find out attacks. The results produced by this approach are The approach is tested on eight neural network configurations compared with present approaches.
    [Show full text]
  • Comparative Analysis of Simulators for Neural Networks Joone and Neuroph
    COMPUTER MODELLING & NEW TECHNOLOGIES 2016 20(1) 16-20 Zdravkova E, Nenkov N Comparative analysis of simulators for neural networks Joone and NeuroPh E Zdravkova, N Nenkov* Faculty of Mathematics and Informatics, University of Shumen “Episkop Konstantin Preslavsky” 115, Universitetska St., Shumen 9712, Bulgaria *Corresponding author’s e-mail: [email protected], [email protected] Received 1 March 2016, www.cmnt.lv Abstract Keywords: This paper describes a comparative analysis of two simulator neural networks - Joone and NeuroPh. Both neural network simulators are object-oriented and java - based. The analysis seeks to show how much these two simulators neural network simulator are similar and how different in their characteristics, what neural networks is suitable to be made through logical function exclusive OR them, what are their advantages and disadvantages, how they can be used interchangeably to give certain neural network architecture desired result. For the purpose of comparative analysis of both the simulator will be realized logic function, which is not among the standard, and relatively complex and is selected as a combination of several standard logical operations. 1 Introduction 2 Methodology Both the simulator selected for the study are Java - based To be tested and analyzed both the simulator will realize and object - oriented simulators. The used simulators are logical function, and which is relatively complex and is not Joone 4.5.2 and NeuroPh 2.92. Joone is object - oriented among the standard. The generated neural network calculate frameworks allows to build different types of neural the result of the following logical function: networks. It is built on combining elements which can also (((( A XOR B) AND C) OR D) ((((E XOR F) AND G) OR be expanded to build new training algorithms and archi- H)) I)) AND ((( J XOR K)(L XOR M)) OR (N O)) tecttures for neural networks.
    [Show full text]