DOCSLIB.ORG

Anomaly Detection on Gas Turbine Time-Series' Data Using Deep

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Anomaly Detection on Gas Turbine Time-Series' Data Using Deep Anomaly Detection on Gas Turbine Time-series’ Data Using Deep LSTM-Autoencoder Marzieh Farahani Marzieh Farahani Autumn 2020 Degree Project in Computational Science and Engineering, 30 credits Supervisor: Lili Jiang Extern Supervisor: Mohamed Elhafiz Hassan Examiner: Eddie Wadbro Master of Science Programme in Computational Science and Engineering, 120 credits Abstract Anomaly detection with the aim of identifying outliers plays a very im- portant role in various applications (e.g., online spam, manufacturing, finance etc.). An automatic and reliable anomaly detection tool with accurate prediction is essential in many domains. This thesis proposes an anomaly detection method by applying deep LSTM (long short-term memory) especially on time-series data. By validating on real-world data at Siemens Industrial Turbomachinery (SIT), the proposed method shows promising performance, and can be employed in different data domains like device logs of turbine machines to provide useful informa- tion on abnormal behaviors. In detail, our proposed method applies an autoencoder to have fea- ture selection by keeping vital features, and learn the time series’s en- coded representation. This approach reduces the extensive input data by pulling out the autoencoder’s latent layer output. For prediction, we then train a deep LSTM model with three hidden layers based on the encoder’s latent layer output. Afterwards, given the output from the prediction model, we detect the anomaly sensors related to the specific gas turbine by using a threshold approach. Our experimental results show that our proposed methods perform well on noisy and real-world dataset in order to detect anomalies. More- over, it confirmed that making predictions based on encoding represen- tation, which is under reduction, is more accurate. We could say ap- plying autoencoder can improve both anomaly detection and prediction tasks. Additionally, the performance of deep neural networks would be significantly improved for data with high complexity. Acknowledgements We are grateful because we managed to complete the master final project in anomaly de- tection of a gas turbine using deep LSTM-Autoencoder within the given time by Marzieh Farahani, a student of the master program of computational science and engineering. This theis is could not be completed without the effort and cooperation of Siemens Company and Umeå University. I also thank both supervisors in Siemens Company and Umeå University, Mr. Mohamed Elhafiz Hassan and Dr. Lili Jiang for the guidance and encouragement in finishing the final project. Last but not least, I would like to thank Siemens Data Scientist members in Siemens Com- pany, My family, and Mr. Mehrdad Farahani for their constant source of inspiration and guidance. Contents 1 Introduction 1 1.1 Objectives2 1.2 Scope and Limitation2 1.3 Literature Review3 1.3.1 Statistical-based methods3 1.3.2 Prediction-based methods4 1.3.3 Reconstruction-based methods5 1.4 Thesis Structure5 2 Principles and Concepts 6 2.1 Time Series and Anomaly Forecasting6 2.1.1 Key Components associated with An Anomaly Detection Problem8 2.1.1.1 Nature/Type of Anomaly8 2.1.1.2 Type of Time-spaces8 2.2 Time Series and Deep LSTM9 2.3 Dimentionality Reduction (Autoencoder) 11 3 Methodology 14 3.1 Dataset 14 3.2 Model Design 19 3.3 Prediction Model 19 3.3.1 Reconstruction Autoencoder 19 3.3.1.1 Reduction Using AE 21 3.3.2 Deep LSTM 23 3.4 Detection Model 26 3.4.1 Anomaly Scoring and Selection of Candidate Set 27 4 Experimental Study and Results Analysis 29 4.1 Prediction Model 29 4.1.1 Reconstruction Autoencoder 29 4.1.2 Deep LSTM 33 4.2 Detection Model 36 5 Conclusion and Future Work 39 5.1 Conclusion 39 5.2 Future Work 39 References 41 1(43) 1 Introduction Anomaly detection (aka outlier detection) is the process of identifying unexpected items, observations or events in data sets, which differ from the norm. As an integral part of most companies and businesses, anomaly detection significantly reduces financial and technical losses. Especially, as time-ordered data in various platforms (industrial, health, economic, and financial) has been exponentially increasing along with the emerging Internet of Things (IoT) [18] , which enables them to collect and share data. This growth creates new business opportunities as well as brings more challenges to detect outliers among the time-series data. However, many companies usually have manual monitoring for identifying anomalies on different underlying bases, which require substantial human effort to monitor daily or weekly reports on operations or performance. Thus, it is challenging for the companies to track all metrics simultaneously and find a correlation between them. Besides the above difficulties, time series data in companies is noisy and in large scale, and especially, the label or the class of the anomalies data is lacking. Many types of re- search are trying to apply data-driven methods. These methods for anomaly detection can be mainly categorized into three types: statistical modelings, such as the k-means cluster- ing and Random forest, temporal feature modeling, which is mainly based on the Long Short-term Memory (LSTM), and spatial feature modeling, which takes advantage of Con- volutions Neural Network CNN [14]. The primary purpose of these methods is developing stable algorithms by adopting system conditions to detect outliers even in different environ- ments. Deep learning methods have become successful due to the capacity to handle non-linearity in complex temporal correlation [8]. Deep learning (DL) is retrieved from classical Machine Learning (ML). Still, deep learning is responsible for the growth of Artificial Intelligence usage by improving the existing algorithms. It has shown high-grade performance because of its power to deal with unstructured and unlabeled data. As well as there is no need for domain knowledge to extract features. Nevertheless, it is fair to say in deep learning approaches, there are limitations such as extra time to train and need lots of training data. Moreover, one of the most considered boundaries of deep learning is the Neural Networks at the core of deep learning are black boxes. The project aims to provide and apply the deep LSTM method with using autoencoder in an unsupervised way to detect time series anomalies. Apart from working with an unla- beled dataset, there is no need to conduct feature engineering because it is a complicated task. Instead, many deep neural network parameters are trained to learn the input data’s critical feature during the training stage. Plus, Autoencoder helps deal with the large scale dimensional inputs data which they are ordered in time. 2(43) 1.1 Objectives Siemens Industrial Turbomachinery (SIT) is one of the biggest international companies in power generation. The company invested in diverse projects to examine and study machine lifetime and its corresponding components to identify how and when various failures influ- enced the system. In recent years, the digitalization transformation benefits from collecting and maintaining data in database formats that carry various valuable information about un- expected events, component repair, and operation outage history. The controlling system, which includes a computing device, gives us information about the hardware component’s thermodynamic and operating parameters using sensors placed along with the turbines sec- tions. This thesis’s principal goal is to develop an advanced maintenance strategy that can help the power plant operators increase their assets’ availability and reliability and minimize their CAPEX and OPEX. Capital Expenditures (CAPEX) are significant purchases a company proceeds on goods or services to develop a company’s future performance. Furthermore, Operating Expenses(OPEX) are the typical cost, such as salaries and rent that a company incurs to run their day-to-day operations [36]. To reach this goal, we have stated a general Deep Learning (DL) model with two desires: • Daily Forecasting in order to predict each sensor’s five minutes head for specific gas turbine machine. • Detection in order to find the list of sensors with anomaly behavior that gave the system hard time. It improves system cost along with the daily Forecasting model. The model is expected: • The model must be generalize. In this study, the model performed for the specific case study. For a different case study the result should be obtained within minimal changes. • The model should be accurate and valided under different environmental conditions. The prediction/estimation approach detects abnormal behavior (in a shape anomaly) on the collected data by comparing it with the desired network outputs. It helps build the decision- making process for the power plant operators automatically and classify the useful pattern during the operations. Detecting anomalies can give pre-warning and reduce system costs to the manufactories. The work of anomaly detection in this thesis especially provides useful information to the department within Siemens Industrial Turbomachinery (SIT). 1.2 Scope and Limitation The thesis scope’s decision has been taken after careful analysis of the customer service dataset. It is required to declare that a turbomachine, such as a gas turbine, generally in- cludes several sections, and each section includes numerous hardware components. Over 3(43) time, multiple elements, including thermal cycling, vibration, and pressure pulses within the gas turbine were measurement by sensors (aka signals). Fifteen gas turbine units were considered according to customers’ commonly requested units with the frequent turbine-model package. Finally, The project begins with one final gas turbine unit, and the rest were left out of the project because they were in the commissioning phase, and the signal value’s quality and quantity were not good enough. Another limitation is related to the records of signal value for the specific gas turbine unit at several time intervals. The records were collected from 2012 until the ongoing year. Nevertheless, some signals did not have any records for some months between years (2012- 2020).
Load more

Recommended publications
  • Intro to Tensorflow 2.0 MBL, August 2019
    Intro to TensorFlow 2.0 MBL, August 2019 Josh Gordon (@random_forests) 1 Agenda 1 of 2 Exercises ● Fashion MNIST with dense layers ● CIFAR-10 with convolutional layers Concepts (as many as we can intro in this short time) ● Gradient descent, dense layers, loss, softmax, convolution Games ● QuickDraw Agenda 2 of 2 Walkthroughs and new tutorials ● Deep Dream and Style Transfer ● Time series forecasting Games ● Sketch RNN Learning more ● Book recommendations Deep Learning is representation learning Image link Image link Latest tutorials and guides tensorflow.org/beta News and updates medium.com/tensorflow twitter.com/tensorflow Demo PoseNet and BodyPix bit.ly/pose-net bit.ly/body-pix TensorFlow for JavaScript, Swift, Android, and iOS tensorflow.org/js tensorflow.org/swift tensorflow.org/lite Minimal MNIST in TF 2.0 A linear model, neural network, and deep neural network - then a short exercise. bit.ly/mnist-seq ... ... ... Softmax model = Sequential() model.add(Dense(256, activation='relu',input_shape=(784,))) model.add(Dense(128, activation='relu')) model.add(Dense(10, activation='softmax')) Linear model Neural network Deep neural network ... ... Softmax activation After training, select all the weights connected to this output. model.layers[0].get_weights() # Your code here # Select the weights for a single output # ... img = weights.reshape(28,28) plt.imshow(img, cmap = plt.get_cmap('seismic')) ... ... Softmax activation After training, select all the weights connected to this output. Exercise 1 (option #1) Exercise: bit.ly/mnist-seq Reference: tensorflow.org/beta/tutorials/keras/basic_classification TODO: Add a validation set. Add code to plot loss vs epochs (next slide). Exercise 1 (option #2) bit.ly/ijcav_adv Answers: next slide.
    [Show full text]
  • Training Autoencoders by Alternating Minimization
    Under review as a conference paper at ICLR 2018 TRAINING AUTOENCODERS BY ALTERNATING MINI- MIZATION Anonymous authors Paper under double-blind review ABSTRACT We present DANTE, a novel method for training neural networks, in particular autoencoders, using the alternating minimization principle. DANTE provides a distinct perspective in lieu of traditional gradient-based backpropagation techniques commonly used to train deep networks. It utilizes an adaptation of quasi-convex optimization techniques to cast autoencoder training as a bi-quasi-convex optimiza- tion problem. We show that for autoencoder configurations with both differentiable (e.g. sigmoid) and non-differentiable (e.g. ReLU) activation functions, we can perform the alternations very effectively. DANTE effortlessly extends to networks with multiple hidden layers and varying network configurations. In experiments on standard datasets, autoencoders trained using the proposed method were found to be very promising and competitive to traditional backpropagation techniques, both in terms of quality of solution, as well as training speed. 1 INTRODUCTION For much of the recent march of deep learning, gradient-based backpropagation methods, e.g. Stochastic Gradient Descent (SGD) and its variants, have been the mainstay of practitioners. The use of these methods, especially on vast amounts of data, has led to unprecedented progress in several areas of artificial intelligence. On one hand, the intense focus on these techniques has led to an intimate understanding of hardware requirements and code optimizations needed to execute these routines on large datasets in a scalable manner. Today, myriad off-the-shelf and highly optimized packages exist that can churn reasonably large datasets on GPU architectures with relatively mild human involvement and little bootstrap effort.
    [Show full text]
  • Ways to Use Machine Learning Approaches for Software Development
    Ways to use Machine Learning approaches for software development Nicklas Jonsson Nicklas Jonsson VT 2018 Examensarbete, 30 hp Supervisor: Eddie wadbro Extern Supervisor: C4 Contexture Examiner: Henrik Bjorklund¨ Master of Science Programme in Computing Science and Engineering, 300 hp Abstract With the rise of machine learning and in particular deep learning enter- ing all different types of fields, including software development. It could be a bit hard to know where to begin to search for the tools when some- one wants to use machine learning for a one’s problems. This thesis has looked at some available technologies of today for applying machine learning to one’s applications. This thesis has looked at some of the available cloud services, frame- works, and libraries for machine learning and it presents three different implementation structures that can be used with these technologies for the problem of image classification. Acknowledgements I want to thank C4 Contexture for giving me the thesis idea, support, and supplying me with a working station. I also want to thank Lantmannen¨ for supplying me with the image data that was used for this thesis. Finally, I want to thank Eddie Wadbro for his guidance during this thesis and of course a big thanks to my family and friends for their support during this period of my life. 1(45) Content 1 Introduction 3 1.1 The Client 3 1.1.1 C4 Contexture PIM software 4 1.2 The data 4 1.3 Goal 5 1.4 Limitation 5 2 Background 7 2.1 Artificial intelligence, machine learning and deep learning.
    [Show full text]
  • Turbo Autoencoder: Deep Learning Based Channel Codes for Point-To-Point Communication Channels
    Turbo Autoencoder: Deep learning based channel codes for point-to-point communication channels Hyeji Kim Himanshu Asnani Yihan Jiang Samsung AI Center School of Technology ECE Department Cambridge and Computer Science University of Washington Cambridge, United Tata Institute of Seattle, United States Kingdom Fundamental Research [email protected] [email protected] Mumbai, India [email protected] Sewoong Oh Pramod Viswanath Sreeram Kannan Allen School of ECE Department ECE Department Computer Science & University of Illinois at University of Washington Engineering Urbana Champaign Seattle, United States University of Washington Illinois, United States [email protected] Seattle, United States [email protected] [email protected] Abstract Designing codes that combat the noise in a communication medium has remained a significant area of research in information theory as well as wireless communica- tions. Asymptotically optimal channel codes have been developed by mathemati- cians for communicating under canonical models after over 60 years of research. On the other hand, in many non-canonical channel settings, optimal codes do not exist and the codes designed for canonical models are adapted via heuristics to these channels and are thus not guaranteed to be optimal. In this work, we make significant progress on this problem by designing a fully end-to-end jointly trained neural encoder and decoder, namely, Turbo Autoencoder (TurboAE), with the following contributions: (a) under moderate block lengths, TurboAE approaches state-of-the-art performance under canonical channels; (b) moreover, TurboAE outperforms the state-of-the-art codes under non-canonical settings in terms of reliability. TurboAE shows that the development of channel coding design can be automated via deep learning, with near-optimal performance.
    [Show full text]
  • Keras2c: a Library for Converting Keras Neural Networks to Real-Time Compatible C
    Keras2c: A library for converting Keras neural networks to real-time compatible C Rory Conlina,∗, Keith Ericksonb, Joeseph Abbatec, Egemen Kolemena,b,∗ aDepartment of Mechanical and Aerospace Engineering, Princeton University, Princeton NJ 08544, USA bPrinceton Plasma Physics Laboratory, Princeton NJ 08544, USA cDepartment of Astrophysical Sciences at Princeton University, Princeton NJ 08544, USA Abstract With the growth of machine learning models and neural networks in mea- surement and control systems comes the need to deploy these models in a way that is compatible with existing systems. Existing options for deploying neural networks either introduce very high latency, require expensive and time con- suming work to integrate into existing code bases, or only support a very lim- ited subset of model types. We have therefore developed a new method called Keras2c, which is a simple library for converting Keras/TensorFlow neural net- work models into real-time compatible C code. It supports a wide range of Keras layers and model types including multidimensional convolutions, recurrent lay- ers, multi-input/output models, and shared layers. Keras2c re-implements the core components of Keras/TensorFlow required for predictive forward passes through neural networks in pure C, relying only on standard library functions considered safe for real-time use. The core functionality consists of ∼ 1500 lines of code, making it lightweight and easy to integrate into existing codebases. Keras2c has been successfully tested in experiments and is currently in use on the plasma control system at the DIII-D National Fusion Facility at General Atomics in San Diego. 1. Motivation TensorFlow[1] is one of the most popular libraries for developing and training neural networks.
    [Show full text]
  • Double Backpropagation for Training Autoencoders Against Adversarial Attack
    1 Double Backpropagation for Training Autoencoders against Adversarial Attack Chengjin Sun, Sizhe Chen, and Xiaolin Huang, Senior Member, IEEE Abstract—Deep learning, as widely known, is vulnerable to adversarial samples. This paper focuses on the adversarial attack on autoencoders. Safety of the autoencoders (AEs) is important because they are widely used as a compression scheme for data storage and transmission, however, the current autoencoders are easily attacked, i.e., one can slightly modify an input but has totally different codes. The vulnerability is rooted the sensitivity of the autoencoders and to enhance the robustness, we propose to adopt double backpropagation (DBP) to secure autoencoder such as VAE and DRAW. We restrict the gradient from the reconstruction image to the original one so that the autoencoder is not sensitive to trivial perturbation produced by the adversarial attack. After smoothing the gradient by DBP, we further smooth the label by Gaussian Mixture Model (GMM), aiming for accurate and robust classification. We demonstrate in MNIST, CelebA, SVHN that our method leads to a robust autoencoder resistant to attack and a robust classifier able for image transition and immune to adversarial attack if combined with GMM. Index Terms—double backpropagation, autoencoder, network robustness, GMM. F 1 INTRODUCTION N the past few years, deep neural networks have been feature [9], [10], [11], [12], [13], or network structure [3], [14], I greatly developed and successfully used in a vast of fields, [15]. such as pattern recognition, intelligent robots, automatic Adversarial attack and its defense are revolving around a control, medicine [1]. Despite the great success, researchers small ∆x and a big resulting difference between f(x + ∆x) have found the vulnerability of deep neural networks to and f(x).
    [Show full text]
  • Artificial Intelligence Applied to Electromechanical Monitoring, A
    ARTIFICIAL INTELLIGENCE APPLIED TO ELECTROMECHANICAL MONITORING, A PERFORMANCE ANALYSIS Erasmus project Authors: Staš Osterc Mentor: Dr. Miguel Delgado Prieto, Dr. Francisco Arellano Espitia 1/5/2020 P a g e II ANNEX VI – DECLARACIÓ D’HONOR P a g e II I declare that, the work in this Master Thesis / Degree Thesis (choose one) is completely my own work, no part of this Master Thesis / Degree Thesis (choose one) is taken from other people’s work without giving them credit, all references have been clearly cited, I’m authorised to make use of the company’s / research group (choose one) related information I’m providing in this document (select when it applies). I understand that an infringement of this declaration leaves me subject to the foreseen disciplinary actions by The Universitat Politècnica de Catalunya - BarcelonaTECH. ___________________ __________________ ___________ Student Name Signature Date Title of the Thesis : _________________________________________________ ________________________________________________________________ ________________________________________________________________ ________________________________________________________________ P a g e II Contents Introduction........................................................................................................................................5 Abstract ..........................................................................................................................................5 Aim .................................................................................................................................................6
    [Show full text]
  • Unsupervised Speech Representation Learning Using Wavenet Autoencoders Jan Chorowski, Ron J
    1 Unsupervised speech representation learning using WaveNet autoencoders Jan Chorowski, Ron J. Weiss, Samy Bengio, Aaron¨ van den Oord Abstract—We consider the task of unsupervised extraction speaker gender and identity, from phonetic content, properties of meaningful latent representations of speech by applying which are consistent with internal representations learned autoencoding neural networks to speech waveforms. The goal by speech recognizers [13], [14]. Such representations are is to learn a representation able to capture high level semantic content from the signal, e.g. phoneme identities, while being desired in several tasks, such as low resource automatic speech invariant to confounding low level details in the signal such as recognition (ASR), where only a small amount of labeled the underlying pitch contour or background noise. Since the training data is available. In such scenario, limited amounts learned representation is tuned to contain only phonetic content, of data may be sufficient to learn an acoustic model on the we resort to using a high capacity WaveNet decoder to infer representation discovered without supervision, but insufficient information discarded by the encoder from previous samples. Moreover, the behavior of autoencoder models depends on the to learn the acoustic model and a data representation in a fully kind of constraint that is applied to the latent representation. supervised manner [15], [16]. We compare three variants: a simple dimensionality reduction We focus on representations learned with autoencoders bottleneck, a Gaussian Variational Autoencoder (VAE), and a applied to raw waveforms and spectrogram features and discrete Vector Quantized VAE (VQ-VAE). We analyze the quality investigate the quality of learned representations on LibriSpeech of learned representations in terms of speaker independence, the ability to predict phonetic content, and the ability to accurately re- [17].
    [Show full text]
  • Tensorflow, Theano, Keras, Torch, Caffe Vicky Kalogeiton, Stéphane Lathuilière, Pauline Luc, Thomas Lucas, Konstantin Shmelkov Introduction
    TensorFlow, Theano, Keras, Torch, Caffe Vicky Kalogeiton, Stéphane Lathuilière, Pauline Luc, Thomas Lucas, Konstantin Shmelkov Introduction TensorFlow Google Brain, 2015 (rewritten DistBelief) Theano University of Montréal, 2009 Keras François Chollet, 2015 (now at Google) Torch Facebook AI Research, Twitter, Google DeepMind Caffe Berkeley Vision and Learning Center (BVLC), 2013 Outline 1. Introduction of each framework a. TensorFlow b. Theano c. Keras d. Torch e. Caffe 2. Further comparison a. Code + models b. Community and documentation c. Performance d. Model deployment e. Extra features 3. Which framework to choose when ..? Introduction of each framework TensorFlow architecture 1) Low-level core (C++/CUDA) 2) Simple Python API to define the computational graph 3) High-level API (TF-Learn, TF-Slim, soon Keras…) TensorFlow computational graph - auto-differentiation! - easy multi-GPU/multi-node - native C++ multithreading - device-efficient implementation for most ops - whole pipeline in the graph: data loading, preprocessing, prefetching... TensorBoard TensorFlow development + bleeding edge (GitHub yay!) + division in core and contrib => very quick merging of new hotness + a lot of new related API: CRF, BayesFlow, SparseTensor, audio IO, CTC, seq2seq + so it can easily handle images, videos, audio, text... + if you really need a new native op, you can load a dynamic lib - sometimes contrib stuff disappears or moves - recently introduced bells and whistles are barely documented Presentation of Theano: - Maintained by Montréal University group. - Pioneered the use of a computational graph. - General machine learning tool -> Use of Lasagne and Keras. - Very popular in the research community, but not elsewhere. Falling behind. What is it like to start using Theano? - Read tutorials until you no longer can, then keep going.
    [Show full text]
  • Deep Learning with Keras I
    Deep Learning with Keras i Deep Learning with Keras About the Tutorial Deep Learning essentially means training an Artificial Neural Network (ANN) with a huge amount of data. In deep learning, the network learns by itself and thus requires humongous data for learning. In this tutorial, you will learn the use of Keras in building deep neural networks. We shall look at the practical examples for teaching. Audience This tutorial is prepared for professionals who are aspiring to make a career in the field of deep learning and neural network framework. This tutorial is intended to make you comfortable in getting started with the Keras framework concepts. Prerequisites Before proceeding with the various types of concepts given in this tutorial, we assume that the readers have basic understanding of deep learning framework. In addition to this, it will be very helpful, if the readers have a sound knowledge of Python and Machine Learning. Copyright & Disclaimer Copyright 2019 by Tutorials Point (I) Pvt. Ltd. All the content and graphics published in this e-book are the property of Tutorials Point (I) Pvt. Ltd. The user of this e-book is prohibited to reuse, retain, copy, distribute or republish any contents or a part of contents of this e-book in any manner without written consent of the publisher. We strive to update the contents of our website and tutorials as timely and as precisely as possible, however, the contents may contain inaccuracies or errors. Tutorials Point (I) Pvt. Ltd. provides no guarantee regarding the accuracy, timeliness or completeness of our website or its contents including this tutorial.
    [Show full text]
  • Intro to Keras
    Intro to Keras Justin Zhang July 2017 1 Introduction Keras is a high-level Python machine learning API, which allows you to easily run neural networks. Keras is simply a specification; it provides a set of methods that you can use, and it will use a backend (TensorFlow, Theano, or CNTK, as chosen by the user) to actually run your code. Like many machine learning frameworks, Keras is a so-called define-and-run framework. This means that it will define and optimize your neural network in a compilation step before training starts. 2 First Steps First, install Keras: pip install keras We'll go over a fully-connected neural network designed for the MNIST (clas- sifying handwritten digits) dataset. # Can be found at https://github.com/fchollet/keras # Released under the MIT License import keras from keras.datasets import mnist from keras .models import S e q u e n t i a l from keras.layers import Dense, Dropout from keras.optimizers import RMSprop First, we handle the imports. keras.datasets includes many popular machine learning datasets, including CIFAR and IMDB. keras.layers includes a variety of neural network layer types, including Dense, Conv2D, and LSTM. Dense simply refers to a fully-connected layer, and Dropout is a layer commonly used to ad- dress the problem of overfitting. keras.optimizers includes most widely used optimizers. Here, we opt to use RMSprop, an alternative to the classic stochas- tic gradient decent (SGD) algorithm. Regarding keras.models, Sequential is most widely used for simple networks; there is also a functional Model class you can use for more complex networks.
    [Show full text]
  • Auto-Keras: an Efficient Neural Architecture Search System
    Auto-Keras: An Efficient Neural Architecture Search System Haifeng Jin, Qingquan Song, Xia Hu Department of Computer Science and Engineering, Texas A&M University {jin,song_3134,xiahu}@tamu.edu ABSTRACT epochs are required to further train the new architecture towards Neural architecture search (NAS) has been proposed to automat- better performance. Using network morphism would reduce the av- ically tune deep neural networks, but existing search algorithms, erage training time t¯ in neural architecture search. The most impor- e.g., NASNet [41], PNAS [22], usually suffer from expensive com- tant problem to solve for network morphism-based NAS methods is putational cost. Network morphism, which keeps the functional- the selection of operations, which is to select an operation from the ity of a neural network while changing its neural architecture, network morphism operation set to morph an existing architecture could be helpful for NAS by enabling more efficient training during to a new one. The network morphism-based NAS methods are not the search. In this paper, we propose a novel framework enabling efficient enough. They either require a large number of training Bayesian optimization to guide the network morphism for effi- examples [6], or inefficient in exploring the large search space [11]. cient neural architecture search. The framework develops a neural How to perform efficient neural architecture search with network network kernel and a tree-structured acquisition function optimiza- morphism remains a challenging problem. tion algorithm to efficiently explores the search space. Intensive As we know, Bayesian optimization [33] has been widely adopted experiments on real-world benchmark datasets have been done to to efficiently explore black-box functions for global optimization, demonstrate the superior performance of the developed framework whose observations are expensive to obtain.
    [Show full text]