www.it-ebooks.info Mastering Machine Learning with scikit-learn Apply effective learning algorithms to real-world problems using scikit-learn Gavin Hackeling BIRMINGHAM - MUMBAI www.it-ebooks.info Mastering Machine Learning with scikit-learn Copyright © 2014 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information. First published: October 2014 Production reference: 1221014 Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK. ISBN 978-1-78398-836-5 www.packtpub.com Cover image by Amy-Lee Winfield [email protected]( ) www.it-ebooks.info Credits Author Project Coordinator Gavin Hackeling Danuta Jones Reviewers Proofreaders Fahad Arshad Simran Bhogal Sarah Guido Tarsonia Sanghera Mikhail Korobov Lindsey Thomas Aman Madaan Indexer Acquisition Editor Monica Ajmera Mehta Meeta Rajani Graphics Content Development Editor Sheetal Aute Neeshma Ramakrishnan Ronak Dhruv Disha Haria Technical Editor Faisal Siddiqui Production Coordinator Kyle Albuquerque Copy Editors Roshni Banerjee Cover Work Adithi Shetty Kyle Albuquerque www.it-ebooks.info About the Author Gavin Hackeling develops machine learning services for large-scale documents and image classification at an advertising network in New York. He received his Master's degree from New York University's Interactive Telecommunications Program, and his Bachelor's degree from the University of North Carolina. To Hallie, for her support, and Zipper, without whose contributions this book would have been completed in half the time. www.it-ebooks.info About the Reviewers Fahad Arshad completed his PhD at Purdue University in the Department of Electrical and Computer Engineering. His research interests focus on developing algorithms for software testing, error detection, and failure diagnosis in distributed systems. He is particularly interested in data-driven analysis of computer systems. His work has appeared at top dependability conferences—DSN, ISSRE, ICAC, Middleware, and SRDS—and he has been awarded grants to attend DSN, ICAC, and ICNP. Fahad has also been an active contributor to security research while working as a cybersecurity engineer at NEEScomm IT. He has recently taken on a position as a systems engineer in the industry. Sarah Guido is a data scientist at Reonomy, where she's helping build disruptive technology in the commercial real estate industry. She loves Python, machine learning, and the startup world. She is an accomplished conference speaker and an O'Reilly Media author, and is very involved in the Python community. Prior to joining Reonomy, Sarah earned a Master's degree from the University of Michigan School of Information. www.it-ebooks.info Mikhail Korobov is a software developer at ScrapingHub Inc., where he works on web scraping, information extraction, natural language processing, machine learning, and web development tasks. He is an NLTK team member, Scrapy team member, and an author or contributor to many other open source projects. I'd like to thank my wife, Aleksandra, for her support and patience and for the cookies. Aman Madaan is currently pursuing his Master's in Computer Science and Engineering. His interests span across machine learning, information extraction, natural language processing, and distributed computing. More details about his skills, interests, and experience can be found at http://www.amanmadaan.in. www.it-ebooks.info www.PacktPub.com Support files, eBooks, discount offers, and more You might want to visit www.PacktPub.com for support files and downloads related to your book. Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details. At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks. TM http://PacktLib.PacktPub.com Do you need instant solutions to your IT questions? PacktLib is Packt's online digital book library. Here, you can access, read, and search across Packt's entire library of books. Why subscribe? • Fully searchable across every book published by Packt • Copy and paste, print, and bookmark content • On demand and accessible via web browser Free access for Packt account holders If you have an account with Packt at www.PacktPub.com, you can use this to access PacktLib today and view nine entirely free books. Simply use your login credentials for immediate access. www.it-ebooks.info www.it-ebooks.info Table of Contents Preface 1 Chapter 1: The Fundamentals of Machine Learning 7 Learning from experience 8 Machine learning tasks 10 Training data and test data 11 Performance measures, bias, and variance 13 An introduction to scikit-learn 16 Installing scikit-learn 16 Installing scikit-learn on Windows 17 Installing scikit-learn on Linux 17 Installing scikit-learn on OS X 18 Verifying the installation 18 Installing pandas and matplotlib 18 Summary 19 Chapter 2: Linear Regression 21 Simple linear regression 21 Evaluating the fitness of a model with a cost function 25 Solving ordinary least squares for simple linear regression 27 Evaluating the model 29 Multiple linear regression 31 Polynomial regression 35 Regularization 40 Applying linear regression 41 Exploring the data 41 Fitting and evaluating the model 44 Fitting models with gradient descent 46 Summary 50 www.it-ebooks.info Table of Contents Chapter 3: Feature Extraction and Preprocessing 51 Extracting features from categorical variables 51 Extracting features from text 52 The bag-of-words representation 52 Stop-word filtering 55 Stemming and lemmatization 56 Extending bag-of-words with TF-IDF weights 59 Space-efficient feature vectorizing with the hashing trick 62 Extracting features from images 63 Extracting features from pixel intensities 63 Extracting points of interest as features 65 SIFT and SURF 67 Data standardization 69 Summary 70 Chapter 4: From Linear Regression to Logistic Regression 71 Binary classification with logistic regression 72 Spam filtering 73 Binary classification performance metrics 76 Accuracy 77 Precision and recall 79 Calculating the F1 measure 80 ROC AUC 81 Tuning models with grid search 84 Multi-class classification 86 Multi-class classification performance metrics 90 Multi-label classification and problem transformation 91 Multi-label classification performance metrics 94 Summary 95 Chapter 5: Nonlinear Classification and Regression with Decision Trees 97 Decision trees 97 Training decision trees 99 Selecting the questions 100 Information gain 103 Gini impurity 108 Decision trees with scikit-learn 109 Tree ensembles 112 The advantages and disadvantages of decision trees 113 Summary 114 [ ii ] www.it-ebooks.info Table of Contents Chapter 6: Clustering with K-Means 115 Clustering with the K-Means algorithm 117 Local optima 123 The elbow method 124 Evaluating clusters 128 Image quantization 130 Clustering to learn features 132 Summary 135 Chapter 7: Dimensionality Reduction with PCA 137 An overview of PCA 137 Performing Principal Component Analysis 142 Variance, Covariance, and Covariance Matrices 142 Eigenvectors and eigenvalues 143 Dimensionality reduction with Principal Component Analysis 146 Using PCA to visualize high-dimensional data 149 Face recognition with PCA 150 Summary 153 Chapter 8: The Perceptron 155 Activation functions 157 The perceptron learning algorithm 158 Binary classification with the perceptron 159 Document classification with the perceptron 166 Limitations of the perceptron 167 Summary 169 Chapter 9: From the Perceptron to Support Vector Machines 171 Kernels and the kernel trick 172 Maximum margin classification and support vectors 176 Classifying characters in scikit-learn 179 Classifying handwritten digits 179 Classifying characters in natural images 182 Summary 185 Chapter 10: From the Perceptron to Artificial Neural Networks 187 Nonlinear decision boundaries 188 Feedforward and feedback artificial neural networks 189 Multilayer perceptrons 189 Minimizing the cost function 191 Forward propagation 192 Backpropagation 198 [ iii ] www.it-ebooks.info Table of Contents Approximating XOR with Multilayer perceptrons 212 Classifying handwritten digits 213 Summary 214 Index 217 [ iv ] www.it-ebooks.info Preface Recent years have seen the rise of machine learning, the study of software that learns from experience. While machine learning is a new discipline, it has found many applications. We rely on some of these applications daily; in some cases, their successes have already