DOCSLIB.ORG

Foundations of Machine Learning Adaptive Computation and Machine Learning

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Foundations of Machine Learning Adaptive Computation and Machine Learning Foundations of Machine Learning Adaptive Computation and Machine Learning Thomas Dietterich, Editor Christopher Bishop, David Heckerman, Michael Jordan, and Michael Kearns, Associate Editors A complete list of books published in The Adaptive Computations and Machine Learning series appears at the back of this book. Foundations of Machine Learning Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar The MIT Press Cambridge, Massachusetts London, England c 2012 Massachusetts Institute of Technology All rights reserved. No part of this book may be reproduced in any form by any electronic or mechanical means (including photocopying, recording, or information storage and retrieval) without permission in writing from the publisher. MIT Press books may be purchased at special quantity discounts for business or sales promotional use. For information, please email special [email protected] or write to Special Sales Department, The MIT Press, 55 Hayward Street, Cam- bridge, MA 02142. This book was set in LATEX by the authors. Printed and bound in the United States of America. Library of Congress Cataloging-in-Publication Data Mohri, Mehryar. Foundations of machine learning / Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. p. cm. - (Adaptive computation and machine learning series) Includes bibliographical references and index. ISBN 978-0-262-01825-8 (hardcover : alk. paper) 1. Machine learning. 2. Computer algorithms. I. Rostamizadeh, Afshin. II. Talwalkar, Ameet. III. Title. Q325.5.M64 2012 006.3’1-dc23 2012007249 10987654321 Contents Preface xi 1 Introduction 1 1.1Applicationsandproblems........................ 1 1.2Definitionsandterminology....................... 3 1.3Cross-validation.............................. 5 1.4Learningscenarios............................ 7 1.5Outline.................................. 8 2 The PAC Learning Framework 11 2.1ThePAClearningmodel......................... 11 2.2Guaranteesforfinitehypothesissets—consistentcase........ 17 2.3Guaranteesforfinitehypothesissets—inconsistentcase....... 21 2.4Generalities................................ 24 2.4.1 Deterministicversusstochasticscenarios............ 24 2.4.2 Bayeserrorandnoise...................... 25 2.4.3 Estimationandapproximationerrors.............. 26 2.4.4 Modelselection.......................... 27 2.5Chapternotes............................... 28 2.6Exercises................................. 29 3 Rademacher Complexity and VC-Dimension 33 3.1Rademachercomplexity......................... 34 3.2Growthfunction............................. 38 3.3VC-dimension............................... 41 3.4Lowerbounds............................... 48 3.5Chapternotes............................... 54 3.6Exercises................................. 55 4 Support Vector Machines 63 4.1Linearclassification............................ 63 4.2SVMs—separablecase......................... 64 vi 4.2.1 Primaloptimizationproblem.................. 64 4.2.2 Supportvectors.......................... 66 4.2.3 Dualoptimizationproblem................... 67 4.2.4 Leave-one-outanalysis...................... 69 4.3SVMs—non-separablecase....................... 71 4.3.1 Primaloptimizationproblem.................. 72 4.3.2 Supportvectors.......................... 73 4.3.3 Dualoptimizationproblem................... 74 4.4Margintheory............................... 75 4.5Chapternotes............................... 83 4.6Exercises................................. 84 5 Kernel Methods 89 5.1Introduction................................ 89 5.2Positivedefinitesymmetrickernels................... 92 5.2.1 Definitions............................ 92 5.2.2 ReproducingkernelHilbertspace................ 94 5.2.3 Properties............................. 96 5.3Kernel-basedalgorithms......................... 100 5.3.1 SVMswithPDSkernels..................... 100 5.3.2 Representertheorem....................... 101 5.3.3 Learningguarantees....................... 102 5.4Negativedefinitesymmetrickernels...................103 5.5Sequencekernels............................. 106 5.5.1 Weightedtransducers...................... 106 5.5.2 Rationalkernels......................... 111 5.6Chapternotes...............................115 5.7Exercises................................. 116 6 Boosting 121 6.1Introduction................................121 6.2AdaBoost................................. 122 6.2.1 Boundontheempiricalerror.................. 124 6.2.2 Relationshipwithcoordinatedescent..............126 6.2.3 Relationshipwithlogisticregression.............. 129 6.2.4 Standarduseinpractice..................... 129 6.3Theoreticalresults............................ 130 6.3.1 VC-dimension-basedanalysis.................. 131 6.3.2 Margin-basedanalysis...................... 131 6.3.3 Marginmaximization...................... 136 6.3.4 Game-theoreticinterpretation..................137 vii 6.4Discussion................................. 140 6.5Chapternotes...............................141 6.6Exercises................................. 142 7 On-Line Learning 147 7.1Introduction................................147 7.2Predictionwithexpertadvice...................... 148 7.2.1 MistakeboundsandHalvingalgorithm............ 148 7.2.2 Weightedmajorityalgorithm.................. 150 7.2.3 Randomizedweightedmajorityalgorithm........... 152 7.2.4 Exponentialweightedaveragealgorithm............ 156 7.3Linearclassification............................159 7.3.1 Perceptronalgorithm.......................160 7.3.2 Winnowalgorithm........................ 168 7.4On-linetobatchconversion....................... 171 7.5Game-theoreticconnection........................174 7.6Chapternotes...............................175 7.7Exercises................................. 176 8 Multi-Class Classification 183 8.1Multi-classclassificationproblem.................... 183 8.2Generalizationbounds.......................... 185 8.3Uncombinedmulti-classalgorithms................... 191 8.3.1 Multi-classSVMs.........................191 8.3.2 Multi-classboostingalgorithms................. 192 8.3.3 Decisiontrees........................... 194 8.4 Aggregated multi-class algorithms ................... 198 8.4.1 One-versus-all...........................198 8.4.2 One-versus-one.......................... 199 8.4.3 Error-correctioncodes...................... 201 8.5Structuredpredictionalgorithms.................... 203 8.6Chapternotes...............................206 8.7Exercises................................. 207 9 Ranking 209 9.1Theproblemofranking......................... 209 9.2Generalizationbound.......................... 211 9.3RankingwithSVMs........................... 213 9.4RankBoost................................ 214 9.4.1 Boundontheempiricalerror.................. 216 9.4.2 Relationshipwithcoordinatedescent..............218 viii 9.4.3 Marginboundforensemblemethodsinranking ....... 220 9.5Bipartiteranking............................. 221 9.5.1 Boostinginbipartiteranking.................. 222 9.5.2 AreaundertheROCcurve................... 224 9.6Preference-basedsetting......................... 226 9.6.1 Second-stagerankingproblem..................227 9.6.2 Deterministicalgorithm..................... 229 9.6.3 Randomizedalgorithm......................230 9.6.4 Extensiontootherlossfunctions................ 231 9.7Discussion................................. 232 9.8Chapternotes...............................233 9.9Exercises................................. 234 10 Regression 237 10.1Theproblemofregression........................ 237 10.2Generalizationbounds.......................... 238 10.2.1Finitehypothesissets...................... 238 10.2.2Rademachercomplexitybounds.................239 10.2.3Pseudo-dimensionbounds.................... 241 10.3Regressionalgorithms.......................... 245 10.3.1Linearregression......................... 245 10.3.2Kernelridgeregression...................... 247 10.3.3Supportvectorregression.................... 252 10.3.4Lasso............................... 257 10.3.5Groupnormregressionalgorithms............... 260 10.3.6On-lineregressionalgorithms.................. 261 10.4Chapternotes...............................262 10.5Exercises................................. 263 11 Algorithmic Stability 267 11.1Definitions.................................267 11.2Stability-basedgeneralizationguarantee................ 268 11.3Stabilityofkernel-basedregularizationalgorithms.......... 270 11.3.1 Application to regression algorithms: SVR and KRR . 274 11.3.2Applicationtoclassificationalgorithms:SVMs........ 276 11.3.3Discussion............................. 276 11.4Chapternotes...............................277 11.5Exercises................................. 277 12 Dimensionality Reduction 281 12.1PrincipalComponentAnalysis..................... 282 ix 12.2KernelPrincipalComponentAnalysis(KPCA)............ 283 12.3KPCAandmanifoldlearning......................285 12.3.1Isomap.............................. 285 12.3.2Laplacianeigenmaps....................... 286 12.3.3Locallylinearembedding(LLE)................ 287 12.4Johnson-Lindenstrausslemma...................... 288 12.5Chapternotes...............................290 12.6Exercises................................. 290 13 Learning Automata and Languages 293 13.1Introduction................................293 13.2Finiteautomata............................. 294 13.3Efficientexactlearning.......................... 295 13.3.1Passivelearning......................... 296 13.3.2Learningwithqueries...................... 297 13.3.3Learningautomatawithqueries................ 298 13.4Identificationinthelimit.......................
Load more

Recommended publications
  • An Introduction to Weighted Automata in Machine Learning
    An Introduction to Weighted Automata in Machine Learning Awni Hannun∗ August 17, 2021 Abstract The goal of this work is to introduce the reader to weighted finite-state automata and their application to machine learning. I begin by motivating the use of automata in machine learning and proceed with an introduction to acceptors, transducers, and their associated properties. Many of the core operations of weighted automata are then described in detail. Following this, the work moves closer to the research frontier by explaining automatic differentiation and its use with weighted automata. The last section presents several extended examples to gain deeper familiarity with weighted automata, their operations, and their use in machine learning. ∗Send correspondence to [email protected] 1 CONTENTS 2 Contents 1 Introduction3 1.1 Monolithic or Modular.........................3 1.2 Advantages of Differentiable Automata................5 1.3 Comparison to Tensors.........................6 1.4 History and Bibliographic Notes...................7 2 Acceptors and Transducers9 2.1 Automata................................9 2.2 Acceptors................................ 10 2.3 Transducers............................... 14 3 Basic Operations 17 3.1 Closure................................. 17 3.2 Union.................................. 19 3.3 Concatenation............................. 21 3.4 Summary................................ 23 4 Advanced Operations 25 4.1 Intersect................................. 25 4.2 Compose................................ 30
    [Show full text]
  • Foundations of Machine Learning Adaptive Computation and Machine Learning
    Foundations of Machine Learning Adaptive Computation and Machine Learning Thomas Dietterich, Editor Christopher Bishop, David Heckerman, Michael Jordan, and Michael Kearns, Associate Editors A complete list of books published in The Adaptive Computations and Machine Learning series appears at the back of this book. Foundations of Machine Learning Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar The MIT Press Cambridge, Massachusetts London, England c 2012 Massachusetts Institute of Technology All rights reserved. No part of this book may be reproduced in any form by any electronic or mechanical means (including photocopying, recording, or information storage and retrieval) without permission in writing from the publisher. MIT Press books may be purchased at special quantity discounts for business or sales promotional use. For information, please email special [email protected] or write to Special Sales Department, The MIT Press, 55 Hayward Street, Cam- bridge, MA 02142. This book was set in LATEX by the authors. Printed and bound in the United States of America. Library of Congress Cataloging-in-Publication Data Mohri, Mehryar. Foundations of machine learning / Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. p. cm. - (Adaptive computation and machine learning series) Includes bibliographical references and index. ISBN 978-0-262-01825-8 (hardcover : alk. paper) 1. Machine learning. 2. Computer algorithms. I. Rostamizadeh, Afshin. II. Talwalkar, Ameet. III. Title. Q325.5.M64 2012 006.3’1-dc23 2012007249 10987654321 Contents Preface xi 1 Introduction 1 1.1Applicationsandproblems........................ 1 1.2Definitionsandterminology....................... 3 1.3Cross-validation.............................. 5 1.4Learningscenarios............................ 7 1.5Outline.................................. 8 2 The PAC Learning Framework 11 2.1ThePAClearningmodel......................... 11 2.2Guaranteesforfinitehypothesissets—consistentcase.......
    [Show full text]
  • 2015-2017 GSAS Bulletin
    New York University Bulletin 2015-2017 New York University Bulletin 2015-2017 Graduate School of Arts and Science Announcement for the 130th and 131st sessions New York University Washington Square New York, New York 10003 Website: www.gsas.nyu.edu Notice: The policies, requirements, course offerings, schedules, activities, tuition, fees, and calendar of the school and its departments and programs set forth in this bulletin are subject to change without notice at any time at the sole discretion of the administration. Such changes may be of any nature, including, but not limited to, the elimination of the school or college, programs, classes, or activities; the relocation of or modification of the content of any of the foregoing; and the cancellation of scheduled classes or other academic activities. Payment of tuition or attendance at any classes shall constitute a student’s acceptance of the administration’s rights as set forth in the above paragraph. Contents Graduate School of Arts and Hebrew and Judaic Studies, Spanish and Portuguese Science: Administration, Skirball Department of . 184 Languages and Literatures . 382 Departments, Programs . 5 History . 194 Study of the Ancient World, History of the Graduate School . 7 Institute for the . 396 Humanities and Social Thought, An Introduction to New York University. 8 John W. Draper Interdisciplinary Admission, Registration, Master’s Program in . 210 and Degree Requirements . .399 New York University and New York . 9 International Relations . 215 Financing Graduate Education . 404 Academic Calendar . 11 Irish and Irish-American Studies . 221 Services and Programs . 408 Departments and Programs Italian Studies . 226 Community Service . 411 Ancient Near Eastern and Egyptian Studies .
    [Show full text]
  • Learning with Weighted Transducers
    Learning with Weighted Transducers Corinna CORTES a and Mehryar MOHRI b,1 a Google Research, 76 Ninth Avenue, New York, NY 10011 b Courant Institute of Mathematical Sciences and Google Research, 251 Mercer Street, New York, NY 10012 Abstract. Weighted finite-state transducers have been used successfully in a variety of natural language processing applications, including speech recognition, speech synthesis, and machine translation. This paper shows how weighted transducers can be combined with existing learning algorithms to form powerful techniques for sequence learning problems. Keywords. Learning, kernels, classification, regression, ranking, clustering, weighted automata, weighted transducers, rational powers series. Introduction Weighted transducer algorithms have been successfully used in a variety of applications in speech recognition [21, 23, 25], speech synthesis [28, 5], optical character recognition [9], machine translation, a variety of other natural language processing tasks including parsing and language modeling,image processing [1], and computationalbiology [15, 6]. This paper outlines the use of weighted transducers in machine learning. A key relevance of weighted transducers to machine learning is their use in kernel methods applied to sequences. Weighted transducers provide a compact and simple rep- resentation of sequence kernels. Furthermore, standard weighted transducer algorithms such as composition and shortest-distance algorithms can be used to efficiently compute kernels based on weighted transducers. 1. Overview of Kernel Methods Kernel methods are widely used in machine learning. They have been successfully used to deal with a variety of learning tasks including classification, regression, ranking, clus- tering, and dimensionality reduction. This section gives a brief overview of these meth- ods. Complex learning tasks are often tackled using a large number of features.
    [Show full text]
  • Language Processing with Weighted Transducers
    TALN 2001, Tours, 2-5 juillet 2001 Language Processing with Weighted Transducers Mehryar Mohri AT&T Labs - Research 180 Park Avenue Florham Park, NJ 07932-0971, USA [email protected] Resum´ e´ - Abstract Les automates et transducteurs ponder´ es´ sont utilises´ dans un ev´ entail d’applications allant de la reconnaissance et synthese` automatiques de la langue a` la biologie informatique. Ils four- nissent un cadre commun pour la representation´ des composants d’un systeme` complexe, ce qui rend possible l’application d’algorithmes d’optimisation gen´ eraux´ tels que la determinisation,´ l’elimination´ des mots vides, et la minimisation des transducteurs ponder´ es.´ Nous donnerons un bref aperc¸u des progres` recents´ dans le traitement de la langue a` l’aide d’automates et transducteurs ponder´ es,´ y compris une vue d’ensemble de la reconnaissance de la parole avec des transducteurs ponder´ es´ et des resultats´ algorithmiques recents´ dans ce domaine. Nous presenterons´ egalement´ de nouveaux resultats´ lies´ a` l’approximation des gram- maires context-free ponder´ ees´ et a` la reconnaissance a` l’aide d’automates ponder´ es.´ Weighted automata and transducers are used in a variety of applications ranging from automatic speech recognition and synthesis to computational biology. They give a unifying framework for the representation of the components of complex systems. This provides opportunities for the application of general optimization algorithms such as determinization, -removal and mini- mization of weighted transducers. We give a brief survey of recent advances in language processing with weighted automata and transducers, including an overview of speech recognition with weighted transducers and recent algorithmic results in that field.
    [Show full text]
  • New York University, Graduate School of Arts and Science Bulletin 2019
    New York University Bulletin / 2019-2021 New York University Bulletin / 2019-2021 Graduate School of Arts and Science Announcement for the 134th and 135th sessions New York University Washington Square New York, New York 10003 Website: gsas.nyu.edu Notice: The policies, requirements, course offerings, schedules, activities, tuition, fees, and calendar of the school and its departments and programs set forth in this bulletin are subject to change without notice at any time at the sole discretion of the administration. Such changes may be of any nature, including, but not limited to, the elimination of the school or college, programs, classes, or activities; the relocation of or modification of the content of any of the foregoing; and the cancellation of scheduled classes or other academic activities. Payment of tuition or attendance at any classes shall constitute a student’s acceptance of the administration’s rights as set forth in the above paragraph. 4 Contents Graduate School of Arts and Environmental Studies . 143 Philosophy . .293 Science: Administration, European and Mediterranean Physics . 300 Departments, Programs . 5 Studies, Center for . 147 Poetics and Theory . 308 History of the Graduate School . 7 Fine Arts, Institute of . 150 Politics . 310 An Introduction to French Literature, New York University . 8 Psychology . 321 Thought and Culture . 156 Schools, Colleges, Institutes & Psychotherapy and French Studies, Institute of . 162 Programs of the University . 8 Psychoanalysis . .338 German . 167 New York University and New York . 10 Religious Studies . 341 Hebrew and Judaic Studies, Academic Calendar . 12 Russian and Slavic Studies . 345 Skirball Department of . 172 Departments and Programs Social and Cultural Analysis .
    [Show full text]