Automated Lazy Metalearning in Introspective Reasoning Systems Doctor

Doctoral theses at NTNU, 2020:193 al thesis T or Gunnar Høst Houeland Tor Gunnar Høst Houeland Automated lazy metalearning in introspective reasoning systems Doctor ISBN 978-82-326-4734-7 (printed ver.) ISBN 978-82-326-4735-4 (electronic ver.) 1503-8181 Doctor NTNU al theses at NTNU, 2020:193 al theses at NTNU, echnology Engineering Philosophiae Doctor Thesis for the Degree of Thesis for the Degree Department of Computer Science Faculty of Information Technology and Electrical Faculty of Information Technology Norwegian University of Science and T ,.1++.6/0,1#)+" &#/'/$,.0&##%.##,$&'),/,-&'#,!0,. .,+"&#'*1+# ,.3#%'++'2#./'04,$!'#+!#+"#!&+,),%4 !1)04,$7+$,.*0',+#!&+,),%4+")#!0.'!)+%'+##.'+% #-.0*#+0,$,*-10#.!'#+!# ,.3#%'++'2#./'04,$!'#+!#+"#!&+,),%4 &#/'/$,.0&##%.##,$&'),/,-&'#,!0,. !1)04,$7+$,.*0',+#!&+,),%4+")#!0.'!)+%'+##.'+% #-.0*#+0,$,*-10#.!'#+!# 5,.1++.6/0,1#)+" 7 -.'+0#"2#. 7 #)#!0.,+'!2#. 7 ,!0,.)0&#/#/0 .'+0#" 4.$'/(/#+0#. Abstract Machine learning systems are becoming increasingly widespread and important, and these days machine learning is used in some form in most industries. However, the application of machine learning technology still has a relatively high barrier to entry, requiring both machine learning expertise and domain knowledge. In this thesis, we present a metareasoning approach to multi-method machine learning that allows the system to adapt and optimize learning for a given domain automatically, without requiring human expert judgment. In contrast to popular deep learning methods for similar situations, the approach presented here does not require specialized hardware nor large data sets. Multiple machine learning components are continuously evaluated at run-time while solving problems, using a framework to analyze overall system performance based on observed prediction performance and time spent. The system automatically learns to prioritize the methods with the best empirical performance for a given domain. In experiments using data sets from the UCI machine learning repository and machine learning methods from the Weka suite, an example implementation out- performed individual methods and other metareasoning approaches. i ii Preface This thesis is submitted to the Norwegian University of Science and Technology (NTNU) for partial fulfillment of the requirements for the degree of philosophiae doctor. The research work reported herein was conducted at the Department of Computer Science (IDI), NTNU. This doctoral work was financed by and conducted at the Department of Com- puter Science, NTNU, Trondheim, with Agnar Aamodt as the main supervisor and Helge Langseth as co-supervisor. Additionally, Google supported me by allowing me to spend part of my time at work to complete writing the last paper “A Learning System based on Lazy Metareasoning” and this thesis itself. iii iv Acknowledgments Firstly, I would like to thank my primary supervisor Agnar Aamodt for his help throughout my research. Our discussions and his feedback were foundational for the research presented in this thesis, and I could not have completed this work without his continued support and encouragement. Secondly, I would also like to thank my co-supervisors, my research group and other researchers at IDI, and my fellow PhD students for their collaboration, helpful feedback, and many interesting discussions. I would also like to thank Google and my managers there for letting me spend “20% time” and take time off when needed to complete the work on my thesis. Finally, I would like to thank my wife Weilin and my parents for their love and support, especially during the personally challenging third phase of my thesis work. I would not have been able to complete this work without them. v vi Contents Abstract i Preface iii Acknowledgments v I Research overview 1 1 Introduction 3 1.1Motivation................................ 3 1.2 Research objectives ............................ 3 1.3 Research methodology .......................... 5 1.4Thesisstructure.............................. 5 1.4.1 PartI ............................... 6 1.4.2 PartII............................... 6 2 Background and design principles 9 2.1Predictingfromexamples........................ 9 2.1.1 Practicalmachinelearning.................... 10 2.2Effectivemachinelearning........................ 11 2.3Lazyandeagerlearning......................... 13 2.4Metareasoningarchitectures....................... 13 2.4.1 Datasetmeta-knowledge.................... 15 2.4.2 Failure-drivenlearning...................... 15 2.4.3 Optimizinggeneralizationperformance............. 16 2.5Multipleclassifiersystems........................ 16 2.6Banditalgorithms............................ 18 2.6.1 Multi-armedbanditpolicies................... 18 2.6.2 Epsilon-basedpolicies...................... 19 2.6.3 UCBandUCTpolicies..................... 19 vii CONTENTS 3 Research summary 21 3.1 Research process overview . ..................... 21 3.1.1 Main research phases ....................... 21 3.2PhaseI:Learningincase-basedreasoning............... 22 3.2.1 PaperA.............................. 24 3.2.2 PaperB.............................. 25 3.3 Phase II: Performance/cost trade-off in specific scenarios ....... 26 3.3.1 PaperC.............................. 27 3.3.2 PaperD.............................. 28 3.3.3 PaperE.............................. 29 3.4PhaseIII:Autonomousself-optimizinglearningsystems....... 31 3.4.1 PaperF.............................. 33 4 Evaluation and discussion 35 4.1 Research questions and contributions .................. 35 4.2Randomdecisiontreealgorithm..................... 38 4.3 Comparing ALMA to Leake and Wilson’s position paper ....... 38 4.4Theeffectofaddingametareasoninglayer............... 40 5 Concluding remarks 43 5.1Futurework................................ 44 II Selected papers 51 A An Introspective Component-Based Approach for Meta-Level Rea- soning in Clinical Decision Support Systems 53 B The Utility Problem for Lazy Learners - Towards a Non-eager Approach 67 C An Efficient Random Decision Tree Algorithm for Case-Based Reasoning Systems 85 D An Efficient Hybrid Classification Algorithm - an Example from Palliative Care 93 E Extended abstract: Combining CBR and BN using metareasoning 103 F A Learning System based on Lazy Metareasoning 117 viii Part I Research overview 1 Chapter 1 Introduction 1.1 Motivation Machine learning systems have achieved considerable success in recent years, and are being used in more and more applications. However, new applications com- monly rely on human experts to perform crucial tasks: modeling the domain, processing the data to be suitable for machine learning, selecting an appropriate machine learning method, optimizing parameters, etc. These decisions are domain- dependent and often rely on domain experts working together with knowledge en- gineers to extract and structure the required domain knowledge, which is costly and time-consuming. This is known as the knowledge acquisition bottleneck, which is limiting the usefulness of machine learning for practical applications. Automatic machine learning (AutoML) is an alternative to this expertise-driven process, where machine learning systems instead learn entirely autonomously from data without human intervention. To achieve this goal, a computer system needs to simultaneously reason about different knowledge types and about how to combine them to meet higher-level goals. This thesis explores how a practical system can effectively combine several types of reasoning based on explicit representations of reasoning methods and metareasoning processes, without requiring human experts or specialized hardware. In particular, the ALMA system presented in this thesis is a metareasoning system [15] addressing three fundamental issues identified by Leake and Wilson [33]: hav- ing a flexible learning focus, enabling multistrategy introspective learning, and monitoring processing characteristics in addition to outcomes. 1.2 Research objectives The overall goal of the research presented in this thesis is to examine the role of learning in introspective reasoning systems, and how this kind of learning can be achieved in fully automated systems that do not require manual expert configura- tion and tuning. 3 1.2. RESEARCH OBJECTIVES This overall goal is divided into four specific research questions. The first two research questions form the background for developing such systems: what advantages do the combination of learning and introspection provide for such systems, and how does it set them apart from other paradigms? The latter two research questions concern the development of a new system demonstrating these advantages, which includes a working metalearning design that can be applied to practical problem-solving. Introspective learning research questions RQ1 How can learners perform better using introspective capabilities? This research question is concerned with examining what the capabilities of introspective reasoners are, and specifically in which ways they can improve learning abilities. The focus is on fully automated reasoning systems, which means that any such introspective capabilities need to be clear and explicit enough to be programmable and able to run autonomously without human assistance. RQ2 How can the performance of reasoning systems with different capabilities be evaluated? Evaluating a system’s performance is essential to know whether it is actually improving or not. A high-level goal might be to increase the intelligence of a system, but the exact

Load more