Zur Erlangung des akademischen Grades eines Doktors der Ingenieurwissenschaften (Dr.-Ing.) von der KIT-Fakultät für Wirtschaftswissenschaften des Karlsruher Instituts für Technologie (KIT) genehmigte Dissertation von Patrick Raoul Philipp, M.Sc. Decision-Making with Multi-Step Expert Advice on the Web Patrick Raoul Philipp Tag der mündlichen Prüfung: 13.04.2018 Referent: PD Dr. Achim Rettinger Korreferent: Prof. Dr. Kristian Kersting Karlsruhe, 2018 0 Abstract This thesis deals with solving multi-step tasks by using advice from experts, which are algo- rithms to solve individual steps of such tasks. We contribute with methods for maximizing the number of correct task solutions by selecting and combining experts for individual task instances and methods for automating the process of solving tasks on the Web, where experts are available as Web services. Multi-step tasks frequently occur in Natural Language Processing (NLP) or Computer Vi- sion, and as research progresses an increasing amount of exchangeable experts for the same steps are available on the Web. Service provider platforms such as Algorithmia monetize ex- pert access by making expert services available via their platform and having customers pay for single executions. Such experts can be used to solve diverse tasks, which often consist of multiple steps and thus require pipelines of experts to generate hypotheses. We perceive two distinct problems for solving multi-step tasks with expert services: (1) Given that the task is sufficiently complex, no single pipeline generates correct solutions for all possible task instances. One thus must learn how to construct individual expert pipelines for individual task instances in order to maximize the number of correct solutions, while also taking into account the costs adhered to executing an expert. (2) To automatically solve multi- step tasks with expert services, we need to discover, execute and compose expert pipelines. With mostly textual descriptions of complex functionalities and input parameters, Web au- tomation entails to integrate available expert services and data, interpreting user-specified task goals or efficiently finding correct service configurations. In this thesis, we present solutions to both problems: (1) We enable to learn well-performing expert pipelines assuming available reference data sets (comprising a number of task instances and solutions), where we distinguish between centralized and decentralized decision-making. We formalize the problem as specialization of a Markov Decision Process (MDP), which we refer to as Expert Process (EP) and integrate techniques from Statistical Relational Learning (SRL) or Multiagent coordination. (2) We develop a framework for automatically discovering, executing and composing expert pipelines by exploiting methods developed for the Semantic Web. We lift the representations of experts with structured vocabularies modeled with the Resource Description Framework (RDF) and extend EPs to Semantic Expert Processes (SEPs) to enable the data-driven execution of experts in Web-based architectures. We evaluate our methods in different domains, namely Medical Assistance with tasks in Image Processing and Surgical Phase Recognition, and NLP for textual data on the Web, where we deal with the task of Named Entity Recognition and Disambiguation (NERD). v 0 Publications Text as well as figures in this thesis have partly been published in the following papers: Philipp Gemmeke, Maria Maleshkova, Patrick Philipp, Michael Götz, Christian We- ber, Benedikt Kämpgen, Sascha Zelzer, Klaus Maier-Hein, and Achim Rettinger. Us- ing linked data and web apis for automating the pre-processing of medical images. In Proceedings of the 5th International Workshop on Consuming Linked Data (COLD'14) co-located with the 13th International Semantic Web Conference (ISWC'14), Riva del Garda, Italy, pages 25–36, 2014. [51] Patrick Philipp, Maria Maleshkova, Achim Rettinger, and Darko Katic. A semantic framework for sequential decision making. In Engineering the Web in the Big Data Era - 15th International Conference, ICWE'15, Rotterdam, The Netherlands, pages 392– 409, 2015. doi:10.1007/978-3-319-19890-3_25. [101] Patrick Philipp, Maria Maleshkova, Darko Katic, Christian Weber, Michael Götz, Achim Rettinger, Stefanie Speidel, Benedikt Kämpgen, Marco Nolden, Anna-Laura Wekerle, Rüdiger Dillmann, Hannes Kenngott, Beat P. Müller-Stich, and Rudi Studer. Toward cognitive pipelines of medical assistance algorithms. Int. J. Computer As- sisted Radiology and Surgery, 11(9):1743–1753, 2016. doi: 10.1007/s11548-015-1322- y. [102] Nicolai Schoch, Patrick Philipp, Tobias Weller, Sandy Engelhardt, Mykola Volovyk, Andreas Fetzer, Marco Nolden, Raffaele De Simone, Ivo Wolf, Maria Maleshkova, Achim Rettinger, Rudi Studer, and Vincent Heuveline. Cognitive tools pipeline for assistance of mitral valve surgery. In Proceedings of SPIE, Medical Imaging'16: Image-Guided Procedures, Robotic Interventions and Modeling, pages 9786–9794, 2016. [114] Patrick Philipp and Achim Rettinger. Reinforcement learning for multi-step expert ad- vice. In Proceedings of the 16th Conference on Autonomous Agents and Multiagent Systems, AAMAS'17, São Paulo, Brazil, pages 962–971, 2017. [100] vii Patrick Philipp, Achim Rettinger, and Maria Maleshkova. On automating de- centralized multi-step service combination. In IEEE International Conference on Web Services, ICWS'17, Honolulu, HI, USA, pages 736–743, 2017. doi: 10.1109/ICWS.2017.89. [104] Patrick Philipp, Maria Maleshkova, Achim Rettinger, and Darko Katic. A semantic framework for sequential decision making. J. Web Eng., 16(5&6):471–504, 2017. [103] viii 0 Contents I Foundations1 1 Introduction & Overview3 1.1 Introduction...................................3 1.2 Challenges....................................7 1.2.1 Learning................................7 1.2.2 Automation...............................9 1.3 Research Questions & Hypotheses....................... 10 1.4 Contributions.................................. 12 1.5 Scope of the Thesis............................... 13 1.6 Outline..................................... 15 2 Scenarios 17 2.1 Named Entity Recognition & Disambiguation................. 17 2.1.1 Description............................... 17 2.1.2 Challenges............................... 18 2.2 Surgical Phase Recognition........................... 18 2.2.1 Description............................... 18 2.2.2 Challenges............................... 19 2.3 Tumor Progression Mapping.......................... 19 2.3.1 Description............................... 19 2.3.2 Challenges............................... 20 3 Preliminaries 23 3.1 Multi-Step Tasks................................ 23 3.2 Supervised Machine Learning......................... 27 3.2.1 Model-specific Assumptions...................... 28 3.2.2 Learning Protocols........................... 29 3.2.3 Prediction Settings........................... 30 3.2.4 Selected Learning Scenarios...................... 30 3.2.5 Evaluating Learned Models...................... 33 3.2.6 Learning Combination Functions over Models............ 34 3.3 Decision-Making................................ 34 3.3.1 Prediction with Expert Advice..................... 35 3.3.2 Markov Decision Processes...................... 38 3.3.3 Multiagent Decision Processes..................... 42 3.4 Semantic Web.................................. 44 3.4.1 Resource Description Framework................... 45 ix Contents 3.4.2 RDF Vocabularies........................... 45 3.4.3 Notation3................................ 46 3.4.4 SPARQL................................ 46 3.4.5 Linked Data............................... 47 3.4.6 Services in the Semantic Web..................... 48 4 Related Work 51 4.1 Multi-Step Tasks in Single-Agent Systems................... 51 4.1.1 Learning- & Decision-Theoretic Approaches............. 51 4.1.2 Service Approaches.......................... 53 4.2 Multi-Step Tasks in Multiagent Systems.................... 53 4.2.1 Multiagent Systems for Single-Agent Tasks.............. 53 4.2.2 Multiagent Systems for Multiagent Tasks............... 54 4.3 Multi-Step Tasks on the Web.......................... 55 4.3.1 Semantic Web Service Description Languages............ 55 4.3.2 Decision-Making Frameworks & Applications............ 55 4.3.3 Workflow Systems........................... 56 II Learning 59 5 Learning with Expert Processes 61 5.1 Introduction................................... 61 5.1.1 Challenges............................... 63 5.1.2 Contributions.............................. 64 5.2 Problem Formalization............................. 64 5.3 Meta Dependencies............................... 68 5.3.1 Single Experts............................. 70 5.3.2 Pairwise Intra-Step Experts....................... 71 5.3.3 Pairwise Inter-Step Experts....................... 71 5.4 Online Reinforcement Learning........................ 74 5.4.1 EWH with Meta Dependencies..................... 74 5.4.2 EWH with Incomplete Information.................. 75 5.5 Batch Reinforcement Learning......................... 77 5.5.1 Probabilistic Soft Logic and Hinge-Loss Markov Random Fields... 78 5.5.2 Meta Dependencies with PSL..................... 79 5.6 Discussion.................................... 81 5.7 Summary.................................... 81 6 Learning with Multiagent Expert Processes 83 6.1 Introduction................................... 83 6.1.1 Challenges............................... 85 6.1.2 Contributions.............................. 85 6.2