Online Machine Learning Techniques for Predicting Operator Performance Master's Thesis by Ahmet Anil Pala Submitted to the Faculty IV, Electrical Engineering and Computer Science Database Systems and Information Management Group in partial fulfillment of the requirements for the degree of Master of Science in Computer Science as part of the ERASMUS MUNDUS programme IT4BI arXiv:1605.01029v1 [cs.LG] 3 May 2016 at the Technische Universitat¨ Berlin August 15, 2015 Thesis Advisor: Max Heimel Thesis Supervisor: Prof. Dr. Volker Markl Eidesstattliche Erkl¨arung Ich erkl¨arean Eides statt, dass ich die vorliegende Arbeit selbstst¨andigverfasst, andere als die angegebenen Quellen/Hilfsmittel nicht benutzt, und die den benutzten Quellen w¨ortlich und inhaltlich entnommenen Stellen als solche kenntlich gemacht habe. Statutory Declaration I declare that I have authored this thesis independently, that I have not used other than the declared sources/resources, and that I have explicitly marked all material which has been quoted either literally or by content from the used sources. Berlin, August 15, 2015 Ahmet Anil Pala TECHNISCHE UNIVERSITAT¨ BERLIN Abstract Electrical Engineering and Computer Science Institute of Software Engineering and Theoretical Computer Science Master of Science Online Machine Learning for Predicting Operator Performance by Ahmet Anil Pala This thesis explores a number of online machine learning algorithms. From a theoret- ical perspective, it assesses their employability for a particular function approximation problem where the analytical models fall short. Furthermore, it discusses the applica- tion of theoretically suitable learning algorithms to the function approximation problem at hand through an efficient implementation that exploits various computational and mathematical shortcuts. Finally, this thesis work evaluates the implemented learning algorithms according to various evaluation criteria through rigorous testing. TECHNISCHE UNIVERSITAT¨ BERLIN Abstract Electrical Engineering and Computer Science Institute of Software Engineering and Theoretical Computer Science Master of Science Online Machine Learning for Predicting Operator Performance by Ahmet Anil Pala Die vorliegende Arbeit untersucht eine Reihe von Online-Algorithmen des Maschinellen Lernens. Aus theoretischer Perspektive wird eine Einsch¨atzungzur Anwendbarkeit dieser Algorithmen f¨urein bestimmtes Funktionsapproximationsproblem gegeben, bei dem analytische Modelle unzureichend sind. Des Weiteren wird die Anwendung der theoretisch ad¨aquatenLernalgorithmen auf das gestellte Problem durch eine effiziente Implementierung diskutiert, die diverse mathematische Abk¨urzungenaufzeigt. Acknowledgements I would like to express my honest gratitude to my advisor, Max Heimel, who guided me through the thesis work with intellectually stimulating discussions on the topics of the thesis. I would like to thank Prof. Volker Markl for helping me choose my final year special- ization offered in Technische Universit¨atBerlin with his inspiring presentation on the specialization. I would also like to thank all the professors who lectured me during my studies. Finally, I would like to thank my family for always being supportive of me throughout my entire life. v Contents Abstract iii Abstract iv Acknowledgementsv List of Figures ix List of Tables xiv 1 Introduction1 1.1 Motivation . .1 1.2 Structure of This Thesis . .2 2 Foundations3 2.1 Query Cost Modeling . .3 2.2 Statistical Learning . .5 2.2.1 Choice of Loss Function . .5 2.2.2 Function Space . .6 2.2.3 Empirical Risk Minimization (ERM) . .8 2.3 Data Stream Learning . 11 2.3.1 Non-Stationarity of Data Distribution in Streams . 13 2.3.2 Stationary Stream Learning . 13 2.3.3 Online Prediction Protocol . 14 2.3.4 Data Horizon and Data Obsolescence . 15 3 Problem 17 3.1 Ocelot Overview . 17 3.1.1 Main Design Principles . 17 3.1.2 Self-Adaptivity . 18 3.2 Learning Problem . 20 3.2.1 Correspondence to Data Stream Learning . 20 3.2.2 Required Properties of the Learning Model . 22 4 Approach 25 4.1 Modeling the problem . 25 vi Contents vii 4.2 Parametric Models . 26 4.2.0.1 Non-Linear feature space mapping . 27 4.2.1 MLE method . 27 4.2.1.1 MLE-based parameter estimation . 27 4.2.1.2 Measure of prediction uncertainty in MLE-method . 29 4.2.2 MAP method . 31 4.2.2.1 MAP-based parameter estimation . 31 4.2.2.2 Measure of prediction uncertainty in MAP method . 33 4.3 Non-Parametric Models . 33 4.3.1 Gaussian Process Regression . 34 4.3.1.1 Optimizing Hyperparameters . 38 4.3.2 Kernel Regression . 42 4.3.2.1 Kernel Density Estimation . 42 4.3.2.2 Nadaraya-Watson Estimator . 44 4.3.2.3 Prediction Bounds Estimation . 45 4.3.2.4 Optimizing Hyperparameters . 46 5 Implementation 48 5.1 Categorization of Regression Algorithms . 48 5.2 Categorization of Online Learners . 50 5.2.1 Forgetting-based Design . 50 5.2.2 Sliding-Window Design . 52 5.3 Online Learner Semantics . 53 5.4 Implementation of Online Learner Operations . 54 5.4.1 MLE method operations . 55 5.4.1.1 Implementation of predict for BayesianMLEForgetting 55 5.4.1.2 Implementation of update for BayesianMLEForgetting . 56 5.4.1.3 Implementation of predict for BayesianMLEWindowed . 57 5.4.1.4 Implementation of update for BayesianMLEWindowed .. 57 5.4.1.5 Implementation of tune for BayesianMLEWindowed ... 58 5.4.2 MAP method operations . 58 5.4.2.1 Implementation of predict for BayesianMAPForgetting 58 5.4.2.2 Implementation of update for BayesianMAPForgetting . 59 5.4.2.3 Implementation of predict for BayesianMAPWindowed . 59 5.4.2.4 Implementation of update for BayesianMAPWindowed .. 59 5.4.2.5 Implementation of tune for BayesianMAPWindowed ... 59 5.4.3 Gaussian Process Regression operations . 60 5.4.3.1 Implementation of predict for GPRegression ...... 60 5.4.3.2 Implementation of update for GPRegression ....... 61 5.4.3.3 Implementation of tune for GPRegression ........ 63 5.4.4 Kernel Regression operations . 64 5.4.4.1 Implementation of predict for KernelRegression ... 64 5.4.4.2 Implementation of update for KernelRegression .... 66 5.4.4.3 Implementation of tune for KernelRegression ..... 66 6 Evaluation 69 6.1 Evaluation Methodology . 69 Contents viii 6.1.1 Prequential Approach in Action . 75 6.1.2 Error Metrics . 76 6.1.3 Prediction Bound Metrics . 77 6.1.4 Space Efficiency Metrics . 78 6.1.5 Time Efficiency Metrics . 78 6.2 Evaluation Setting . 79 6.2.1 Evaluated Online Learners . 80 6.2.2 Synthetic Data Characteristics . 82 6.2.3 Ocelot Runtime Measurement Data Characteristics . 86 6.3 Analysis of Test Results . 86 6.3.1 Online Learners vs Batch Learners . 87 6.3.2 A general picture . 89 6.3.3 Sliding window size . 91 6.3.3.1 GPRegression Window Size . 100 6.3.3.2 KernelRegression Window Size . 103 6.3.3.3 BayesianMLE and BayesianMAP Window Size . 104 6.3.4 Forgetting-factor . 105 6.3.5 Feature Space Mapping . 107 6.3.6 Algorithm comparison . 109 6.3.7 Visualization of the cost function approximations by the short- listed learners on synthetic data sets . 123 6.3.8 Visualization of the cost function approximations by the short- listed learners on the measurement data . 130 7 Conclusion 136 7.1 Final Remarks on Online Regression Algorithms . 136 7.2 Future Work . 137 Bibliography 138 List of Figures 3.1 Overview of the device and algorithm selection in Ocelot. Note that the numbers written in the boxes that represent the active algorithm variant set denote the average runtime of the corresponding variant. 20 3.2 Runtime Predictor is visualized as a stream learner. Note that the irregu- lar spacing between the arriving items as well as between the predictions illustrates that the frequency of runtime predictions are requested by the algorithm selection logic of the Ocelot is variable . 21 5.1 Sliding window on a data stream is visualized . 52 5.2 State-transition diagram of the forgetting-based learners . 53 5.3 State-transition diagram of the sliding-windowed learners . 54 5.4 Ad-Hoc prediction bounds method is illustrated . 56 6.1 Inaccuracy Comparison of Batch and Online Learners. The bar values are obtained through aggregating the corresponding statistics over 576 stream simulations on 36 online learners and 576 tests on 12 batch learners respectively. 87 6.2 Inaccuracy Comparison of Batch and Online Learners on Non-Static Data. The bar values are obtained through aggregating the corresponding statis- tics over 276 stream simulations on 52 online learners and 276 tests on 12 batch learners respectively. 88 6.3 Side-by-side Accuracy Comparison of Batch and Online versions of Kernel Regression with training set of 96 and sliding window size of 96 respec- tively. The test data used is SYNTH ND CD 2000 2 10 1 22 and the resolution of the sliding error window accuracy evaluation method is set to96....................................... 89 6.4 A General Comparison of Online Algorithms. The values on columns are aggregated over 576 stream simulations on 52 online learners and grouped by the algorithm family. 90 6.5 Accuracy comparison of sliding windowed learners with varying window sizes. SMSE and SMSE ST values are aggregated over 576 stream simulations on 8 sliding windowed online learners (2 BayesianMAP, 2 BayesianMLE, 3 GPRegression and 1 KernelRegression variants) for each window size . 92 6.6 Accuracy comparison of batch learners with varying training set sizes. SMSE values are aggregated over 288 tests (with static data) on 4 different batch algorithm for each training set size . 92 6.7 Side-by-side Accuracy Comparison of BayesianMAPWindowed with differ- ent window sizes. The test data used is SYNTH ND CD 2000 2 10 1 22 and the resolution of the sliding error window accuracy evaluation method is set to 96 . 93 ix List of Figures x 6.8 Side-by-side Accuracy Comparison of Online GPWindowedGaussianKernelZeroMean variants with different window sizes.