IMPROVING DECISION TREE AND NEURAL NETWORK LEARNING FOR EVOLVING DATA-STREAMS by DIEGO MARRON´ VIDA Advisors: Eduard Ayguad´e Jos´e Ramon´ Herrero Albert Bifet DISSERTATION Submitted in partial fulfilment of the requirements for the degree of Doctor of Philosophy in the Department of Computer Architecture Universitat Polit`ecnica de Catalunya 2019 Barcelona, Spain Abstract High-throughput real-time Big Data stream processing requires fast incremental algorithms that keep models consistent with most recent data. In this scenario, Hoeffding Trees are considered the state-of-the- art single classifier for processing data streams and they are widely used in ensemble combinations. This thesis is devoted to the improvement of the performance of algorithms for machine learning/artificial intelligence on evolving data streams. In particular, we focus on improving the Hoeffding Tree clas- sifier and its ensemble combinations, in order to reduce its resource consumption and its response time latency, achieving better throughput when processing evolving data streams. First, this thesis presents a study on using Neural Networks (NN) as an alternative method for processing data streams. The use of random features for improving NNs training speed is proposed and important is- sues are highlighted about the use of NN on a data stream setup. These issues motivated this thesis to go in the direction of improving the cur- rent state-of-the-art methods: Hoeffding Trees and their ensemble com- binations. Second, this thesis proposes the Echo State Hoeffding Tree (ESHT), as an extension of the Hoeffding Tree to model time-dependencies typ- ically present in data streams. The capabilities of the new proposed ar- chitecture on both regression and classification problems are evaluated. Third, a new methodology to improve the Adaptive Random Forest (ARF) is developed. ARF has been introduced recently, and it is con- sidered the state-of-the-art classifier in the MOA framework (a popular framework for processing evolving data streams). This thesis proposes the Elastic Swap Random Forest, an extension to ARF that reduces the number of base learners in the ensemble down to one third on average, while providing accuracy similar to that of the standard ARF with 100 trees. And finally, a last contribution on a multi-threaded high performance scalable ensemble design that is highly adaptable to a variety of hard- ware platforms, ranging from server-class to edge computing. The pro- posed design achieves throughput improvements of 85x (Intel i7), 143x (Intel Xeon parsing from memory), 10x (Jetson TX1, ARM) and 23x (X- Gene2, ARM) compared to single-threaded MOA on i7. In addition, the proposal achieves 75% parallel efficiency when using 24 cores on the Intel Xeon. i ACKNOWLEDGEMENTS This dissertation would not be possible without guidance and continuous support of my advisors, Eduard Ayguad´e, Jos´e Ramon´ Herrrero, and Albert Bifet. Special mention to Nacho Navarro, for taking me as his PhD student, and giving full resources and freedom to pursue my research; I know how happy and proud he would be to see this work completed. You all have been great role models as researchers, mentors and friends. I would like to thank to my colleagues at Barcelona Supercomputing Cen- ter, Toni Navarro, Miquel Vidal, Marc Jorda,` Kevin Sala and Pau Farr´e. For your patience and the insane funny moments and crazy moments during this journey. I would like to mention my earlier colleagues at BSC, Lluis Vilanova and Javier Cabezas for their mentorship and help at the beginning of this PhD. Last, but certainly not least, I am extremely grateful to Judit, Valeria and Hugo for your incredible patience, unconditional support and for inspiring me to pursue my dreams; half of this is your merit. I am also deeply grateful to my Parents, sister and bother in-law for your patience and support dur- ing these years, and most important for teaching me to never give up even when circumstances are not favourable. All of you always believed in me and wanted the best for me. Certainly, without you, none of this would have been possible. Thank you. iii Table of Contents Page List of Tables ix List of Figures xi 1 Introduction 1 1.1 Supervised Machine Learning . 1 1.2 Processing Data Streams in Real Time . 2 1.3 Decision Trees and Ensembles for Mining Big Data Streams . 3 1.4 Neural Networks . 4 1.5 Contributions of this Thesis . 5 1.5.1 Neural Networks and data streams . 5 1.5.2 Echo State Hoeffding Tree learning . 5 1.5.3 Resource-aware Elastic Swap Random Forest . 6 1.5.4 Ultra-low latency Random Forest . 6 1.6 Organization . 7 1.7 Publications . 8 2 Preliminaries and Related Work 11 2.1 Concept Drifting . 13 2.1.1 ADWIN Drift Detector . 14 2.2 Incremental Decision and Regression Trees . 15 2.2.1 Hoeffding Tree . 16 2.2.2 FIMT-DD . 19 2.2.3 Performance Extensions . 20 2.3 Ensemble Learning . 21 2.3.1 Online Bagging . 22 2.3.2 Leveraging Bagging . 22 2.3.3 Adaptive Random Forest . 23 2.4 Neural Networks for Data Streams . 23 2.4.1 Reservoir Computing: The Echo State Network . 24 2.5 Taxonomy . 26 3 Methodology 29 v TABLE OF CONTENTS TABLE OF CONTENTS 3.1 MOA . 30 3.2 Datasets . 30 3.2.1 Synthetic datasets . 31 3.2.2 Real World Datasets . 34 3.2.3 Datasets Summary . 35 3.3 Evaluation Setup . 35 3.3.1 Metrics . 35 3.3.2 Evaluation Schemes . 37 4 Data Stream Classification using Random Features 39 4.1 Random Projection Layer for Data Streams . 40 4.1.1 Gradient Descent with momentum . 40 4.1.2 Activation Functions . 42 4.2 Evaluation . 45 4.2.1 Activation functions . 45 4.2.2 RPL comparison with other data streams methods . 49 4.2.3 Batch vs Incremental . 50 4.3 Summary . 51 5 Echo State Hoeffding Tree Learning 53 5.1 The Echo State Hoeffding Tree . 53 5.2 Evaluation . 54 5.2.1 Regression evaluation methodology: learning functions . 55 5.2.2 Regression evaluation . 56 5.2.3 Classification evaluation methodology and real-world datasets . 64 5.2.4 Classification evaluation . 64 5.3 Summary . 66 6 Resource-aware Elastic Swap Random Forest for Evolving Data Streams 69 6.1 Preliminaries . 70 6.2 ELASTIC SWAP RANDOM FOREST . 72 6.3 Experimental Evaluation . 76 6.3.1 SWAP RANDOM FOREST . 77 6.3.2 ELASTIC SWAP RANDOM FOREST . 78 6.3.3 ELASTIC RANDOM FOREST . 83 6.4 Summary . 87 7 Low-Latency Multi-threaded Ensemble Learning for Dynamic Big Data Streams 89 7.1 LMHT Design Overview . 90 7.1.1 Tree Structure . 90 vi TABLE OF CONTENTS TABLE OF CONTENTS 7.1.2 Leaves and Counters . 92 7.2 Multithreaded Ensemble Learning . 93 7.2.1 Instance Buffer . 94 7.2.2 Random Forest Workers and Learners . 94 7.3 Implementation Notes . 95 7.4 Experimental Evaluation . 95 7.4.1 Hoeffding Tree Accuracy . 96 7.4.2 Hoeffding Tree Throughput Evaluation . 97 7.4.3 Random Forest Accuracy and Throughput . 99 7.4.4 Random Forest Scalability . 103 7.5 Summary . 104 8 Conclusions 109 8.1 Summary of Results . 109 8.2 Future Work . 111 Bibliography 113 vii List of Tables TABLE Page 3.2.1Synthetic (top) and real-world (bottom) datasets used for perfor- mance evaluation and comparison. 36 4.2.1Random numbers initialization strategy for the different activation functions . 45 4.2.2ELEC dataset best results obtained by RPL with different activation functions . 46 4.2.3COVT Evaluation . 47 4.2.4SUSY Evaluation . 48 4.2.5RPL accuracy (%) comparison against other popular data streams methods. 50 4.2.6SGD Batch vs Incremental . 50 5.2.1Map from ASCII domain to 4-symbols . 63 5.2.2Email address detector results . 63 5.2.3ESHT performance comparison against a single HT, and the best results obtained on each dataset. 66 5.2.4Comparing ESHT execution time against other data-streams meth- ods. 66 6.1.1Difference in accuracy for difference ARF sizes with respect the best one. A negative result means worse then the best. 71 6.1.2Average number of background learners for ARF with 100 and 50 learners. Note that RTG dataset has no drift, thus, ARF needed 0 background learners . 72 6.3.1Synthetic (top) and real-world (bottom) datasets used for perfor- mance evaluation and comparison. Synthetic datasets drift type: A (abrupt), G (gradual) I.F (incremental fast), I.M (incremental moderate), N (None) . 76 6.3.2Accuracy comparison between ARF100 and SRF with jFSj = 35 and jFSj = 50 ............................. 78 6.3.3ESRF comparison with ARF100. Resource-constrained scenario: Tg = 0:01 and Ts = 0:001 ....................... 83 ix List of Tables List of Tables 6.3.4ESRF comparison with ARF100. Ts = Tg = 0:001 . 83 6.3.5ESRF resize factor r = 5 comparison with ARF100. Tg = 0:01 and Ts = 0:5 ................................. 87 6.3.6ERF comparison with ARF100. Resource-constrained scenario: Tg = 0:01 and Ts = 0:001 ........................... 87 6.3.7ERF comparison with ARF100. Resource-constrained scenario: Tg = 0:1 and Ts = 0:1 ............................ 88 7.4.1Datasets used in the experimental evaluation, including both real world and synthetic datasets . 96 7.4.2Platforms used in the experimental evaluation . 97 7.4.3Single Hoeffding Tree accuracy comparison . 97 7.4.4Single Hoeffding Tree throughput (instances per ms) on Intel (top) and ARM (bottom) compared to MOA. # indicates speed-down (MOA is faster) . 98 7.4.5Comparing LMHT parser overhead (instances per ms). Parser in- cludes time to parse and process input data; No Parser means data is already parsed in memory.