Linköping University | Department of Computer and Information Science Master’s thesis, 30 ECTS | Computer Science 2019 | LIU-IDA/LITH-EX-A--19/071--SE GreenML – A methodology for fair evaluation of machine learning algorithms with respect to resource consumption Grön maskininlärning – En metod för rättvis utvärdering av maskininlärningsalgorithmer baserat på resursanvändning Anton Dalgren Ylva Lundegård Examiner : Petru Eles External supervisor : Armin Catovic Linköpings universitet SE–581 83 Linköping +46 13 28 10 00 , www.liu.se Upphovsrätt Detta dokument hålls tillgängligt på Internet - eller dess framtida ersättare - under 25 år från publiceringsdatum under förutsättning att inga extraordinära omständigheter uppstår. Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner, skriva ut enstaka kopior för enskilt bruk och att använda det oförändrat för ickekommersiell forskning och för undervisning. Överföring av upphovsrätten vid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning av dokumentet kräver upphovsmannens medgivande. För att garantera äktheten, säkerheten och tillgängligheten finns lösningar av teknisk och administrativ art. Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i den omfattning som god sed kräver vid användning av dokumentet på ovan beskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådan form eller i sådant sammanhang som är kränkande för upphovsmannens litterära eller konstnärliga anseende eller egenart. För ytterligare information om Linköping University Electronic Press se förlagets hemsida http://www.ep.liu.se/. Copyright The publishers will keep this document online on the Internet - or its possible replacement - for a period of 25 years starting from the date of publication barring exceptional circumstances. The online availability of the document implies permanent permission for anyone to read, to download, or to print out single copies for his/hers own use and to use it unchanged for non-commercial research and educational purpose. Subsequent transfers of copyright cannot revoke this permission. All other uses of the document are conditional upon the consent of the copyright owner. The publisher has taken technical and administrative measures to assure authenticity, security and accessibility. According to intellectual property law the author has the right to be mentioned when his/her work is accessed as described above and to be protected against infringement. For additional information about the Linköping University Electronic Press and its procedures for publication and for assurance of document integrity, please refer to its www home page: http://www.ep.liu.se/. Anton Dalgren © Ylva Lundegård Abstract Impressive results can be achieved when stacking deep neural networks hierarchies together. Several machine learning papers claim state-of-the-art results when evaluating their models with different accuracy metrics. However, these models come at a cost, which is rarely taken into consideration. This thesis aims to shed light on the resource consumption of machine learning algorithms, and therefore, five efficiency metrics are proposed. These should be used for evaluating machine learning models, taking accuracy, model size, and time and energy consumption for both training and inference into account. These metrics are intended to allow for a fairer evaluation of machine learning models, not only looking at accuracy. This thesis presents an example of how these metrics can be used by applying them to both text and image classification tasks using the algorithms SVM, MLP, and CNN. Acknowledgments We want to thank Armin Catovic, our supervisor at Ericsson, for assisting us during the entire thesis. We would also like to thank our examiner Petru Eles. A special thanks to our families and friends for their support, encouragement and laughter during these years at Linköping University, we would not have been able to this without you. We had a blast! Anton Dalgren and Ylva Lundegård Linköping, August 2019 iv Contents Abstract iii Acknowledgments iv Contents v List of Figures vi List of Tables vii 1 Introduction 1 1.1 Motivation . 1 1.2 Aim............................................ 1 1.3 Research questions . 2 1.4 Delimitations . 2 2 Background 3 3 Theory 4 3.1 Machine Learning . 4 3.2 Neural Networks . 7 3.3 Frameworks . 13 3.4 Related work . 14 4 Method 18 4.1 Efficiency metric . 18 4.2 Implementation . 19 4.3 Measurements . 26 5 Results 30 5.1 Image classification . 30 5.2 Text classification . 36 6 Discussion 42 6.1 Results . 42 6.2 Method . 45 6.3 The work in a wider context . 50 7 Conclusion 52 7.1 Research questions . 52 7.2 Future work . 53 Bibliography 54 v List of Figures 3.1 The optimal hyperplane in a linear SVM . 7 3.2 Example of a MLP structure . 8 3.3 A 2x2 receptive field of a feature map . 9 3.4 An example of a CNN structure . 9 ( ) = 1 3.5 The Sigmoid activation function f x 1+e´x ..................... 11 3.6 The ReLU activation function f (x) = maxt0, xu .................... 12 3.7 Two neural networks where one has been applying dropout . 12 4.1 The MNIST dataset . 20 4.2 The Fashion-MNIST dataset with labels . 21 4.3 The CIFAR10 dataset with labels . 22 4.4 Flowchart over software implementation of evaluation tool . 28 5.1 CIFAR-10 - Accuracy . 32 5.2 CIFAR-10 - Model Size Efficiency . 32 5.3 CIFAR-10 - Training Time Efficiency . 33 5.4 CIFAR-10 - Training Energy Efficiency . 33 5.5 CIFAR-10 - Inference Time Efficiency . 34 5.6 CIFAR-10 - Inference Energy Efficiency . 34 5.7 CIFAR-10 - Training energy in relation to accuracy . 35 5.8 20 newsgroups - Accuracy . 38 5.9 20 newsgroups - Model Size Efficiency . 38 5.10 20 newsgroups - Training Time Efficiency . 39 5.11 20 newsgroups - Training Energy Efficiency . 39 5.12 20 newsgroups - Inference Time Efficiency . 40 5.13 20 newsgroups - Inference Energy Efficiency . 40 5.14 20 newsgroups - Training energy in relation to accuracy . 41 6.1 Expected energy usage by datacenters from 2010-2030 . 51 vi List of Tables 3.1 Confusion matrix . 5 5.1 All results for image classification . 31 5.2 Metric result for image classification . 31 5.3 All results for text classification . 37 5.4 Metric result for text classification . 37 vii 1 Introduction This chapter describes the motivation for this thesis. It also presents the research questions and delimitations. 1.1 Motivation The amount of compute needed for AI, which machine learning is a part of, is growing rapidly. The amount of compute for the largest AI training runs is doubling every 3.5 month. Since 2012, the amount of compute needed has increased by 300 000 times. [1] Several machine learning papers claim state-of-the-art results when evaluating the models with different accuracy metrics. It is possible to get impressive results when stacking deep neural networks hierarchies together and using the latest hardware. However, this comes at a cost. This cost includes time, memory and energy, and it is rarely taken into account in research papers [26]. It is easy to get blinded by the drastically increasing results of the models. This growth brings issues that become increasingly important, and we need to start paying attention to the cost of machine learning. These machine learning algorithms are starting to get too complex for the hardware that we have today. Instead of only focusing on making better hardware, we should also focus on having resource consumption in mind when choosing a machine learning model. 1.2 Aim This thesis aims to test a set of metrics for evaluating machine learning algorithms, taking into account time, memory, energy and accuracy. When doing so, it is also essential to make a comparison between the different algorithms as fair as possible. That is because the algorithms are all designed to work in different ways and have different strengths and weaknesses. With that said, the goal is that this thesis leads to an example of how to make a more fair evaluation between algorithms, and not only favor the ones with the best accuracy. 1 1.3. Research questions 1.3 Research questions This thesis tries to answer the following questions: 1. How can we make a fair comparison between machine learning algorithms with respect to resource consumption? 2. How to compare machine learning algorithms to each other using efficiency metrics based on resource consumption? 1.4 Delimitations The only measurements that are taken into account when evaluating these machine learning algorithms are accuracy, model size, energy consumption for the CPU and GPU, and time consumption for training and inference. More things could be taken into account, for example, memory consumption during runtime and some measurement on how serious the errors are. However, this is outside the scope of this thesis. This thesis should not either be seen as an evaluation of machine learning algorithms. It instead provides a methodology for evaluating a certain implementation of a machine learning algorithm. 2 2 Background This thesis was conducted at Ericsson. Ericsson is a Swedish company that was founded in 1876 in Stockholm. They have about 95 000 employees across the world, of which about 12 500 in Sweden. It is one of the worlds leading provider of Information and Communication Technology (ICT) solutions. About 40% of the world’s mobile traffic is carried through Ericsson’s networks. 1 Armin Catovic, who is a senior specialist in machine learning at Ericsson, proposed this thesis. The authors did not have much prior experience in machine learning but were interested in the evaluation. The thesis was conducted at Ericsson’s headquarter in Kista (Stockholm). 1 https://www.ericsson.com/en/about-us 3 3 Theory This chapter will describe the theory needed for understanding the method used in this thesis. 3.1 Machine Learning Machine learning is a buzzword that is being used a lot nowadays.